minio-dotnet on mint:edge fails

[balamurugana]

The exception is something like below

Unhandled Exception: System.AggregateException: One or more errors occurred. (One or more errors occurred. (Assert.Fail failed. )) ---> System.AggregateException: One or more errors occurred. (Assert.Fail failed. ) ---> Microsoft.VisualStudio.TestTools.UnitTesting.AssertFailedException: Assert.Fail failed. 
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.HandleFail(String assertionName, String message, Object[] parameters)
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.Fail()
   at Minio.Functional.Tests.FunctionalTest.<PutObject_Task>d__52.MoveNext() in /mint/run/core/minio-dotnet/FunctionalTest.cs:line 755
   --- End of inner exception stack trace ---
   at System.ThrowHelper.ThrowAggregateException(List`1 exceptions)
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout)
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks)
   at Minio.Functional.Tests.FunctionalTest.<ListObjects_Test5>d__72.MoveNext() in /mint/run/core/minio-dotnet/FunctionalTest.cs:line 1635
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Minio.Functional.Tests.FunctionalTest.Main(String[] args) in /mint/run/core/minio-dotnet/FunctionalTest.cs:line 216

Such exception is seen in every mint automation run.

[nitisht]

@poornas could you please take a look..

[nitisht]

Observation: this failure is not visible when Mint is run locally against XL deployment

$ sudo docker run -e "SERVER_ENDPOINT=127.0.0.1:9000" -e "ACCESS_KEY=minio" -e "SECRET_KEY=minio123" -e "ENABLE_HTTPS=0" -e "MINT_MODE"=full --net=host minio/mint:edge

Running with
SERVER_ENDPOINT: 127.0.0.1:9000
ACCESS_KEY:      minio
SECRET_KEY:      ***REDACTED***
ENABLE_HTTPS:    0
SERVER_REGION:   us-east-1
MINT_DATA_DIR:   /mint/data
MINT_MODE:       full

To get logs, run 'docker cp a887484d8e72:/mint/log /tmp/mint-logs'
(1/11) Running aws-sdk-go tests ... done in 0 seconds
(2/11) Running aws-sdk-php tests ... done in 41 seconds
(3/11) Running aws-sdk-ruby tests ... done in 1 seconds
(4/11) Running awscli tests ... done in 24 seconds
(5/11) Running mc tests ... done in 5 seconds
(6/11) Running minio-dotnet tests ... done in 52 seconds
(7/11) Running minio-go tests ... done in 22 seconds
(8/11) Running minio-java tests ... done in 52 seconds
(9/11) Running minio-js tests ... done in 21 seconds
(10/11) Running minio-py tests ... done in 11 seconds
(11/11) Running s3cmd tests ... done in 8 seconds

All tests ran successfully

[poornas]

@nitisht, not aware how the mint automation is being run? If this is a q repo - don't have access.

[nitisht]

okay, let me check with @balamurugana probably this is not a minio-dotnet sdk issue.

[poornas]

The Assert.Fail() on line 755 should be removed - it is not displaying the actual exception returned to the client. From testing locally against s3 gateway, connection is being closed by foreign host.

ERRO[0309] Unable to save an object /miniodotnetf4mg05ek8cq88ta/miniodotnetuw2qd5745  cause=Put https://miniodotnetf4mg05ek8cq88ta.s3.amazonaws.com/miniodotnetuw2qd5745: Connection closed by foreign host https://miniodotnetf4mg05ek8cq88ta.s3.amazonaws.com/miniodotnetuw2qd5745. Retry again. source=[gateway-handlers.go:307:gatewayAPIHandlers.PutObjectHandler()] stack=cmd/gateway/s3/gateway-s3.go:294:(*s3Objects).PutObject() cmd/gateway-handlers.go:261:(GatewayLayer).PutObject-fm() cmd/gateway-handlers.go:305:gatewayAPIHandlers.PutObjectHandler() cmd/gateway-router.go:111:(gatewayAPIHandlers).PutObjectHandler-fm() /home/kris/go/src/net/http/server.go:1918:HandlerFunc.ServeHTTP() vendor/github.com/gorilla/mux/mux.go:107:(*Router).ServeHTTP() cmd/generic-handlers.go:558:pathValidityHandler.ServeHTTP() <autogenerated>:1:(*pathValidityHandler).ServeHTTP() cmd/generic-handlers.go:65:requestSizeLimitHandler.ServeHTTP() <autogenerated>:1:(*requestSizeLimitHandler).ServeHTTP() cmd/crossdomain-xml-handler.go:51:crossDomainPolicy.ServeHTTP() <autogenerated>:1:(*crossDomainPolicy).ServeHTTP() cmd/generic-handlers.go:219:redirectHandler.ServeHTTP() <autogenerated>:1:(*redirectHandler).ServeHTTP() cmd/generic-handlers.go:267:minioReservedBucketHandler.ServeHTTP() <autogenerated>:1:(*minioReservedBucketHandler).ServeHTTP() cmd/generic-handlers.go:246:cacheControlHandler.ServeHTTP() <autogenerated>:1:(*cacheControlHandler).ServeHTTP() cmd/generic-handlers.go:337:timeValidityHandler.ServeHTTP() <autogenerated>:1:(*timeValidityHandler).ServeHTTP() vendor/github.com/rs/cors/cors.go:190:(*Cors).Handler.func1() /home/kris/go/src/net/http/server.go:1918:HandlerFunc.ServeHTTP() cmd/generic-handlers.go:448:resourceHandler.ServeHTTP() <autogenerated>:1:(*resourceHandler).ServeHTTP() cmd/auth-handler.go:234:authHandler.ServeHTTP() <autogenerated>:1:(*authHandler).ServeHTTP() pkg/http/server.go:107:(*Server).Start.func1() /home/kris/go/src/net/http/server.go:1918:HandlerFunc.ServeHTTP() /home/kris/go/src/net/http/server.go:2619:serverHandler.ServeHTTP() /home/kris/go/src/net/http/server.go:1801:(*conn).serve()

[poornas]

@balamurugana/@nitisht , can you please take a look at server side logs for the mint automation ? You might want to see if error is due to low disk space - and server not being able to save the object.

[balamurugana]

@poornas there is no server log and low disk space problem i.e. no problem with the server. Interestingly below is an additional error from mint i.e. the test program crashes. Note that there is no OOM.

./run.sh: line 26:  1045 Aborted                 (core dumped) /mint/run/core/minio-dotnet/out/Minio.Functional.Tests >> "$output_log_file" 2> "$error_log_file"

[kannappanr]

PR https://github.com/minio/minio-dotnet/pull/206 has fixed this issue. Closing the issue as fixed.