search

aws / aws-sdk-net

The official AWS SDK for .NET. For more information on the AWS SDK for .NET, see our web site:
http://aws.amazon.com/sdkfornet/
Apache License 2.0
2.07k stars 861 forks source link

Continued use of AmazonS3Client leads to Kestrel error 'System.InvalidOperationException: Handle is already used by another Socket.' #1766

Closed tyler-shuhnicki closed 3 years ago

tyler-shuhnicki commented 3 years ago

Description

Our application is created using Elastic beanstalk. It is a .NET 5 app. We utilize nginx and kestrel on an Ubuntu EC2 instance.

We are hosting our .cshtml views on S3 allowing for an independent update of our views files, no longer requiring an app restart on the server. Our application must be live 24/7, creating some unique architecture.

To do this, I have implemented a custom file provider that retrieves files from S3. The file provider can work in three ways:

  1. Load a file once from S3, and continue to use the loaded file for the life of the application.
  2. Always load a file from S3, never caching
  3. (ideally) check a file for an update, and retrieve a new file from S3 if the last modified date is different from the date we have.

If the application is set to run as '1', the application is stable - it is fast, and works as expected indefinitely. Obviously, option 2 is not really idea due to performance, and has not been attempted. Method 3 is currently where we are at.

The application runs normally and stable for about two hours. After about 2 hours of idling, the Kestrel server totally locks up and refuses to respond to requests. Checking the logs, the only useful information in the stack trace is this: System.InvalidOperationException: Handle is already used by another Socket.

Some more detail - the difference between method 1 and 3 is pretty subtle. We are maintaining a dictionary in-memory of the files we have loaded and their timestamps. When a page is requested, we call ListObjectsV2Async (with just the page name to only return one result), compare the last modified date, and return a change token based on the difference of the last modified date. Seeing as method 1 is completely stable and method 3 degrades, it seems to me that that something unusual interaction is happening with the S3 client and the servers sockets - especially with the constant health check from Elastic Beanstalk. To be completely clear, the only extra code between method 1 and method 3 is the AWS S3 call to ListObjectsV2Async. Method 1 is the exact same file provider, only with returning a "false" watch token.

Is there a way that we need to be closing sockets that we are not? Is this potentially an issue with the S3 SDK itself? The relevant code is in the method "public IChangeToken Watch" as posted below. I have made sure to dispose anything that is disposable, but it does not make a difference. I am hoping someone is able to point me in the right direction.

Reproduction Steps

File Provider

  using System;
  using System.Collections;
  using System.Collections.Generic;
  using System.Diagnostics;
  using System.IO;
  using System.Linq;
  using System.Net;
  using System.Runtime.Caching;
  using Amazon;
  using Amazon.S3;
  using Amazon.S3.Model;
  using CFConnect.Models;
  using Microsoft.AspNetCore.Mvc.Razor;
  using Microsoft.Extensions.FileProviders;
  using Microsoft.Extensions.Options;
  using Microsoft.Extensions.Primitives;

  namespace CFConnect.Extensions
  {

    public class S3FileProvider : IFileProvider
    {
        private IAmazonS3 client;
        private string bucket;
        private static Dictionary<string, DateTimeOffset> watchFiles;

        public S3FileProvider(IAmazonS3 _client, string _bucket)
        {
            client = _client;
            bucket = _bucket;
            watchFiles = new Dictionary<string, DateTimeOffset>();
        }

        public IDirectoryContents GetDirectoryContents(string subpath)
        {
            throw new NotImplementedException();
        }

        public IFileInfo GetFileInfo(string subpath)
        {
            if (string.IsNullOrEmpty(subpath))
                return new NotFoundFileInfo(subpath);

            S3FileInfo s3File = new S3FileInfo(ref client, bucket, subpath);
            if (!watchFiles.ContainsKey(subpath))
            {
                //store the lastmodified date in a dictioanry, match it to the filename
                watchFiles.Add(subpath, s3File.LastModified);
            }
            else
            {
                watchFiles[subpath] = s3File.LastModified;
            }
            return s3File;
        }
        public IChangeToken Watch(string filter)
        {
            DateTimeOffset time;
            //check if the file is in the list our list
            //if the time = default, it's not *really* there and should not be considered to be cached
            if (watchFiles.TryGetValue(filter, out time) && time != default(DateTime)){
                //get file info from S3
                ListObjectsV2Response view = client.ListObjectsV2Async(new ListObjectsV2Request { BucketName = bucket, Prefix = filter.TrimStart('/') }).Result;
                if(((DateTimeOffset)view.S3Objects[0].LastModified).CompareTo(time) > 0) //IF the last modified is LATER than the one we have (great than 0), trigger a change
                {
                    return new S3ChangeToken(true);
                }
                return new S3ChangeToken(false); // it's up to date and does not need to be updated
            }
            else
            {
                //this will trigger a "yes this file has changed, go get it"
                return new S3ChangeToken(true); //file is not in collection; have it get a new file which should update last modified date
            }
        }

        public class S3ChangeToken : IChangeToken
        {

            bool compare;
            public S3ChangeToken(bool _compare)
            {
                compare = _compare;
            }
            public bool HasChanged
            {
                get
                {
                    return compare;
                }
            }
            public bool ActiveChangeCallbacks => false;

            public IDisposable RegisterChangeCallback(Action<object> callback, object state) => EmptyDisposable.Instance;

            internal class EmptyDisposable : IDisposable
            {
                public static EmptyDisposable Instance { get; } = new EmptyDisposable();
                private EmptyDisposable() { }
                public void Dispose() { }
            }
        }
    }

    public class S3FileInfo : IFileInfo
    {
        private string key;

        private bool exists;
        private byte[] viewContent;
        private DateTime lastModified;

        public S3FileInfo(ref IAmazonS3 _client, string _bucket, string _subpath)
        {//pass s3client by ref to avoid making a ton of httpconnections
            this.key = _subpath;
            try
            {
                using (GetObjectResponse view = _client.GetObjectAsync(_bucket, key.TrimStart('/')).Result)
                {
                    lastModified = view.LastModified;

                    using (var responseMem = new MemoryStream())
                    {
                        view.ResponseStream.CopyTo(responseMem);
                        viewContent = responseMem.ToArray();
                    }

                    exists = true;
                }
            }
            catch (AmazonS3Exception e)
            {
                if (e.StatusCode == HttpStatusCode.NotFound)
                {
                    exists = false;
                }
                else
                {
                    throw;
                }

            }
            catch (AggregateException e)
            {
                AmazonS3Exception ex = (AmazonS3Exception)e.InnerException;
                if (ex.StatusCode == HttpStatusCode.NotFound)
                {
                    exists = false;
                }
                else
                {
                    throw;
                }
            }
        }

        public bool Exists => exists;

        public long Length
        {
            get
            {
                using (var stream = new MemoryStream(viewContent))
                {
                    return stream.Length;
                }
            }
        }

        /// <summary>
        /// A http url to the file, including the file name.
        /// </summary>
        public string PhysicalPath => null;

        public string Name => Path.GetFileName(key);

        public DateTimeOffset LastModified => lastModified;

        public bool IsDirectory => false;

        public Stream CreateReadStream()
        {
            return new MemoryStream(viewContent);
        }
    } 
}

Startup.cs to add file provider

services.Configure<MvcRazorRuntimeCompilationOptions>(options => {
                    options.FileProviders.Clear();
                    options.FileProviders.Add(new S3FileProvider(client, bucketName));
                });

Logs

Stack Trace

   Dec 29 23:06:06 web[23605]: warn: Microsoft.AspNetCore.Server.Kestrel[0]
    Dec 29 23:06:06 web[23605]: Connection processing ended abnormally.
    Dec 29 23:06:06 web[23605]: System.InvalidOperationException: Handle is already used by another Socket.
    Dec 29 23:06:06 web[23605]: at System.Net.Sockets.SocketAsyncEngine.RegisterCore(IntPtr socketHandle, SocketAsyncContext context)
    Dec 29 23:06:06 web[23605]: at System.Net.Sockets.SocketAsyncEngine.RegisterSocket(IntPtr socketHandle, SocketAsyncContext context)
    Dec 29 23:06:06 web[23605]: at System.Net.Sockets.SocketAsyncContext.Register()
    Dec 29 23:06:06 web[23605]: at System.Net.Sockets.SocketAsyncContext.OperationQueue1.StartAsyncOperation(SocketAsyncContext context, TOperation operation, Int32 observedSequenceNumber, CancellationToken cancellationToken)
    Dec 29 23:06:06 web[23605]: at System.Net.Sockets.SocketAsyncContext.ReceiveAsync(Memory`1 buffer, SocketFlags flags, Int32& bytesReceived, Action5 callback, CancellationToken cancellationToken)
    Dec 29 23:06:06 web[23605]: at System.Net.Sockets.SocketAsyncEventArgs.DoOperationReceive(SafeSocketHandle handle, CancellationToken cancellationToken)
    Dec 29 23:06:06 web[23605]: at System.Net.Sockets.Socket.ReceiveAsync(SocketAsyncEventArgs e, CancellationToken cancellationToken)
    Dec 29 23:06:06 web[23605]: at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketReceiver.WaitForDataAsync()
    Dec 29 23:06:06 web[23605]: at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.ProcessReceives()
    Dec 29 23:06:06 web[23605]: at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.DoReceive()
    Dec 29 23:06:06 web[23605]: at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
    Dec 29 23:06:06 web[23605]: at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
    Dec 29 23:06:06 web[23605]: at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token)
    Dec 29 23:06:06 web[23605]: at System.IO.Pipelines.Pipe.DefaultPipeReader.ReadAsync(CancellationToken cancellationToken)
    Dec 29 23:06:06 web[23605]: at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.BeginRead(ValueTask1& awaitable)
    Dec 29 23:06:06 web[23605]: at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication1 application)
    Dec 29 23:06:06 web[23605]: at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication1 application)

Environment

This is a :bug: bug-report

ashishdhingra commented 3 years ago

Hi @tyler-shuhnicki,

Good morning.

Thanks for posting the issue. Looking high level at the code, one possible reasoning could be that the IAmazonS3 client is not disposed, causing the issue. You might want to try implementing IDisposable interface on S3FileProvider class and in Dispose() method, close the IAmazonS3 object, something like:

public class S3FileProvider : IFileProvider, IDisposable
{
    ...
    public void Dispose()
    {
        if (client != null) client.Dispose(); 
    }
}

And then possible use S3FileProvider instance in a using block:

using(S3FileProvider s3FileProvider = new S3FileProvider(s3clientObject, bucketName))
{
    ...
}

Although the unused objects might be disposed off by .NET framework automatically, it is a good idea to use disposable pattern to close non-managed connections (sockets in this case) to avoid starvation.

Thanks, Ashish

tyler-shuhnicki commented 3 years ago

Hi Ashish,

Thanks for your quick response. I've actually just implemented something similar, and will know in about 2 hours if it is more similar.

I had read that the Amazon S3 Client was intended to be created once and used in multiple places, thus the design given above. Is it the case that it is more preferable to use a dispose of a client after use?

Thanks! Tyler

ashishdhingra commented 3 years ago

Hi Ashish,

Thanks for your quick response. I've actually just implemented something similar, and will know in about 2 hours if it is more similar.

I had read that the Amazon S3 Client was intended to be created once and used in multiple places, thus the design given above. Is it the case that it is more preferable to use a dispose of a client after use?

Thanks! Tyler

Hi @tyler-shuhnicki,

We need to see how IAmazonS3 instance injected into S3FileProvider class. Are you creating S3 client instance manually and then using it to create instance of S3FileProvider class? Or is it being injected via DI framework as a Singleton instance? If it is the first case, then you might run into the issue that you mentioned.

Thanks, Ashish

tyler-shuhnicki commented 3 years ago

Hi @ashishdhingra,

Previously, we were creating an instance of the S3 client in the Startup.cs class and passing it into the File provider, like so:

services.Configure<MvcRazorRuntimeCompilationOptions>(options => {
                    options.FileProviders.Clear();
                    options.FileProviders.Add(new S3FileProvider(awsAccessKey, awsSecretKey, bucketName));
                });

Currently, I have update the code to pass in the credentials, and the fileprovider will create them itself. The Startup.CS code now looks like:

IAmazonS3 client = new AmazonS3Client(awsAccessKey, awsSecretKey, RegionEndpoint.USEast1);
                services.Configure<MvcRazorRuntimeCompilationOptions>(options => {
                    options.FileProviders.Clear();
                    options.FileProviders.Add(new S3FileProvider(client, bucketName));
                });

and the full File Provider class looks like:

using System;
using System.Collections;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Net;
using System.Runtime.Caching;
using Amazon;
using Amazon.S3;
using Amazon.S3.Model;
using CFConnect.Models;
using Microsoft.AspNetCore.Mvc.Razor;
using Microsoft.Extensions.FileProviders;
using Microsoft.Extensions.Options;
using Microsoft.Extensions.Primitives;

namespace CFConnect.Extensions
{
    public class S3FileProvider : IFileProvider
    {
        private string awsAccessKey;
        private string awsSecretKey;
        private string bucket;
        private static Dictionary<string, DateTimeOffset> watchFiles;
        public S3FileProvider(string _awsAccessKey, string _awsSecretKey, string _bucket)
        {
            awsAccessKey = _awsAccessKey;
            awsSecretKey = _awsSecretKey;
            bucket = _bucket;
            watchFiles = new Dictionary<string, DateTimeOffset>();
        }

        public IDirectoryContents GetDirectoryContents(string subpath)
        {
            throw new NotImplementedException();
        }

        public IFileInfo GetFileInfo(string subpath)
        {
            if (string.IsNullOrEmpty(subpath))
                return new NotFoundFileInfo(subpath);

            S3FileInfo s3File = new S3FileInfo(awsAccessKey, awsSecretKey, bucket, subpath);
            if (!watchFiles.ContainsKey(subpath))
            {
                //store the lastmodified date in a dictioanry, match it to the filename
                watchFiles.Add(subpath, s3File.LastModified);
            }
            else
            {
                watchFiles[subpath] = s3File.LastModified;
            }
            return s3File;
        }

        /// <summary>
        /// Watch is not supported.
        /// </summary>
        public IChangeToken Watch(string filter)
        {
            //return new S3ChangeToken(false);
            //TODO: this needs to work. something that has to do with AWS SDK not releasing sockets.
            DateTimeOffset time;
            //check if the file is in the list our list
            //if the time = default, it's not *really* there and should not be considered to be cached
            if (watchFiles.TryGetValue(filter, out time) && time != default(DateTime))
            {
                //get file info from S3
                using (AmazonS3Client client = new AmazonS3Client(awsAccessKey, awsSecretKey, RegionEndpoint.USEast1))
                {
                    GetObjectMetadataResponse view = client.GetObjectMetadataAsync(bucket, filter.TrimStart('/')).Result;
                    if (((DateTimeOffset)view.LastModified).CompareTo(time) > 0) //IF the last modified is LATER than the one we have (great than 0), trigger a change
                    {
                        return new S3ChangeToken(true);
                    }
                    return new S3ChangeToken(false); // it's up to date and does not need to be updated
                }
            }
            else
            {
                //this will trigger a "yes this file has changed, go get it"
                return new S3ChangeToken(true); //file is not in collection; have it get a new file which should update last modified date
            }
        }

        public class S3ChangeToken : IChangeToken
        {

            bool compare;
            public S3ChangeToken(bool _compare)
            {
                compare = _compare;
            }
            public bool HasChanged
            {
                get
                {
                    return compare;
                }
            }
            public bool ActiveChangeCallbacks => false;

            public IDisposable RegisterChangeCallback(Action<object> callback, object state) => EmptyDisposable.Instance;

            internal class EmptyDisposable : IDisposable
            {
                public static EmptyDisposable Instance { get; } = new EmptyDisposable();
                private EmptyDisposable() { }
                public void Dispose() { }
            }
        }
    }

    public class S3FileInfo : IFileInfo
    {
        private string key;

        private bool exists;
        private byte[] viewContent;
        private DateTime lastModified;

        public S3FileInfo(string accessKey, string secretKey, string _bucket, string _subpath)
        {//pass s3client by ref to avoid making a ton of httpconnections
            this.key = _subpath;
            try
            {
                using (AmazonS3Client client = new AmazonS3Client(accessKey, secretKey, RegionEndpoint.USEast1))
                {
                    using (GetObjectResponse view = client.GetObjectAsync(_bucket, key.TrimStart('/')).Result)
                    {
                        lastModified = view.LastModified;

                        using (var responseMem = new MemoryStream())
                        {
                            view.ResponseStream.CopyTo(responseMem);
                            viewContent = responseMem.ToArray();
                        }

                        exists = true;
                    }
                }
            }
            catch (AmazonS3Exception e)
            {
                if (e.StatusCode == HttpStatusCode.NotFound)
                {
                    exists = false;
                }
                else
                {
                    throw;
                }

            }
            catch (AggregateException e)
            {
                AmazonS3Exception ex = (AmazonS3Exception)e.InnerException;
                if (ex.StatusCode == HttpStatusCode.NotFound)
                {
                    exists = false;
                }
                else
                {
                    throw;
                }
            }
        }

        public bool Exists => exists;

        public long Length
        {
            get
            {
                using (var stream = new MemoryStream(viewContent))
                {
                    return stream.Length;
                }
            }
        }

        public string PhysicalPath => null;

        public string Name => Path.GetFileName(key);

        public DateTimeOffset LastModified => lastModified;

        public bool IsDirectory => false;

        public Stream CreateReadStream()
        {
            return new MemoryStream(viewContent);
        }
    }

}

As you can see, all references to a static S3Client have been removed and they are exclusively in using statements. It appears a little slower than the previous implementation, but hopefully it will be stable.

EDIT: As it turns out, after the same amount of time the web server failed again with the same error message. Any time S3client was used, it was also disposed of.

ashishdhingra commented 3 years ago

Hi @tyler-shuhnicki,

Could you also please try wrapping GetObjectMetadataResponse view = client.GetObjectMetadataAsync(bucket, filter.TrimStart('/')).Result; in S3FileProvider.Watch() method in a using block?

Thanks, Ashish

tyler-shuhnicki commented 3 years ago

Hi @ashishdhingra,

I thought the same - but, GetObjectMetadataResponse does not implement IDisposable and can't be wrapped in a using block.

Thanks again, Tyler

ashishdhingra commented 3 years ago

Hi @tyler-shuhnicki,

Thanks for your response. Could you also please the complete stack trace as the one mentioned in the issue doesn't indicate the execution caused by S3 operation? I'm not sure what are the requirements of the ASP.NET IFileProvider interface and if we are missing some pattern here.

Thanks, Ashish

tyler-shuhnicki commented 3 years ago

Hi @ashishdhingra,

Sure. In the stack trace below, as far as I can tell, this is the same error three times - note the 3 different time stamps. 

Dec 30 18:34:55 web: #033[40m#033[32minfo#033[39m#033[22m#033[49m: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2]
Dec 30 18:34:55 web: Executed action Landing.Controllers.LandingController.Index (Anonymous.PluginModule) in 5492.7737ms
Dec 30 18:34:55 web: #033[40m#033[32minfo#033[39m#033[22m#033[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
Dec 30 18:34:55 web: Executed endpoint 'Landing.Controllers.LandingController.Index (Anonymous.PluginModule)'
Dec 30 18:34:55 web: #033[41m#033[30mfail#033[39m#033[22m#033[49m: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
Dec 30 18:34:55 web: An unhandled exception has occurred while executing the request.
Dec 30 18:34:55 web: System.Runtime.InteropServices.COMException (0x8007054F): An internal error occurred.
Dec 30 18:34:55 web: (0x8007054F)
Dec 30 18:34:55 web: at System.Runtime.Loader.AssemblyLoadContext.LoadFromStream(IntPtr ptrNativeAssemblyLoadContext, IntPtr ptrAssemblyArray, Int32 iAssemblyArrayLen, IntPtr ptrSymbols, Int32 iSymbolArrayLen, ObjectHandleOnStack retAssembly)
Dec 30 18:34:55 web: at System.Runtime.Loader.AssemblyLoadContext.InternalLoad(ReadOnlySpan`1 arrAssembly, ReadOnlySpan`1 arrSymbols)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Razor.RuntimeCompilation.RuntimeViewCompiler.CompileAndEmit(RazorCodeDocument codeDocument, String generatedCode)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Razor.RuntimeCompilation.RuntimeViewCompiler.CompileAndEmit(String relativePath)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Razor.RuntimeCompilation.RuntimeViewCompiler.OnCacheMiss(String normalizedPath)
Dec 30 18:34:55 web: --- End of stack trace from previous location ---
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Razor.Compilation.DefaultRazorPageFactoryProvider.CreateFactory(String relativePath)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Razor.RazorViewEngine.GetViewStartPages(String path, HashSet`1 expirationTokens)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Razor.RazorViewEngine.CreateCacheResult(HashSet`1 expirationTokens, String relativePath, Boolean isMainPage)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Razor.RazorViewEngine.OnCacheMiss(ViewLocationExpanderContext expanderContext, ViewLocationCacheKey cacheKey)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Razor.RazorViewEngine.LocatePageFromViewLocations(ActionContext actionContext, String pageName, Boolean isMainPage)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Razor.RazorViewEngine.FindView(ActionContext context, String viewName, Boolean isMainPage)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.ViewEngines.CompositeViewEngine.FindView(ActionContext context, String viewName, Boolean isMainPage)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor.FindView(ActionContext actionContext, ViewResult viewResult)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor.ExecuteAsync(ActionContext context, ViewResult result)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.ViewResult.ExecuteResultAsync(ActionContext context)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|29_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
Dec 30 18:34:55 web: --- End of stack trace from previous location ---
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
Dec 30 18:34:55 web: --- End of stack trace from previous location ---
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
Dec 30 18:34:55 web: #033[40m#033[32minfo#033[39m#033[22m#033[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Dec 30 18:34:55 web: Request finished HTTP/1.1 GET http://172.27.184.68/Landing - - - 500 - text/plain 5496.3953ms
Dec 30 18:34:55 web: #033[40m#033[1m#033[33mwarn#033[39m#033[22m#033[49m: Microsoft.AspNetCore.Server.Kestrel[0]
Dec 30 18:34:55 web: Connection processing ended abnormally.
Dec 30 18:34:55 web: System.InvalidOperationException: Handle is already used by another Socket.
Dec 30 18:34:55 web: at System.Net.Sockets.SocketAsyncEngine.RegisterCore(IntPtr socketHandle, SocketAsyncContext context)
Dec 30 18:34:55 web: at System.Net.Sockets.SocketAsyncEngine.RegisterSocket(IntPtr socketHandle, SocketAsyncContext context)
Dec 30 18:34:55 web: at System.Net.Sockets.SocketAsyncContext.Register()
Dec 30 18:34:55 web: at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.StartAsyncOperation(SocketAsyncContext context, TOperation operation, Int32 observedSequenceNumber, CancellationToken cancellationToken)
Dec 30 18:34:55 web: at System.Net.Sockets.SocketAsyncContext.ReceiveAsync(Memory`1 buffer, SocketFlags flags, Int32& bytesReceived, Action`5 callback, CancellationToken cancellationToken)
Dec 30 18:34:55 web: at System.Net.Sockets.SocketAsyncEventArgs.DoOperationReceive(SafeSocketHandle handle, CancellationToken cancellationToken)
Dec 30 18:34:55 web: at System.Net.Sockets.Socket.ReceiveAsync(SocketAsyncEventArgs e, CancellationToken cancellationToken)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketReceiver.WaitForDataAsync()
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.ProcessReceives()
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.DoReceive()
Dec 30 18:34:55 web: at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
Dec 30 18:34:55 web: at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
Dec 30 18:34:55 web: at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token)
Dec 30 18:34:55 web: at System.IO.Pipelines.Pipe.DefaultPipeReader.ReadAsync(CancellationToken cancellationToken)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.BeginRead(ValueTask`1& awaitable)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
Dec 30 18:34:55 web: at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
Dec 30 18:35:04 web: #033[40m#033[1m#033[33mwarn#033[39m#033[22m#033[49m: Microsoft.AspNetCore.Server.Kestrel[0]
Dec 30 18:35:04 web: Connection processing ended abnormally.
Dec 30 18:35:04 web: System.InvalidOperationException: Handle is already used by another Socket.
Dec 30 18:35:04 web: at System.Net.Sockets.SocketAsyncEngine.RegisterCore(IntPtr socketHandle, SocketAsyncContext context)
Dec 30 18:35:04 web: at System.Net.Sockets.SocketAsyncEngine.RegisterSocket(IntPtr socketHandle, SocketAsyncContext context)
Dec 30 18:35:04 web: at System.Net.Sockets.SocketAsyncContext.Register()
Dec 30 18:35:04 web: at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.StartAsyncOperation(SocketAsyncContext context, TOperation operation, Int32 observedSequenceNumber, CancellationToken cancellationToken)
Dec 30 18:35:04 web: at System.Net.Sockets.SocketAsyncContext.ReceiveAsync(Memory`1 buffer, SocketFlags flags, Int32& bytesReceived, Action`5 callback, CancellationToken cancellationToken)
Dec 30 18:35:04 web: at System.Net.Sockets.SocketAsyncEventArgs.DoOperationReceive(SafeSocketHandle handle, CancellationToken cancellationToken)
Dec 30 18:35:04 web: at System.Net.Sockets.Socket.ReceiveAsync(SocketAsyncEventArgs e, CancellationToken cancellationToken)
Dec 30 18:35:04 web: at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketReceiver.WaitForDataAsync()
Dec 30 18:35:04 web: at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.ProcessReceives()
Dec 30 18:35:04 web: at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.DoReceive()
Dec 30 18:35:04 web: at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
Dec 30 18:35:04 web: at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
Dec 30 18:35:04 web: at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token)
Dec 30 18:35:04 web: at System.IO.Pipelines.Pipe.DefaultPipeReader.ReadAsync(CancellationToken cancellationToken)
Dec 30 18:35:04 web: at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.BeginRead(ValueTask`1& awaitable)
Dec 30 18:35:04 web: at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
Dec 30 18:35:04 web: at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
Dec 30 18:35:10 web: #033[40m#033[1m#033[33mwarn#033[39m#033[22m#033[49m: Microsoft.AspNetCore.Server.Kestrel[0]
Dec 30 18:35:10 web: Connection processing ended abnormally.
Dec 30 18:35:10 web: System.InvalidOperationException: Handle is already used by another Socket.
Dec 30 18:35:10 web: at System.Net.Sockets.SocketAsyncEngine.RegisterCore(IntPtr socketHandle, SocketAsyncContext context)
Dec 30 18:35:10 web: at System.Net.Sockets.SocketAsyncEngine.RegisterSocket(IntPtr socketHandle, SocketAsyncContext context)
Dec 30 18:35:10 web: at System.Net.Sockets.SocketAsyncContext.Register()
Dec 30 18:35:10 web: at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.StartAsyncOperation(SocketAsyncContext context, TOperation operation, Int32 observedSequenceNumber, CancellationToken cancellationToken)
Dec 30 18:35:10 web: at System.Net.Sockets.SocketAsyncContext.ReceiveAsync(Memory`1 buffer, SocketFlags flags, Int32& bytesReceived, Action`5 callback, CancellationToken cancellationToken)
Dec 30 18:35:10 web: at System.Net.Sockets.SocketAsyncEventArgs.DoOperationReceive(SafeSocketHandle handle, CancellationToken cancellationToken)
Dec 30 18:35:10 web: at System.Net.Sockets.Socket.ReceiveAsync(SocketAsyncEventArgs e, CancellationToken cancellationToken)
Dec 30 18:35:10 web: at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketReceiver.WaitForDataAsync()
Dec 30 18:35:10 web: at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.ProcessReceives()
Dec 30 18:35:10 web: at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.DoReceive()
Dec 30 18:35:10 web: at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
Dec 30 18:35:10 web: at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
Dec 30 18:35:10 web: at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token)
Dec 30 18:35:10 web: at System.IO.Pipelines.Pipe.DefaultPipeReader.ReadAsync(CancellationToken cancellationToken)
Dec 30 18:35:10 web: at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.BeginRead(ValueTask`1& awaitable)
Dec 30 18:35:10 web: at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
Dec 30 18:35:10 web: at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)

I've noticed that the beginning of the issues mentions another location initially - I will post that code below. I have not mentioned it until now because that piece of code does not throw an exception if the code mentioned in the "Watch" function is not present. In fact, as far as I know the below code only runs initially on startup, and has no reason to be running. Currently, I have updated the code to use a singleton version of the S3 client and see if that will help. If it does not, perhaps the function below will provide some answers.

Startup.cs

var MVCBuilder = services.AddControllersWithViews()
                    .ConfigureApplicationPartManager(apm => LoadPlugInModules.load(bucketName, client).ForEach(m => apm.ApplicationParts.Add(m)))
                    .AddRazorRuntimeCompilation();

LoadPlugInModules.Load

public static List<AssemblyPart> load(string bucketName, IAmazonS3 client)
        {
            try
            {
                List<AssemblyPart> parts = new List<AssemblyPart>();
                ListObjectsRequest request = new ListObjectsRequest
                {
                    BucketName = bucketName,
                    Prefix = "Modules/"
                };

                ListObjectsResponse response = client.ListObjectsAsync(request).Result;

                foreach (S3Object obj in response.S3Objects)
                {
                    if (obj.Key.EndsWith(".dll"))
                    {
                        GetObjectRequest requestObj = new GetObjectRequest
                        {
                            BucketName = bucketName,
                            Key = obj.Key
                        };

                        using (GetObjectResponse responseObj = client.GetObjectAsync(requestObj).Result)
                        using (Stream responseStream = responseObj.ResponseStream)
                        using (StreamReader reader = new StreamReader(responseStream))
                        using (var streamReader = new MemoryStream())
                        {
                            responseStream.CopyTo(streamReader);
                            var assm = Assembly.Load(streamReader.ToArray());
                            parts.Add(new PlugInModule(assm));

                        }
                    }
                }
                return parts;
            }
            catch (Exception ex)
            {
                string t = ex.Message;
                return null;
            }
        }
github-actions[bot] commented 3 years ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

tyler-shuhnicki commented 3 years ago

@ashishdhingra Apologies, I closed the issue accidentally and have reopened.

tyler-shuhnicki commented 3 years ago

Posting here in case anyone finds themselves in this situation.

I've solved the issue of stability. Firstly, I had a misunderstanding - The function "watch" was not called every time a file was requested. Instead, the function "has changed" inside of the handler is called, meaning the code needed to be moved there.

Secondly, (seemingly a fundamental resolution), I threw it in a try catch block. Hopefully the catch is never being hit, but in the event it is a random error, it will be caught.

It also appears to be stable and fast while using one instance of S3client. See the working code below.

using System;
using System.Collections;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Net;
using System.Runtime.Caching;
using Amazon;
using Amazon.S3;
using Amazon.S3.Model;
using CFConnect.Models;
using Microsoft.AspNetCore.Mvc.Razor;
using Microsoft.Extensions.FileProviders;
using Microsoft.Extensions.Options;
using Microsoft.Extensions.Primitives;

namespace CFConnect.Extensions
{

    public class S3FileProvider : IFileProvider
    {
        //private string awsAccessKey;
        //private string awsSecretKey;
        private static IAmazonS3 client;
        private string bucket;
        private static Dictionary<string, DateTimeOffset> watchFiles;
        public S3FileProvider(string _awsAccessKey, string _awsSecretKey, string _bucket)
        {
            client = new AmazonS3Client(_awsAccessKey, _awsSecretKey, RegionEndpoint.USEast1);
            //awsAccessKey = _awsAccessKey;
            //awsSecretKey = _awsSecretKey;
            bucket = _bucket;
            watchFiles = new Dictionary<string, DateTimeOffset>();
        }

        public IDirectoryContents GetDirectoryContents(string subpath)
        {
            throw new NotImplementedException();
        }

        public IFileInfo GetFileInfo(string subpath)
        {
            if (string.IsNullOrEmpty(subpath))
                return new NotFoundFileInfo(subpath);

            //S3FileInfo s3File = new S3FileInfo(awsAccessKey, awsSecretKey, bucket, subpath);
            S3FileInfo s3File = new S3FileInfo(ref client, bucket, subpath);
            if (!watchFiles.ContainsKey(subpath))
            {
                //store the lastmodified date in a dictioanry, match it to the filename
                watchFiles.Add(subpath, s3File.LastModified);
            }
            else
            {
                watchFiles[subpath] = s3File.LastModified;
            }
            return s3File;
        }

        public IChangeToken Watch(string filter)
        {
            return new S3ChangeToken(ref client, ref watchFiles, bucket, filter);
        }

        public class S3ChangeToken : IChangeToken
        {
            private IAmazonS3 client;
            private string bucket, subpath;
            private Dictionary<string, DateTimeOffset> watchFiles;
            public S3ChangeToken(ref IAmazonS3 _client, ref Dictionary<string, DateTimeOffset> _watchFiles, string _bucket, string _subpath)
            {
                client = _client;
                watchFiles = _watchFiles;
                bucket = _bucket;
                subpath = _subpath;
            }
            public bool HasChanged
            {
                get
                {
                    try
                    {
                        DateTimeOffset time;
                        if (watchFiles.TryGetValue(subpath, out time))
                        {
                            if(time == default(DateTime))
                            {
                                //this is a false file.
                                //the file doesnt actually exist and we can just skip this. 
                                //see it tries to load a file for every possible search added in View Expanders, which is like 10 modules
                                //we save a lot of comparison time by skipping this
                                return false;
                            }
                            //get file info from S3
                            GetObjectMetadataResponse view = client.GetObjectMetadataAsync(bucket, subpath.TrimStart('/')).Result;
                            if (((DateTimeOffset)view.LastModified).CompareTo(time) != 0) //IF the last modified is LATER than the one we have (great than 0), trigger a change
                            {
                                return true;
                            }
                            else
                            {
                                return false; // it's up to date and does not need to be updated
                            }
                        }
                        else
                        {
                            return true; //file is not in collection; have it get a new file which should update last modified date
                        }
                    }
                    catch (Exception e)
                    {
                        //should this be true or no?
                        return true;
                    }
                }
            }
            public bool ActiveChangeCallbacks => false;

            public IDisposable RegisterChangeCallback(Action<object> callback, object state) => EmptyDisposable.Instance;

            internal class EmptyDisposable : IDisposable
            {
                public static EmptyDisposable Instance { get; } = new EmptyDisposable();
                private EmptyDisposable() { }
                public void Dispose() { }
            }
        }
    }

    public class S3FileInfo : IFileInfo
    {
        private string key;

        private bool exists;
        private byte[] viewContent;
        private DateTime lastModified;

        public S3FileInfo(ref IAmazonS3 client, string _bucket, string _subpath)
        {//pass s3client by ref to avoid making a ton of httpconnections
            this.key = _subpath;
            try
            {
                using (GetObjectResponse view = client.GetObjectAsync(_bucket, key.TrimStart('/')).Result)
                {
                    lastModified = view.LastModified;

                    using (var responseMem = new MemoryStream())
                    {
                        view.ResponseStream.CopyTo(responseMem);
                        viewContent = responseMem.ToArray();
                    }

                    exists = true;
                }

            }
            catch (AmazonS3Exception e)
            {
                if (e.StatusCode == HttpStatusCode.NotFound)
                {
                    exists = false;
                }
                else
                {
                    throw;
                }

            }
            catch (AggregateException e)
            {
                AmazonS3Exception ex = (AmazonS3Exception)e.InnerException;
                if (ex.StatusCode == HttpStatusCode.NotFound)
                {
                    exists = false;
                }
                else
                {
                    throw;
                }
            }
        }

        public bool Exists => exists;

        public long Length
        {
            get
            {
                using (var stream = new MemoryStream(viewContent))
                {
                    return stream.Length;
                }
            }
        }

        public string PhysicalPath => null;

        public string Name => Path.GetFileName(key);

        public DateTimeOffset LastModified => lastModified;

        public bool IsDirectory => false;

        public Stream CreateReadStream()
        {
            return new MemoryStream(viewContent);
        }
    }

}
github-actions[bot] commented 3 years ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.