Closed steentottrup closed 3 years ago
steentottrup
commented
3 years ago Downloading and installting the 'Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017 and 2019' on the server seems to fix the problem. I would have like to know that before throwing away 3-4 hours during the weekend.
michaelgsharp
commented
3 years ago Hey @steentottrup, apologies for the issues you faced. We are thinking that probably has something to do with the specific packages that were used, probably either tensorflow or the vision package. Could you provide a sample project with a simple pipeline and the same packages so we can look into it further? Also, were those redistributables installed as 3 separate packages? Or just 1 package that contained them all?
luisquintanilla
commented
3 years ago I tried publishing this .NET 5 Web API in Windows Server 2016 and did not run into issues. The VM was clean (no VS or Microsoft Visual C++ Redist). Only the .NET 5 SDK. Note that this solution does not use TensorFlow.NET or Microsoft.ML.Vision.
https://github.com/luisquintanilla/MinimalMLNetWebApi/tree/net50
mstfbl
commented
3 years ago @steentottrup I'm closing this issue as we were not able to reproduce your issue and as you've not answered @michaelgsharp's query. Please feel free to reopen the issue in the future.
System information
Issue
What did you do? I'm building an ASP.NET Core Web Application, Razor Pages and Api controllers. I'm using ML.NET, with ML.NET Vision, ML.NET ImageAnalytics and Tensorflow (SciSharp.TensorFlow.Redist). The solution has been build on AppVeyor (with MSBuild). I've tried doing a
dotnet publish -r win-x64 --self-contained falseanddotnet publish -r win-x64 --self-contained trueanddotnet publish -r win-x64but none of it has worked. The tensorflow dll is included in the publish folder, so I'm guessing it's one of it's dependencies that are missing.What happened? Everything is working perfectly on my developer machine, IIS EXpress, Visual Studio 2019, etc. When trying to deploy the finished solution to a server, the ASP.NET Core Web Application runs as expected, but once I hit the endpoint that uses ML.NET, it fails.
What did you expect? I was expecting the web application to work on the server running on IIS , as it is locally.
Source code / logs
2020-11-29 19:22:10.030 +01:00 [Error] An unhandled exception has occurred while executing the request. System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.DllNotFoundException: Unable to load DLL 'tensorflow' or one of its dependencies: The specified module could not be found. (0x8007007E) at Tensorflow.c_api.TF_NewGraph() at Tensorflow.Graph..ctor() at Microsoft.ML.TensorFlow.TensorFlowUtils.LoadTFSession(IExceptionContext ectx, Byte[] modelBytes, String modelFile) at Microsoft.ML.Vision.ImageClassificationModelParameters..ctor(IHostEnvironment env, ModelLoadContext ctx) at Microsoft.ML.Vision.ImageClassificationModelParameters.Create(IHostEnvironment env, ModelLoadContext ctx) --- End of inner exception stack trace --- at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions) at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters) at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs) at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstance(IHostEnvironment env, Object args, Object[] extra) at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes](IHostEnvironment env, Type signatureType, TRes& result, String name, String options, Object[] extra) at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes,TSig](IHostEnvironment env, TRes& result, String name, String options, Object[] extra) at Microsoft.ML.ModelLoadContext.TryLoadModelCore[TRes,TSig](IHostEnvironment env, TRes& result, Object[] extra) at Microsoft.ML.ModelLoadContext.TryLoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra) at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra) at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra) at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, String name, Object[] extra) at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, String name, Object[] extra) at Microsoft.ML.Data.MulticlassPredictionTransformer.Create(IHostEnvironment env, ModelLoadContext ctx) --- End of inner exception stack trace --- at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions) at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters) at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs) at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstance(IHostEnvironment env, Object args, Object[] extra) at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes](IHostEnvironment env, Type signatureType, TRes& result, String name, String options, Object[] extra) at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes,TSig](IHostEnvironment env, TRes& result, String name, String options, Object[] extra) at Microsoft.ML.ModelLoadContext.TryLoadModelCore[TRes,TSig](IHostEnvironment env, TRes& result, Object[] extra) at Microsoft.ML.ModelLoadContext.TryLoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra) at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra) at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra) at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, String name, Object[] extra) at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, String name, Object[] extra) at Microsoft.ML.Data.TransformerChaingAwaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.gAwaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
1..ctor(IHostEnvironment env, ModelLoadContext ctx) at Microsoft.ML.Data.TransformerChain.Create(IHostEnvironment env, ModelLoadContext ctx) --- End of inner exception stack trace --- at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions) at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters) at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs) at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstance(IHostEnvironment env, Object args, Object[] extra) at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes](IHostEnvironment env, Type signatureType, TRes& result, String name, String options, Object[] extra) at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes,TSig](IHostEnvironment env, TRes& result, String name, String options, Object[] extra) at Microsoft.ML.ModelLoadContext.TryLoadModelCore[TRes,TSig](IHostEnvironment env, TRes& result, Object[] extra) at Microsoft.ML.ModelLoadContext.TryLoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra) at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra) at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra) at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, String name, Object[] extra) at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, String name, Object[] extra) at Microsoft.ML.Data.TransformerChain1..ctor(IHostEnvironment env, ModelLoadContext ctx) at Microsoft.ML.Data.TransformerChain.Create(IHostEnvironment env, ModelLoadContext ctx) --- End of inner exception stack trace --- at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions) at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters) at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs) at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstance(IHostEnvironment env, Object args, Object[] extra) at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes](IHostEnvironment env, Type signatureType, TRes& result, String name, String options, Object[] extra) at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes,TSig](IHostEnvironment env, TRes& result, String name, String options, Object[] extra) at Microsoft.ML.ModelLoadContext.TryLoadModelCore[TRes,TSig](IHostEnvironment env, TRes& result, Object[] extra) at Microsoft.ML.ModelLoadContext.TryLoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra) at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra) at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra) at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra) at Microsoft.ML.ModelOperationsCatalog.Load(Stream stream, DataViewSchema& inputSchema) at Microsoft.ML.ModelOperationsCatalog.Load(String filePath, DataViewSchema& inputSchema) at BrickId.ObjectDetection.Torso.TorsoPrediction.Predict(ModelInput input) in C:\projects\cmd\brickid\src\BrickId.ObjectDetection.Torso\TorsoPrediction.cs:line 19 at BrickId.Core.CQS.Queries.Preditions.Torso.GetTorsoPreditionQueryHandler.HandleAsync(GetTorsoPreditionQuery query, CancellationToken cancellationToken) in C:\projects\cmd\brickid\src\BrickId.Core\CQS\Queries\Predictions\Torso\GetTorsoPreditionQueryHandler.cs:line 44 at CreativeMinds.CQS.Permissions.GenericPermissionCheckAsyncQueryHandlerDecorator2.HandleAsync(TQuery query, CancellationToken cancellationToken) at CreativeMinds.CQS.Validators.GenericValidationAsyncQueryHandlerDecorator2.HandleAsync(TQuery query, CancellationToken cancellationToken) at lambda_method386(Closure , Object ) at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.