search

Webreaper / Damselfly

Damselfly is a server-based Photograph Management app. The goal of Damselfly is to index an extremely large collection of images, and allow easy search and retrieval of those images, using metadata such as the IPTC keyword tags, as well as the folder and file names. Damselfly includes support for object/face detection.
GNU General Public License v3.0
1.47k stars 77 forks source link

Damselfly keeps crashing #528

Open gouthamravee opened 5 months ago

gouthamravee commented 5 months ago

Damselfly version: v4.1.3.0

docker compose 

services:
  damselfly:
    container_name: damselfly
    image: webreaper/damselfly
    network_mode: host
    environment:
      - TZ="America/New_York"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config
      - ./thumbs:/thumbs
      - /mnt/photography:/pictures
    restart: unless-stopped

I added my whole 1.8TB photography collection to damselfly and I keep seeing messages like what you see below. I have AI object detection and face recognition turned off. Thumbnail generation is running.

Damselfly is running in docker, inside a Debian based VM in proxmox. The VM has 20 cores and 15GB of ram assigned to it.

Do these errors just mean damselfly doesn't have the thumbnails for the images yet? Is that related to the 'killed' message, and what seems like a crash. Ram usage is very low, only about 9% of total ram is being actively used. 

damselfly  | [03:04:09.670-.NET TP Worker-ERR] Unable to process /thumb/Large/47038: Object reference not set to an instance of an object.
damselfly  | [03:04:09.674-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1910.dng: Object reference not set to an instance of an object.
damselfly  | [03:04:09.701-.NET TP Worker-ERR] Unable to process /thumb/Large/47490: Object reference not set to an instance of an object.
damselfly  | Killed
damselfly exited with code 137
damselfly  | Preparing to start Damselfly....
Webreaper commented 5 months ago

Are you able to attach the image it's processing when it fails? Or can you email it to me at info@damselfly.info?

Webreaper commented 5 months ago

I've pushed a potential fix for this (entirely speculative as I can't repro). Can you give it a go by pulling the dev version (pull webreaper/damselfly:dev) once the build has completed (here: https://github.com/Webreaper/Damselfly/actions/runs/8843240288)?

gouthamravee commented 5 months ago

@Webreaper I'm not sure which image its failing at, To make sure I send you the right one, in the logs I posted would it be the last image it couldn't find thumbnails for? 

damselfly  | [03:04:09.674-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1910.dng: Object reference not set to an instance of an object.
damselfly  | [03:04:09.701-.NET TP Worker-ERR] Unable to process /thumb/Large/47490: Object reference not set to an instance of an object.

This is another error I've seen often, I believe this happens when ever I refresh the web page for damesfly when its going through my reverse proxy. 

damselfly  | [14:14:16.507-.NET TP Worker-WRN] Failed to determine the https port for redirect.
damselfly  | [14:14:18.219-.NET TP Worker-ERR] An unhandled exception has occurred while executing the request.
damselfly  | System.InvalidOperationException: Unable to resolve service for type 'Microsoft.AspNetCore.ApiAuthorization.IdentityServer.IClientRequestParametersProvider' while attempting to activate 'Damselfly.Web.Server.Controllers.OidcConfigurationController'.
damselfly  |    at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.ThrowHelperUnableToResolveService(Type type, Type requiredBy)
damselfly  |    at lambda_method1131(Closure, IServiceProvider, Object[])
damselfly  |    at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass6_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)

I will try the dev version and let you know, there are still around 20k images left to process for thumbnails.

gouthamravee commented 5 months ago

With the dev version I'm not seeing a crash after about 15 minutes of it running. I am still seeing the errors about the thumbnails but the application doesn't seem to be getting killed. I'm also seeing thumbnails loading on the web page.

A bit more detail, I'm using a reverse proxy (nginx) to access damesfly through my public domain. It seems like something about that setup isn't working absolutely perfectly.

In addition, the path for all my photos to damselfly is a NFS share. The images aren't physically on the same hardware as damselfly. The images are on another server sitting right next to the server damselfly is running on. There's 4x 1Gbs network connections between the two systems.

Here's the log I'm seeing now. 

damselfly  | Preparing to start Damselfly....
damselfly  |   ./Damselfly.Web.Server /pictures --config=/config --thumbs=/thumbs
damselfly  | [14:22:05.995-0001-INF] === Damselfly Blazor Server Log Started ===
damselfly  | [14:22:06.317-0001-INF] Running Sqlite DB migrations...
damselfly  | [14:22:09.148-0001-INF] Running Sqlite DB optimisation...
damselfly  | [14:22:10.650-0001-INF] DB optimisation complete.
damselfly  | [14:22:10.863-0001-INF] Full-text search index rebuilt.
damselfly  | [14:22:10.875-0001-INF] Preloading config cache with all settings...
damselfly  | [14:22:10.965-0001-INF] Loaded 27 settings into config cache.
damselfly  | [14:22:11.096-0001-INF] Preloading config cache with all settings...
damselfly  | [14:22:11.211-0001-INF] Loaded 27 settings into config cache.
damselfly  | [14:22:11.873-0001-INF]   Executing: convert --version
damselfly  | [14:22:11.909-0001-INF] ImageMagick found: vVersion: ImageMagick 7.1.1-29 Q16-HDRI x86_64 21991 https://imagemagick.org
damselfly  | [14:22:11.937-0001-INF] Watermark font installed: /app/wwwroot/fonts/arial.ttf
damselfly  | [14:22:11.940-0001-INF] Created downloads folder: /app/wwwroot/downloads
damselfly  | [14:22:11.944-0001-INF] Task scheduler started.
damselfly  | [14:22:11.950-0001-INF] Registered job processing source: ThumbnailService
damselfly  | [14:22:11.952-0001-INF]   Executing: exiftool -ver
damselfly  | [14:22:12.139-0001-INF] ExifTool Version: v12.78
damselfly  |
damselfly  | [14:22:12.210-0001-INF] Registered job processing source: ExifService
damselfly  | [14:22:12.212-0001-INF] Warming up image cache with up to 2000 most recent images.
damselfly  | [14:22:15.503-0001-INF] Enriched and cached 2000 in 3238ms
damselfly  | [14:22:15.504-0001-INF] Image Cache primed with 2000 images.
damselfly  | [14:22:15.504-0001-INF] Started Work service thread.
damselfly  | [14:22:15.543-0001-INF] Registered job processing source: MetaDataService
damselfly  | [14:22:15.544-0001-INF] Registered job processing source: IndexingService
damselfly  | [14:22:17.733-0001-INF] Registered job processing source: ImageRecognitionService
damselfly  | [14:22:18.917-0001-INF] Adding scheduled task: CleanupThumbs every 1 week
damselfly  | [14:22:18.923-0001-INF] Adding scheduled task: CleanupDownloads every 6 hours
damselfly  | [14:22:18.923-0001-INF] Adding scheduled task: CleanupKeywordOps every 12 hours
damselfly  | [14:22:18.923-0001-INF] Adding scheduled task: DumpPerformance every 1 day
damselfly  | [14:22:18.924-0001-INF] Startup complete. Reducing console logging level to [Warning/Error].
damselfly  | [14:22:18.963-0001-WRN] No XML encryptor configured. Key {f16ca4a0-534b-43eb-bf85-b30109a47de0} may be persisted to storage in unencry
damselfly  | [14:22:48.180-.NET TP Worker-WRN] Failed to determine the https port for redirect.
damselfly  | [14:36:20.852-.NET TP Worker-ERR] An unhandled exception has occurred while executing the request.
damselfly  | System.InvalidOperationException: Unable to resolve service for type 'Microsoft.AspNetCore.ApiAuthorization.IdentityServer.IClientRequestParametersProvider' while attempting to activate 'Damselfly.Web.Server.Controllers.OidcConfigurationController'.
damselfly  |    at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.ThrowHelperUnableToResolveService(Type type, Type requiredBy)
damselfly  |    at lambda_method1141(Closure, IServiceProvider, Object[])
damselfly  |    at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass6_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
damselfly  | [14:36:32.515-.NET TP Worker-ERR] An unhandled exception has occurred while executing the request.
damselfly  | System.InvalidOperationException: Unable to resolve service for type 'Microsoft.AspNetCore.ApiAuthorization.IdentityServer.IClientRequestParametersProvider' while attempting to activate 'Damselfly.Web.Server.Controllers.OidcConfigurationController'.
damselfly  |    at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.ThrowHelperUnableToResolveService(Type type, Type requiredBy)
damselfly  |    at lambda_method1141(Closure, IServiceProvider, Object[])
damselfly  |    at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass6_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
damselfly  | [14:36:53.552-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1937-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:36:53.580-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1920-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:36:53.590-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-21/_DSC1957-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:36:53.593-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1926-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:36:53.594-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1904-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:36:53.636-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1914-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:36:53.946-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-21/_DSC2181.dng: Object reference not set to an instance of an object.
damselfly  | [14:37:00.083-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-21/_DSC2179.dng: Object reference not set to an instance of an object.
damselfly  | [14:37:00.099-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1897-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:39:05.172-.NET TP Worker-WRN] HTTP Exception: TypeError: NetworkError when attempting to fetch resource. (/api/config)
damselfly  | [14:39:05.173-.NET TP Worker-WRN]   Inner Exception: TypeError: NetworkError when attempting to fetch resource.
damselfly  | [14:39:05.174-.NET TP Worker-WRN] HTTP Exception: TypeError: NetworkError when attempting to fetch resource. (/api/config)
damselfly  | [14:39:05.174-.NET TP Worker-WRN]   Inner Exception: TypeError: NetworkError when attempting to fetch resource.
damselfly  | [14:38:25.708-.NET TP Worker-ERR] An unhandled exception has occurred while executing the request.
damselfly  | System.InvalidOperationException: Unable to resolve service for type 'Microsoft.AspNetCore.ApiAuthorization.IdentityServer.IClientRequestParametersProvider' while attempting to activate 'Damselfly.Web.Server.Controllers.OidcConfigurationController'.
damselfly  |    at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.ThrowHelperUnableToResolveService(Type type, Type requiredBy)
damselfly  |    at lambda_method1141(Closure, IServiceProvider, Object[])
damselfly  |    at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass6_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
damselfly  | [14:38:36.299-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1937-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:36.301-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1926-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:36.425-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1920-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:37.114-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1914-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:37.241-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-21/_DSC1957-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:37.243-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1904-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:37.525-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-21/_DSC2179.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:37.887-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1897-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:38.799-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-21/_DSC2181.dng: Object reference not set to an instance of an object.

The HTTP and System.InvalidOperationException: Unable to resolve service for type errors happen when ever I refresh the web page for damselfly.

Weird thing now is even though thumbnails are loading, if I try to open the image by double clicking I get an "image not found" error.

Edit:

Updated too soon, it did end up crashing again. 

damselfly  | [14:38:37.887-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1897-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:38.799-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-21/_DSC2181.dng: Object reference not set to an instance of an object.
damselfly  | Killed
damselfly exited with code 137
damselfly  | Preparing to start Damselfly....
damselfly  |   ./Damselfly.Web.Server /pictures --config=/config --thumbs=/thumbs
damselfly  | [14:49:14.046-0001-INF] === Damselfly Blazor Server Log Started ===
damselfly  | [14:49:14.434-0001-INF] Running Sqlite DB migrations...
damselfly  | [14:49:17.049-0001-INF] Running Sqlite DB optimisation...

Another update, sorry for all these edits.

This time it looks like some resource usage caused the whole VM to lock up, when I started damsefly up again I saw a new error. 

damselfly  | [15:36:44.108-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2022/2022-12-22/_DSC6513.dng: Object reference not set to an instance of an object.
damselfly  | [15:36:44.337-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2022/2022-12-22/_DSC6523.dng: Object reference not set to an instance of an object.
damselfly  | [15:42:08.982-Kestrel Timer-WRN] As of "04/26/2024 15:38:57 +00:00", the heartbeat has been running for "00:00:02.1575431" which is longer than "00:00:01". This could be caused by thread pool starvation.
damselfly  | [15:42:13.369-.NET TP Worker-ERR] Conversion failed. Unable to start process: Failed
damselfly  | [15:42:13.378-.NET TP Worker-ERR] Failed commandline was: convert  -define jpeg:size=800x800 "/pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2022/2022-03-12/_DSC9872-Pano-Edit.tif" -quality 90 -unsharp 0.5x0.5+1.25+0.0 -thumbnail 800x800 -auto-orient -write "/thumbs/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2022/2022-03-12/_DSC9872-Pano-Edit_l.JPG"  -thumbnail 320x320 -auto-orient -write "/thumbs/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2022/2022-03-12/_DSC9872-Pano-Edit_m.JPG"   -extent 120:120 -gravity center -thumbnail 120x120 -auto-orient "/thumbs/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2022/2022-03-12/_DSC9872-Pano-Edit_s.JPG"
Webreaper commented 5 months ago

Not sure how much I can help with the proxy thing. Never had that reported, and I won't be able to repro.

For the crash, can you send me _DSC1897-HDR.dng, _DSC2181.dng, _DSC6523.dng and _DSC6523.dng so I can test why they're causing the crash? (or just send one, and then I can see if i can repro).

gouthamravee commented 5 months ago

@Webreaper

No worries on the reverse proxy thing, I can troubleshoot that on my own.

I've emailed you the requested pictures, its only 3 so I don't mind sending you more data to test with.

gouthamravee commented 5 months ago

After a few hours it seems like a memory leak crashed most of the VM. The last logs in netdata show a spike in memory usage, and proxmox shows a ton of disk writes before the crash.

The last logs are 

17:49:59.710    WRN     As of ""2024-04-26T17:49:52.0714653+00:00"", the heartbeat has been running for ""00:00:01.1135167"" which is longer than ""00:00:01"". This could be caused by thread pool starvation.
17:48:50.015    ERR     System.Net.Http.HttpRequestException: net_http_message_not_success_statuscode_reason, 504, Gateway Timeout
17:48:50.011    WRN     HTTP Exception: net_http_message_not_success_statuscode_reason, 504, Gateway Timeout (/api/data/stats)
17:48:41.145    ERR     System.Net.Http.HttpRequestException: net_http_message_not_success_statuscode_reason, 504, Gateway Timeout
17:48:41.142    WRN     HTTP Exception: net_http_message_not_success_statuscode_reason, 504, Gateway Timeout (/api/logs/0)
17:48:36.496    ERR     System.Net.Http.HttpRequestException: net_http_message_not_success_statuscode_reason, 504, Gateway Timeout
17:48:36.488    WRN     HTTP Exception: net_http_message_not_success_statuscode_reason, 504, Gateway Timeout (/api/data/newversion)
17:48:35.325    WRN     As of ""2024-04-26T17:46:32.6490334+00:00"", the heartbeat has been running for ""00:00:01.5247904"" which is longer than ""00:00:01"". This could be caused by thread pool starvation.
17:11:42.641    ERR     Failed commandline was: convert -define jpeg:size=800x800 "/pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2021/2021-04-03/_DSC8841.ARW" -quality 90 -unsharp 0.5x0.5+1.25+0.0 -thumbnail 800x800 -auto-orient -write "/thumbs/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2021/2021-04-03/_DSC8841_l.JPG" -thumbnail 320x320 -auto-orient -write "/thumbs/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2021/2021-04-03/_DSC8841_m.JPG" -extent 120:120 -gravity center -thumbnail 120x120 -auto-orient "/thumbs/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2021/2021-04-03/_DSC8841_s.JPG"
17:11:42.641    ERR     Conversion failed. Unable to start process: Failed
17:11:15.317    INF     Full index compelete.
17:02:09.474    INF     Performing full index.

Tried this again with 24GB of ram to the VM and same thing happened.

gouthamravee commented 5 months ago

@Webreaper any luck in recreating this?

Webreaper commented 5 months ago

Sorry, stuff cropped up so haven't had time. Still on my list to investigate.

gouthamravee commented 5 months ago

No worries, let me know if you need anything else when you get to it. Thank you.

gouthamravee commented 4 months ago

@Webreaper Throwing more RAM at it seems to have solved the issue. I gave the VM 24GB and added another 64GB to the host so there's more room for all the VMs running there. Damselfly seems to have finished all the indexing tasks for the 7000 new images I've added since the issue started.

I'm letting it do the AI processing now.

Webreaper commented 4 months ago

It's a bit hungry. 😁