Large amounts of images causes crash

[mist0706]

Describe the bug I have been using zoneminder for quite a while and due to the structure of folders where the images are places i am forced to glob a large amount of files (900k+). When starting the trigger container it seems it has to go through all the files and store some of this data in memory. After reaching about 8GB the container is OOM killed.

Log entries

2021-03-05T09:24:34+01:00 [Main] ****************************************
2021-03-05T09:24:34+01:00 [Main] Up and running!

<--- Last few GCs --->

[1:0x593c3b0]   174657 ms: Scavenge 4032.2 (4123.6) -> 4026.4 (4124.9) MB, 24.6 / 0.1 ms  (average mu = 0.684, current mu = 0.415) allocation failure
[1:0x593c3b0]   174720 ms: Scavenge 4035.8 (4125.4) -> 4031.4 (4132.4) MB, 11.7 / 0.0 ms  (average mu = 0.684, current mu = 0.415) allocation failure
[1:0x593c3b0]   176549 ms: Mark-sweep 4043.2 (4132.9) -> 4021.5 (4136.9) MB, 1764.6 / 10.5 ms  (average mu = 0.527, current mu = 0.218) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
 1: 0xa747f0 node::Abort() [node]
 2: 0x9a1c4d node::FatalError(char const*, char const*) [node]
 3: 0xc5d08e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xc5d407 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xe26c95  [node]
 6: 0xe56223 v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [node]
 7: 0xe62dd6 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [node]
 8: 0xe4eeff v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [node]
 9: 0xe4f178 v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [node]
10: 0xe40f99 v8::internal::ItemParallelJob::Run() [node]
11: 0xe64d60 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [node]
12: 0xe655a3 v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [node]
13: 0xe65965 v8::internal::MarkCompactCollector::Evacuate() [node]
14: 0xe780d1 v8::internal::MarkCompactCollector::CollectGarbage() [node]
15: 0xe33f98 v8::internal::Heap::MarkCompact() [node]
16: 0xe35928 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
17: 0xe38ecc v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
18: 0xdfcb2a v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
19: 0xdf84f2 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawArray(int, v8::internal::AllocationType) [node]
20: 0xdf85a4 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller(v8::internal::Handle<v8::internal::Map>, int, v8::internal::Handle<v8::internal::Oddball>, v8::internal::AllocationType) [node]
21: 0x103cfde v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Allocate(v8::internal::Isolate*, int, v8::internal::AllocationType) [node]
22: 0x103d176 v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Rehash(v8::internal::Isolate*, v8::internal::Handle<v8::internal::OrderedHashMap>, int) [node]
23: 0x1138fa8 v8::internal::Runtime_MapGrow(int, unsigned long*, v8::internal::Isolate*) [node]
24: 0x14ea8f9  [node]
2021-03-05T09:27:21+01:00 [Main] ****************************************

Installation details

• OS:Linux durian 5.8.0-38-generic #43~20.04.1-Ubuntu SMP Tue Jan 12 16:39:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux docker-compose.yml:

  
  version: "3.8"

services: trigger: container_name: trigger volumes:

Change d:/myfolder/myimages to point to the folder that will have the images

  # to analyze. Only change the local path that is before the :/aiinput portion.
  # Don't change the :/aiinput part. For example, if you are on Windows and your
  # images are stored locally in d:/blueiris/capturedImages your final line should
  # look like this:
  # d:/bleueIris/capturedImages:/aiinput
  - /mnt/ssd-vol/zoneminder/ai:/aiinput

environment:
  # Change this to match the timezone the images are produced in,
  # Typically this will be the timezone of the machine running
  # the Docker container. For a list of valid timezone values
  # see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones.
  # The value to use is in the "TZ database name" column.
  - TZ=Europe/Stockholm

ports:
  # This port is used by the local web server when annotated images are enabled.
  # If you change the port used by the local web server in the settings.json file
  # this also has to change to match.
  - 4242:4242

# ------------------------------------------------------------------------
# Don't change anything below this line unless you know what you are doing
secrets:
  - triggers
  - settings
image: danecreekphotography/node-deepstackai-trigger:latest
restart: always
depends_on:
  - deepstack-ai

deepstack-ai: container_name: deepstack-ai image: deepquestai/deepstack:gpu runtime: nvidia restart: always ports:

• 5000:5000

  deploy:

  #      resources:
  #        reservations:
  #          devices:
  #          - driver: nvidia
  #            count: 1
  #            capabilities: [gpu, utility]

  volumes:

• /opt/deepstack:/datastore environment:

• VISION-DETECTION=True

  deepstack_ui: container_name: deepstack-ui restart: unless-stopped image: robmarkcole/deepstack-ui environment:

• DEEPSTACK_IP=10.7.1.2
• DEEPSTACK_PORT=5000
• DEEPSTACK_API_KEY=""
• DEEPSTACK_TIMEOUT=20
• DEEPSTACK_CUSTOM_MODEL=fire
• DEEPSTACK_UI_DEBUG_MODE=True ports:
• '8501:8501'

volumes: localstorage:

secrets: settings:

This should point to the location of the settings.json configuration file

file: ./settings.json

triggers:

This should point to the location of the triggers.json configuration file

file: ./triggers.json

**Additional context**
The assumption is that the code creates a massive index on startup. My hope is that the code can be changed in such a way that it does not create a this index of files to determine if the file was created before the service was started.

[neilenns]

I'm not familiar with zoneminder but you'll need to configure it to automatically purge its drop folder to keep the number of images there to a reasonable amount.

See https://github.com/danecreekphotography/node-deepstackai-trigger/wiki/Troubleshooting#problem-the-container-causes-80-cpu-usage for what this configuration looks like when using BlueIris.