search

WildMeOrg / scout

MIT License
6 stars 5 forks source link

NAS interactions slow for Image ingestion and ML processing #50

Closed TanyaStere42 closed 7 months ago

TanyaStere42 commented 1 year ago

NAS mounted over 2.5GBE connection to /mnt/nas Tested image ingestion of 100 JPGs at around 10MB/image Images ingest at ~24/minute - 100K images would take around 3 days. Ingestion seems to use only one core.

NAS mounted over 2.5GB connection. Annotation of 10 images with model v2. The 256x256 image chips are created on the NAS but this takes almost two minutes to create the 1,717 chips, and then another ~35s to process them. Processing 10 images took over 20 minutes (so 100K images would take 6 months)

Seems like the image chips should, at least, be processed on the internal drive, NOT the NAS?

simbamangu commented 7 months ago

This is partially fixed @TanyaStere42 because the 255x chips are now processed in-memory, but the image ingestion is still super slow. Opening a new issue with more description and testing of image ingestion.