During a stress test we surfaced that the ingest module is the current bottleneck for ingesting large amounts of camera data.
The process ingested ~1gb of data before failing to populate the inference queue due to the ingest lambda running out of space on disk. As lambas re-use an execution environment during heavy load, this causes the lamba to eventually fall over.
Actions
Ensure all disk actions use a TemporaryDirectory path location so that the directory is removed after the execution finishes (either successful or error state)
Context
During a stress test we surfaced that the ingest module is the current bottleneck for ingesting large amounts of camera data.
The process ingested ~1gb of data before failing to populate the inference queue due to the ingest lambda running out of space on disk. As lambas re-use an execution environment during heavy load, this causes the lamba to eventually fall over.
Actions
Ensure all disk actions use a TemporaryDirectory path location so that the directory is removed after the execution finishes (either successful or error state)
cc/ @rbavery @geohacker @nathanielrindlaub