tnc-ca-geo / animl-ingest

Lambda function for processing camera trap images
Other
0 stars 1 forks source link

Double-check that non .zip, .jpg, and .png files are handled correctly in bulk upload #48

Closed nathanielrindlaub closed 1 year ago

nathanielrindlaub commented 1 year ago

Hey @ingalls,

I tried zipping up a .mp4 file and it looked like from the logs it gets mistakenly handled as a batch once it's extracted from zip and passed back to image-ingest:

  1. First it got unzipped as designed by ingest-zip: https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:log-groups/log-group/$252Faws$252Fbatch$252Fjob/log-events/animl-ingest-dev-batch-job$252Fdefault$252F72974e0a66014796837a6bf7fe21c3d9
  2. then the .mp4 got put back in the ingestion s3 bucket, triggering image-ingest again: https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:log-groups/log-group/$252Faws$252Flambda$252FIngestImage-dev/log-events/2023$252F04$252F25$252F$255B$2524LATEST$255Dcf125d6499784f7a9bae50996d0ee4d2$3Fstart$3D1682398323769$26refEventId$3D37518736343037570017021690359944626557452613983480971266
  3. Then ingest-zip gets invoked with just the raw .mp4 again and it fails: https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:log-groups/log-group/$252Faws$252Fbatch$252Fjob/log-events/animl-ingest-dev-batch-job$252Fdefault$252Fb8d86705f37d4c5fbc7ba9637f2a2c1e

It also doesn't remove the files from the s3 bucket when it fails... I guess the preferred behavior would be to move files that fail for being the wrong type into a folder in the deal-letter bucket.

ingalls commented 1 year ago

I'll dig into this and work on a resolution.

ingalls commented 1 year ago

Screenshot from 2023-05-03 14-59-59

this should be working now