search

tnc-ca-geo / animl-ingest

Lambda function for processing camera trap images
Other
0 stars 1 forks source link

Look into SQS for messaging layer instead of posting to backend #7

Closed nathanielrindlaub closed 3 years ago

nathanielrindlaub commented 4 years ago

https://aws.amazon.com/sqs/

"here is another (really common) idea: you could have your lambda publish the messages it consumes off of s3 to a queue (https://aws.amazon.com/sqs/) then your express serbice can just poll that queue for messages This is a really common pattern because its (a) super flexible, you can have multiple consumers of the queue (with some configuration, using sns and multiple queues) (b) it doesnt slam your express server in the case of a spike" - ethan

postfalk commented 4 years ago

Yeah, a message queue is a good idea. I using message queues in a lot of my apps. But I did not use the AWS service yet. I usually use Celery which is a Python abstraction for many message queues, I am using it with rabbitmq. However, that is a lot of complexity we acquiring all at once. I guess it would be better to focus on a detail like the Lambda function to get it right and keep the environment in which it operates to a minimum for now.

I hope we could meet for an architectural discussion.

nathanielrindlaub commented 4 years ago

I agree yeah this is always something we could add later.

Sure thing on architecture meeting. I have no meetings at all today or tomorrow. LMK when works for you.

postfalk commented 4 years ago

I think, it needs me to study what you did so far. Can you document and push to Github. Then we should talk about it next week. And I am not sure about the format, in person would be definitely the easiest. Did you see anything about virtual white-boarding?

On the other hand, always happy to hop on a call to chat but it will probably be all over the place.

nathanielrindlaub commented 4 years ago

Ok sounds good. Everything is up to date on github. There are a handful of related repos:

node script running on pi - https://github.com/tnc-ca-geo/animl-base lambda function - https://github.com/tnc-ca-geo/animl-lambda front and back end app - https://github.com/tnc-ca-geo/animl sagemaker related stuff - https://github.com/tnc-ca-geo/animl-ml

So far everything I've done maps to what I outlined in the diagram I made. Where it all currently stands is: I've got a camera deployed in my back alley, I've got the buckeye base server software and node script running on the raspberry pi and pushing images up to s3 as they come in, the lambda function is extracting metadata when they get added to s3, and the sagemaker endpoint with megadetector on it is working and ready for inference requests.

The next push is to get the express app a little more fleshed out (routing the post requests coming from lambda, saving image metadata documents, possibly working on making inference requests to sagemaker when new images come in and appending the mongoDB records with object bounding boxes... that kind of thing). I'm going to spend some time today and tomorrow on that stuff.

I haven't heard of virtual white-boarding. What is that?

postfalk commented 4 years ago

Wow Natty, you are really kicking it. Or maybe I am so ineffective these days. I don't know what virtual white-boarding is but I wish I would. Let me study the application tomorrow and discuss how to proceed on Monday. Looking forward to see your backyard. Any interesting animls yet?

nathanielrindlaub commented 4 years ago

Haha so far just a lot of shots of the neighbor's shins. But it's only been there for less than 48 hours so hopefully I'll get something good one of these days.

nathanielrindlaub commented 3 years ago

Closing out for now. Might revisit and reopen down the road.