search

wibd-ls / wibd-ls1

WiBD Learn and Share
1 stars 1 forks source link

Explore AWS Batch #12

Closed mcandrews17 closed 6 years ago

mcandrews17 commented 6 years ago

Look into applying AWS batch to Rekognition, especially for doing a portion of the images at a time and picking up where we left off.

mcandrews17 commented 6 years ago

AWS Batch looks like a good option. I will need to do more research into the mechanics of Batch, but this article is almost exactly what we're trying to do, and even discusses indexing the text in the images using Elasticsearch with Kibana (this could be useful for the business owners). It also addresses the concern about knowing which images we have already run because the Lambda is triggered to run only on images newly added to the S3 bucket.

Gunjan-Sharma commented 6 years ago

This is awesome and exactly what we need. Just have to package the Taasseract API in the application. Thanks for finding it! Have you played with setting it up yet? Would be good to know the costs of using each of these before we try the pipeline.

Gunjan

Sent from my iPhone

On Aug 9, 2018, at 8:48 PM, mcandrews17 notifications@github.com wrote:

AWS Batch looks like a good option. I will need to do more research into the mechanics of Batch, but this article is almost exactly what we're trying to do, and even discusses indexing the text in the images using Elasticsearch with Kibana (this could be useful for the business owners). It also addresses the concern about knowing which images we have already run because the Lambda is triggered to run only on images newly added to the S3 bucket.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

Gunjan-Sharma commented 6 years ago

Megan - We talked about this today. We are thinking of leaving it simple and just use a Lambda trigger on new file arrival in S3. If that works, let's not invest in this architecture.

mcandrews17 commented 6 years ago

Sorry—was not receiving notifications of your comments for some reason. I haven't played with setting it up yet, but it does seem like Kanan has been able to get the pipeline working on Lambda, and we can implement a trigger for file arrival in s3. I will close this issue and continue working with her. We can circle back to AWS Batch in the future if need be.