search

ncihtan / IDP

1 stars 0 forks source link

Consider modifying lambda to trigger after metadata submission #22

Open elysekozlowski opened 3 years ago

ychae commented 3 years ago

@milen-sage this was something that came out of the 2/19 IDP meeting -- would you be able to think it over and let us know if this something we should include in the first data release process?

milen-sage commented 3 years ago

We are considering what's the best way to trigger the image pre-processing script and where the latter would be hosted, since the pre-processing computation will not be executed in the lambda function itself (the lambda function is too resource-constrained). There are different options including AWS Batch and @clarisse-lau is investigating them.

Depending on the option she picks, we can decide on what set of conditions would actually trigger the pre-processing computation (e.g. story file and image file are submitted; and/or metadata manifest file is submitted). For now we are just considering triggering computation when both required inputs to the pre-processing script are provided: image + story file.