search

nrdg / cloudknot

A python library to run your existing code on AWS Batch
https://nrdg.github.io/cloudknot/
Other
71 stars 17 forks source link

batch jobs are failing when calling from AWS Lambda #186

Open ishanbhagawati opened 5 years ago

ishanbhagawati commented 5 years ago

Hi I have created a Batch job using cloudknot and then I want to periodically call it after every 1 hour. For this I am using a AWS Lambda that submits the job. However after submitting my jobs are failing with the following error usage: fileName.py [-h] [--starmap] [--arrayjob] [--sse {AES256,aws:kms}] bucket analyse-conversation.py: error: the following arguments are required: bucket

I have also tried mentioning parameters "CLOUDKNOT_JOBS_S3_BUCKET" and "CLOUDKNOT_S3_JOBDEF_KEY" while submitting the jobs from lambda.

I looked into the code I guess its something has to do with "script.template" file . Can you help me to fix this issue

welcome[bot] commented 5 years ago

👋 Thanks for opening your first issue here! We appreciate your help making Cloudknot better.

arokem commented 5 years ago

Sorry for the slow response here. I am not sure I understand where this error appears. Is that the traceback from within your Lambda job?

ishanbhagawati commented 4 years ago

@arokem no, it appears in my failed batch jobs logs. I have found a way around though. What I have found out is that the batch jobs that I create using cloudknot are dependent on the file input.pickle CLOUDKNOT_JOBS_S3_BUCKET . So for the time being what I am doing is copying this pickle file to the new job folder in s3 and then submit the job.

But while this works this seems kind of hack