How to properly handle failed SSM parameter fetching

[thenano]

Hi,

First of all thank you very much for the work put into this, it was a crucial component of moving some of my workloads into lambda, and it works like a charm! I've recently started experiencing intermittent lambda executions where my crypteia env variables don't get swapped out for the SSM secrets. I haven't dug into the root cause yet, but I assume something is failing when fetching the secrets from SSM, so I was wondering if there is some agreed upon way to handle these scenarios. My use case is running lamby rails jobs that get triggered via an sqs queue (via lambdakiq). Sometimes the jobs will execute and I'll get errors because I'll just have x-crypteia-ssm:.. in my envs instead of the secrets. To make matters worse, I'm pretty sure that when facing high loads lambda copies over my jobs and I get more failures because they don't seem to startup and try to load the vars again. I'm wondering if there's some way I can just kill the execution of the container straight away, and then my queue retry mechanisms will kick in and execute the job again? Any help here is greatly appreciated, and apologies if this isn't the right place to post this question.

Thanks!

[jeremiahlukus]

I think this should do the same as ssm and fail on deploy if unable to fetch a param.

[thenano]

@jeremiahlukus wow thanks so much for the fix! is there change needed to our dockerfiles to get this update?

[metaskills]

There will be but the build systems have to get fixed and we will need to do a preview release with testing to make sure everything works.

[jeremiahlukus]

@thenano you can follow https://github.com/rails-lambda/crypteia/issues/48

for updates. You can expect it to be tested and released this week though

[thenano]

Thank you so much @jeremiahlukus @metaskills