lithops-cloud / lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
http://lithops.cloud
Apache License 2.0
317 stars 105 forks source link

AWS Lambda function name #1210

Closed sergii-mamedov closed 10 months ago

sergii-mamedov commented 10 months ago

Is it possible to specify normal short names for lamda function? Currently, the auto-generated entities are very long and have redundant entities. Examle: lithops_v3-0-1_lhdm__metaspace-aws-lambda--301c_512MB.

This was originally designed like this because this was supporting some internal logic so that from the function name we were able to get the container name, the memory and the user. Not sure if this is still necessary, need to check. In other backends we switched from these kind of names to lithops-worker-xxxxxxx where xxxxx is the hash of the function name you see. Would this work for you? or what is for you a "normal short name"?

I usually create names of docker images like metaspace-aws-lambda:3.0.1 or metaspace-ibm-ce:3.0.1 where in the tag I store the current version of lithops. I'd like to have full freedom to provide names, but lithops-worker-xxxxxxxx works fine for me as well. Although I want to have separate lambda functions for different environments (production, staging) so I need to be able to specify this in the function name.

JosepSampe commented 10 months ago

Hi @sergii-mamedov, in #1211 I changed the function name, so now it looks like this:

image

Regarding having Lambda functions for different environments (production, staging); In most of the backends, like CE, they have the concept of namespace that can be used for this. In lambda we don't have this, so I was thinking on creating a namesapce config key in the aws_lamda section to simulate virtual namespaces and then prefix the function names with it, so that if you set namesapce: prod, you will see something like: prod-lithops-worker-75h4-302dev0-ae547294ec. What do you think?

sergii-mamedov commented 10 months ago

I think this would be the best option considering the limitations of AWS Lambda.

JosepSampe commented 10 months ago

I added the 'namesapce' config key in #1213

sergii-mamedov commented 10 months ago

I only have one concern. For each new release of the lambda function, I will have a separate Log group in the Cloud Watch. I have already encountered this problem during tests.

JosepSampe commented 10 months ago

Do you mean you will have a separate group because of the Lithops version appearing in the name of the worker function? or is there another reason?

sergii-mamedov commented 10 months ago

Do you mean you will have a separate group because of the Lithops version appearing in the name of the worker function?

Yes

sergii-mamedov commented 10 months ago

Of course, I can override the name of the log group, but I have to remember to do it every time.

JosepSampe commented 10 months ago

In this case, I'm not sure if it is a good idea to change and remove the lithops version from the function name. Changing this may introduce inconsistencies, as the current implementation relies on this specific naming convention to detect whether the corresponding worker version is deployed or not. Removing the version from the name has the potential to lead to inconsistencies, as Lithops won't be able to determine which version contains the "function worker" being invoked.

sergii-mamedov commented 10 months ago

Ok, then I'll manually specify the name of the log group for the lambda function. Thanks a lot.

aitorarjona commented 10 months ago

@sergii-mamedov would a configuration key to specify the log groups to attach the function to solve your issue?

sergii-mamedov commented 10 months ago

@aitorarjona yes