lithops-cloud / lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
http://lithops.cloud
Apache License 2.0
317 stars 105 forks source link

Runtime for AWS EC2 #1101

Closed sergii-mamedov closed 1 year ago

sergii-mamedov commented 1 year ago

Unfortunately, there is no example Dockerfile for AWS EC2 in the runtime directory. There is also a link to the AWS EC2 runtime on the configuration description page, which also leads to a non-existent page.

I would appreciate an example Dockerfile for AWS EC2.

JosepSampe commented 1 year ago

@sergii-mamedov WIll update the docs.

However, note that you can use any docker conatiner that contains all your required libraries, since lithops overwrites the default docker entrypoint, so it doesn´t matter wich docker container you use if it is based on Debian/Ubuntu. For example, you can use an IBM Cloud Functions container if you want.

Also note that, for better cold start times, the recomended approach is to use the VM itself as the runtime, without a docker container. For example, you can create a VM in you AWS region, access the VM, install all the dependencies in the VM itself (apt-get, pip3 install, ...), stop the VM, create a VM Image, and then put the image_id in your lithops config file:

aws_ec2:
    ...
    target_ami: <TARGET_AMI>
    ...

In this second case, Lithops will use the default python3 interpreter of the VM