search

lithops-cloud / lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
http://lithops.cloud
Apache License 2.0
320 stars 106 forks source link

mess with docker runtimes #386

Closed gilv closed 3 years ago

gilv commented 4 years ago

Currently every compute engine has a default runtime. I propose to make a unique convention for default runtimes, something like 

lithops-<backend-name>-<python-version>-<custom-text>:<version id>

I also would like to explore an option to automatically update all default runtimes for all backends once we release new Lithops version. Otherwise it almost impossible to track those runtimes vs Lithops releases.

@JosepSampe what do you think?

JosepSampe commented 4 years ago

Are you referring to knative-based backends? i.e. code engine? In vanilla knative there is already a convention and it is: runtime_name = '{}/{}-v{}:{}'.format(docker_user, RUNTIME_NAME_DEFAULT, python_version, revision) for example: jsampe/lithops-knative-v3.6:2.1.0

gilv commented 4 years ago

@JosepSampe hmm....i will double check myself. thanks

gilv commented 4 years ago

@JosepSampe this ibm cf

RUNTIME_DEFAULT = {'3.5': 'ibmfunctions/lithops:3.5',
                   '3.6': 'ibmfunctions/action-python-v3.6',
                   '3.7': 'ibmfunctions/action-python-v3.7:1.6.0',
                   '3.8': 'jsampe/action-python-v3.8'}

this is OpenWhisk

RUNTIME_DEFAULT = {'3.5': 'ibmfunctions/lithops:3.5',
                   '3.6': 'ibmfunctions/action-python-v3.6',
                   '3.7': 'ibmfunctions/action-python-v3.7:1.6.0',
                   '3.8': 'jsampe/action-python-v3.8'}

This is code engine

RUNTIME_DEFAULT = {'3.5': 'cactusone/lithops-code-engine-v3.5',
                   '3.6': 'cactusone/lithops-code-engine-v3.6',
                   '3.7': 'cactusone/lithops-code-engine-v3.7',
                   '3.8': 'cactusone/lithops-code-engine-v3.8'}

I don't know, i feel something is not consistent here.

JosepSampe commented 4 years ago

They are not consistent because IBM CF/OpenWhisk use default IBM CF Runtimes that have their own names lithops package is not included in those runtimes.

In code engine, you had to create your own runtimes becasue IBM CF runtimes are not compatible and they don't include lithops lib. And in Knative/CE lithops lib must be installed in the runtime.

The solution: create the lithops docker hub account and push all runtimes there with the same name convention