search

InseeFrLab / images-datascience

Collection of Docker images to build the data science catalog of the Onyxia project
MIT License
24 stars 22 forks source link

Upgrades r spark ubuntu #88

Closed ChatBear closed 1 year ago

ChatBear commented 1 year ago

close #57
close #68

alexisdondon commented 1 year ago

here maybe a forgotten ls https://github.com/InseeFrLab/images-datascience/blob/upgrades/python-datascience/Dockerfile#L15

i ve checked py4j version https://github.com/InseeFrLab/images-datascience/blob/upgrades/spark/Dockerfile#L17 for spark it did not change for 3.3.1 but will change in next versions.

i ve not made all dev build chain python script but a few it seems to go well!

alexisdondon commented 1 year ago

We could maybe make workflow and dev build chain coherent it seems to me that for example jupyter-minimal is in build chain but not in workflow?

We could also discuss about the next step to validate this PR.

In the best world we would want to validate whole workflow withtout pushing. It is not possible with the new workflow wihtout artefacts.

In this PR, the updates will generate new tags as all versions have changed, we can launch the 'docker build and push' workflow manually on this branch to see if all goes well, in the worst scenario we will have partial new images on dockerhub. --> except base image that i think have latest tag and will be erase. it s probably no deal?

Probably, one day we would want the workflow build and push to generate tagname based on branch . For example, building on upgrades could generate all tag onyxia-python-minima:upgrades-classictaggenerated (py3.10.8...) in this case we could validate the workflow being sur not to erase the current production images no?

if not branchname anything discriminant from main like commit sha slug or PR id.

The other alternative is to make a dedicated workflow that run dev/build_chain.py scripts but we could maybe run out of space on a github runner.