diocas
commented
2 years ago Hi Luca, thanks for the PR.
Many things you're adding should not come to this repo, but are helpful nonetheless to understand what you're doing at JRC. As we agreed, I will help build a JupyterHub docker image for JRC, so that you don't need to fork/copy our code and instead just add the extra changes on top. This is still in the pipeline, I just need to find some time.
This also means that your JeoSpawner might still live on your side and not on our repo if we cannot merge everything you need (I still need to check your code to see if we can do it or to identify common objectives and propose a common solution). But, for sure we don't want the code in a separate folder. If it can come to our side, it needs to be merged with the rest. If it cannot, we can put in your own repos.
So, I was expecting that you did a PR with the necessary changes to our Spawner to make them usable for both of us to create the external resources cluster. Also, given the documentation that my colleagues sent, it might need changes to the k8s config/helm charts as well (in here: https://github.com/swan-cern/swan-charts). The advantage of their proposed approach was to give the possibility to scale the Dask/Spark clusters from the user container dynamically, if I'm not mistaken, and it had less changes. You might still need different code in the Spawner (the one you have now) for your other resources that don't have a Jupyter extension in order to configure the rest, though... Let me know if we can check this approach first and wether it would work for all or just some of your cases. If we still need extra code for the use cases not covered by this approach, I see no objection into adding it in the SwanSpawner as an option that you can enable via config.
Please leave the PR open, I will check it for reference, but can you check my comment on the Spawner and do a separate PR? Thanks!
First commit with full relevant files for the deployment