search

xcube-dev / xcube

xcube is a Python package for generating and exploiting data cubes powered by xarray, dask, and zarr.
https://xcube.readthedocs.io/
MIT License
201 stars 20 forks source link

build docker image from latest release tag #912

Closed TejasMorbagal closed 1 year ago

TejasMorbagal commented 1 year ago

[Description of PR]

Checklist: ~ [ ] Add unit tests and/or doctests in docstrings~ ~ [ ] Add docstrings and API docs for any new/modified user-facing classes and functions~ ~ [ ] New/modified features documented in `docs/source/~ ~* [ ] Changes documented inCHANGES.md`~

codecov[bot] commented 1 year ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (4f0941f) 93.33% compared to head (cfe19a6) 93.33%. Report is 1 commits behind head on master.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #912 +/- ## ======================================= Coverage 93.33% 93.33% ======================================= Files 447 447 Lines 34924 34924 ======================================= Hits 32596 32596 Misses 2328 2328 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

TonioF commented 1 year ago

I am quite certain the reason why some packages are missing is because some of the plugin environments still try to limit gdal. In the dockerfile we install the environment dependencies and then install xcube and the plugins from source (python setup.py install or pipi install .), meaning that conda does not know about restrictions imposed by xcube or any of the previously installed plugins. Given that we (a) are only dealing with released versions and (b) the script is run manually, I think the dockerfile could be changed to install xcube and its plugins from conda-forge rather than source.

TejasMorbagal commented 1 year ago

I am quite certain the reason why some packages are missing is because some of the plugin environments still try to limit gdal. In the dockerfile we install the environment dependencies and then install xcube and the plugins from source (python setup.py install or pipi install .), meaning that conda does not know about restrictions imposed by xcube or any of the previously installed plugins. Given that we (a) are only dealing with released versions and (b) the script is run manually, I think the dockerfile could be changed to install xcube and its plugins from conda-forge rather than source.

We won't be able to have a docker image for dev releases with this approach, and the manual triggered workflow is a short-term workaround till we fix the failing unit tests.

TonioF commented 1 year ago

I am quite certain the reason why some packages are missing is because some of the plugin environments still try to limit gdal. In the dockerfile we install the environment dependencies and then install xcube and the plugins from source (python setup.py install or pipi install .), meaning that conda does not know about restrictions imposed by xcube or any of the previously installed plugins. Given that we (a) are only dealing with released versions and (b) the script is run manually, I think the dockerfile could be changed to install xcube and its plugins from conda-forge rather than source.

We won't be able to have a docker image for dev releases with this approach, and the manual triggered workflow is a short-term workaround till we fix the failing unit tests.

Fair enough. It is still very unfortunate that by installing a plugin we can override dependency restrictions. We now need to change the environments in some of the plugins, just as we had to change them when the gdal issue occurred. This shouldn't be necessary.

TejasMorbagal commented 1 year ago

I am quite certain the reason why some packages are missing is because some of the plugin environments still try to limit gdal. In the dockerfile we install the environment dependencies and then install xcube and the plugins from source (python setup.py install or pipi install .), meaning that conda does not know about restrictions imposed by xcube or any of the previously installed plugins. Given that we (a) are only dealing with released versions and (b) the script is run manually, I think the dockerfile could be changed to install xcube and its plugins from conda-forge rather than source.

We won't be able to have a docker image for dev releases with this approach, and the manual triggered workflow is a short-term workaround till we fix the failing unit tests.

Fair enough. It is still very unfortunate that by installing a plugin we can override dependency restrictions. We now need to change the environments in some of the plugins, just as we had to change them when the gdal issue occurred. This shouldn't be necessary.

Yes, that's right. The idea of not installing plugins was also motivated by the fact that none of our viewer deployments actually needs plugins installed. So, why do we install plugins in the docker image ? This also makes the docker image larger.