Python dependencies are commented in requirements text files

[konstantinos2018]

@jcrattz Hi. I have an issue with the python dependencies, specifically with GDAL. I saw that you have commented the python packages in the requirements files such as this

Is there a reason for this?

Best Regards, Kostas

[jcrattz]

It is not needed in the requirements.txt file since the dependency is managed by the conda environment.

[tahvane1]

I have same problem and no it is not managed by conda (as it is commented out). Installation instructions are not very good and in dev docker file has faulty entrypoint (not working at least for me)

[tahvane1]

I can confirm that Python env in dev docker is completely messed up. I managed to get system up and running by using requirements file (and commenting out virtuelenv stuff in entrypoint)... now just have to find out somewhere where to get default users... Can somebody more knowledgeable try to follow installation instructions and fix what has been broken?

[jcrattz]

GDAL should be present in the environment from the base image jcrattzama/datacube-core:1.8.3 referenced in the base Dockerfile. The commented dependencies in requirements.txt should all come from that environment, which uses Conda. The only dependencies in requirements.txt should be for the UI specifically, not the ODC installation that runs for the UI.

This works for me in the development environment, so should be fixed in the next release (end of this month). I have some other notes from Kostas that also need to be incorporated.

[jcrattz]

I can confirm that this is working fine for the current commit.

GDAL, rasterio, and other ODC dependencies are included (shown by pip3 list) in the odc conda environment that activates on start (denoted by the leading (odc) in the bash prompt shown by dev-ssh after dev-up).

Running service --status-all shows that the apache2, celerybeat, and data_cube_ui services are running (denoted by a + sign). The celeryd service is not running (denoted by a - sign), but that should be irrelevant for functional purposes. The apps also work for me.

So you may need to pull the latest commit and then run dev-restart. If you want changes to the code to take effect in the environment, leave the container (Ctrl + D) and then run dev-restart.

What errors are occurring?

[tahvane1]

Hi, for starters docker.io/jcrattzama/data_cube_ui:odc1.8.3_base does not have miniconda (/miniconda/odc) it has /odc instead. Is right image in docker hub? jcrattzama/datacube-base:odc1.8.3 image seems to have miniconda...

Br, Erkka

[tahvane1]

I can build base image myself by adding apt-get update to install_apache_deps.sh and adding setuptools to requirements_core file. /app/datacube_env is still missing though...

[jcrattz]

Well the base image for the UI is jcrattzama/data_cube_ui:odc1.8.3_base, but the base image of that image is jcrattzama/datacube-base:odc1.8.3.

I spotted the problem. In build/docker/base/Dockerfile, replace FROM jcrattzama/datacube-core:1.8.3 with

ARG BASE_IMG
FROM $BASE_IMG

Then in Makefile, replace dev-up: with dev-up: base-build

Then run make dev-restart. Upon running make dev-ssh, you should see (odc) in the shell prompt denoting the conda environment odc is active. And the output of service --status-all should look like this:

 [ - ]  apache-htcacheclean
 [ + ]  apache2
 [ + ]  celerybeat
 [ - ]  celeryd
 [ + ]  data_cube_ui
 [ - ]  dbus
 [ ? ]  hwclock.sh
 [ - ]  procps
 [ - ]  unattended-upgrades

Thank you for helping to catch this problem. I am surprised this was still working in my environment as long as it was. Let me know whether or not this fixes the problem for you.

[tahvane1]

Still had (same) issue with apache. After fixing that apache2 is up but other services not. Errors in log stat: cannot stat '/app/datacube_env': No such file or directory

chown: cannot access '/app/datacube_env': No such file or directory

• Starting Apache httpd web server apache2

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.23.0.13. Set the 'ServerName' directive globally to suppress this message

*

build/native/entrypoint.sh: line 42: datacube_env/bin/activate: No such file or directory

build/native/entrypoint.sh: line 43: datacube: command not found

Should it use $DC_UI_PYTHONHOME? At that point it seems to just initialize ODC.

Changing that, running startup scripts and changing log folder permission and I'm able to open up web page. I'm not sure yet if it is working or not as only apache2 has + in service status.

[jcrattz]

Try:

make dev-down
docker image pull jcrattzama/datacube-base:odc1.8.3
docker image pull jcrattzama/data_cube_ui:odc1.8.3_base
docker image pull jcrattzama/data_cube_ui:odc1.8.3_dev
make dev-restart

What is the container environment like after running these commands?

Building may be faster than pulling (varies by download speed), in which case you must still pull the datacube-base image, but you can replace pull with rm -f for the data_cube_ui images. Either way, it will probably take a few minutes and using pull is safer.

[tahvane1]

I used dev-up-no-build to test this (as dev-restart makes build). This way apache2, data_cube_ui and celerybeat are healthy. So build process seem to be broken.

[jcrattz]

We had to correct one of the base Docker images (used in a FROM instruction) - not a UI-specific image. The build process works, but the image was old (pushing was done manually).