Failing to get install_dependencies.sh working

[diplonics]

Can you expand/explain in a bit more detail how the install_dependencies.sh should work. From the documentation here: https://docs.eo4a.science/service_development/components.html#dependency-installation it suggests that the install_dependencies.sh script is automatically run but we are unable to get the dependencies installed to get CFPixels running properly.

Our install_dependencies.sh file currently looks like this but doesn't seem to be running?

!/bin/sh

apt update apt-get install python-pip pip install geojson pip install pathlib pip install shapely

[derekocallaghan]

In local standalone mode, install_dependencies.sh is only executed each time the EO4Atlantic container (eo4a-standalone) is created, where eo4adev will only create the latter if it doesn't already exist. In the platform itself, install_dependencies.sh is executed every time as a new Docker container is created for each service execution.

Assuming eo4a-standalone exists locally, can you remove it with docker stop, docker rm, rerun eo4adev, and see what happens? See https://docs.eo4a.science/service_development/iterative.html#useful-operations for examples of these docker commands

[derekocallaghan]

Another point to mention is that pip (in the Anaconda bundle) is already available in the container, so it doesn't need to installed via install_dependencies.sh, as the conda version will be used in the subsequent calls to pip install ....

[dairejpwalsh]

Trying to use pip to install python package. Even if I restart the service as you suggest I still get,

owslib.wps.WPSException : {'locator': None, 'code': 'NoApplicableCode', 'text': 'Process error: EO4AProcess.py._run_process Line 384 Process ended with non-zero exitcode:\nTraceback (most recent call last):\n File "/opt/eo4a/services/cfpixels/run.py", line 1, in \n import cfp\n File "/opt/eo4a/services/cfpixels/cfp.py", line 32, in \n import geojson\nImportError: No module named geojson\n'}

Contents of install_dependencies.sh, apt update sudo apt-get install python-pip sudp pip install geojson sudo pip install pathlib sudo pip install shapely

[derekocallaghan]

Looks like this appeared prior to my previous point about not installing python-pip, as the conda versions of pip and python are used with any service code running in the container. I suggesting removing the apt update...apt-get install commands.

[derekocallaghan]

Also, the sudo prefix is not required as everything is run as root in the container.

[dairejpwalsh]

I removed the,

sudo apt-get install python-pip

and have now the contents of install_dependencies.sh are,

# !/bin/sh
pip install geojson
pip install pathlib
pip install shapely

Even after restarting the service I am still getting the same error,

No module named geojson

[derekocallaghan]

Can you remove the container as described above, rerun eo4adev, and attach the output here. Can you also specify the values of the various EO4A_... environment variables, if these have been set.

[derekocallaghan]

In addition to the output from eo4adev, can the container log output from docker logs -f eo4a-standalone also be attached.

[derekocallaghan]

Also, prior to rerunning the above steps, can you change #!/bin/sh to #!/bin/sh -x in install_dependencies.sh, so that any output will show up in the container log.

[derekocallaghan]

I'm merging these suggestions into a single post as it's easier to read:

• Stop+remove the container, rerun eo4adev, and attach the output
• After running eo4adev, attach the output of docker logs eo4a-standalone
• If you're using the EO4A_SERVICES_DIR to specify the location of the service git repo (or directory somewhere in the repo), attach the output of tree $EO4A_SERVICES_DIR so that I can see the contents of the repo (this is run on the host machine, and you may need to install the tree utility first)
• Afterwards, you can connect to the running container with docker exec -it eo4a-standalone bash, and try manually running the three pip install... commands.
• Try restarting the notebook you use for testing, and rerun reload_services(wps). This should pick up the 3 installed packages.

[derekocallaghan]

Having tried to reproduce the problem here, I suspect that these three dependencies are not fully installed prior to your Python code that tries to import them. It may just be a case of running the reload_services(wps) from the notebook, once they have been fully installed (i.e. after seeing a message like 2018-01-25 14:59:38,930 INFO exited: install_dependencies (exit status 0; expected) in the container log).

[dairejpwalsh]

Log in link,

https://drive.google.com/open?id=1nZRxSN9kboTlI9Cw3Utpj5azdecY1NA7

docker exec -it eo4a-standalone bash,

This work and allows me to install dependencies. No error after installing with this method

[derekocallaghan]

Thanks for the log. Good to hear that the manual installation works.

[derekocallaghan]

Closing as platform deployment has confirmed that install_dependencies.sh is working.