Some pip3 packages not found by Jobe but working in Python3

[kulttuuri]

Hello again!

I've forked this repository and added some pip3 packages that we use here to teach AI / Data Science. You can see our Dockerfile from here.

The problem I am currently facing is that most of the pip libraries do work, but some of them do not work. These libraries are giving me the "ModuleNotFoundError" when trying to run my code from the CodeRunner Moodle plugin: spacy, pillow, skimage, pandas. The weird thing is that if I login into my running docker container and run python3 and try importing the libraries, they all get imported without problems. The same thing goes when I even login to user jobe00 / jobe01, start Python3 and try importing these libraries. They work from there, but not when trying to import through the CodeRunner Moodle plugin.

Exact error messages when trying to import are here:

import spacy

***Run error***
Traceback (most recent call last):
  File "__tester__.python3", line 1, in <module>
    import spacy
ModuleNotFoundError: No module named 'spacy'

from PIL import Image

***Run error***
Traceback (most recent call last):
  File "__tester__.python3", line 4, in <module>
    from PIL import Image
ModuleNotFoundError: No module named 'PIL'

import skimage

***Run error***
Traceback (most recent call last):
   File "__tester__.python3", line 4, in <module>
      import skimage
ModuleNotFoundError: No module named 'skimage'
--

import pandas as pd

***Run error***
Traceback (most recent call last):
  File "__tester__.python3", line 2, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'

Any ideas? :)

[trampgeek]

The Python environment is a mysterious black art. See https://xkcd.com/1987/

The two most obvious things I can suggest are:

1. Are you quite sure you haven't somehow got two different Python 3 versions, with (perhaps) some shared modules and some unshared? To check, run the following program from the command line within Jobe and from Moodle:

   import sys
   print(sys.executable)

2. Jobe runs tasks using an old version of a program called runguard from the domjudge program contest server. This cleans the environment in various ways for security reasons. It's possible that the packages that don't work require certain environment variables that are being zapped by runguard. It's difficult to test for this without modifying the runguard code itself within Jobe. You could try modifying the the above test program to also print things like sys.path, sys.modules, though if you do find a difference you're then deep into the dark arts.

If you get sufficiently desperate, there's a sandbox parameter 'debug' - or perhaps it's 'debugging' - that stops Jobe from deleting the working directory of each job. So if you set the sandbox parameters to

{"debug": 1, "debugging": 1}

to cover both options, run a job from Moodle, then log into the container, you'll find the run(s) in the directory /home/jobe/runs. There you can inspect the actual shell command used to run the job via runguard, and you can experiment with that.

Good luck! Let us know what you find :)

[kulttuuri]

I think I found the root cause. When I run the print(sys.executable) in the Jobe server's Python3, I get:

/usr/local/bin/python3

And when I run this same code in the Coderunner, I get:

/usr/bin/python3

So it seems like that the Jobe server is running the system's default Python version (possibly with a hardcoded path somewhere in its code, hopefully not? :)), instead of the one that I locally installed in the system. Can I change the Python executable that the Jobe server uses somehow or how should I proceed in order to change this?

My ideal situation would not to depend on the operating system's default Python version but stick to some version for a year or so and then we'll do a manual update at one point when we have tested that the new Python executable still works with all of our pip libraries. Currently, if we update Ubuntu and they decide to switch over to some latest 3.9.x version of Python, it might (this has happened multiple times) break the libraries that we currently use.

I have two ways I can probably solve this, as far as I have determined:

1. Just use the operating system's default Python located in /usr/bin/python3 and install all the pip packages to this. Jobe uses this by default to run the code. The con is that if Ubuntu decides to switch over to some new Python version while we update the Jobe server, it might break some libraries.
2. Switch Jobe to use the /usr/local/bin/python3 executable which has the pip libraries that I have installed and I can lock to a certain Python version. This would require me to probably configure this executable to be used somewhere in Jobe and probably a good amount of testing that all works well if switching over to this way.

[trampgeek]

In the current release of Jobe the version that is run is determined by the file python3_task.php, buried down in jobe/application/libraries. There's a method

    public function getExecutablePath() {
        global $PYTHON3_VERSION;
        return "/usr/bin/$PYTHON3_VERSION";
     }

where the variable $PYTHON3_VERSION is a configuration parameter in jobe/applicaton/config/config.php. So I think you should be able to add a link in /usr/bin to your preferred Python. Note the warning in the config file about the need to reboot (or equivalent) after changing version.

With two versions of Python on your machine, I think you should avoid using pip3 directly in your dockerfile. Instead load modules for the required python version by using the syntax

python3.x -m pip install module_name

That's guaranteed to install for the correct version.

[kulttuuri]

Fantastic. What I actually did is that I just switched over to the operating system's Python executable (at /usr/bin) to prevent all the hassle. Later on, I will create a Docker healthcheck or similar in which I will try to import all the pip libraries and if I'll get an error with that, the build fails. This way I can be sure that all should be working.

BTW unrelated to this issue:

I noticed that in the Dockerfile here the php-cli \ command is being ran twice.

We also had some problems with the openjdk-14-jdk package as the latest Ubuntu release (20.04.3) that comes with the Docker image has that package completely removed from the apt repository. We changed the package to opendjk-16-jdk and all seems to be working fine. Haven't checked if they have added the openjdk-14-jdk package back, but you might want to check that. We've also not run any Java code, so I don't know if there are any problems with the latest 16 package.

Nevertheless, Thanks Richard! :)