Problem with python bytecode cache files (__pycache__)

[moazreyad]

When I run SciUnit with a Python example on Ubuntu 22.04, I get this error:

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
Traceback (most recent call last):
  File "/usr/lib/python3.10/encodings/__init__.py", line 31, in <module>
ModuleNotFoundError: No module named 'codecs'

[moazreyad]

I found the reason of the problem:

Sciunit does not support Python bytecode cache files (__pycache__). Sciunit expect to find Python source files such as codecs.py in locations like:

cde-root/usr/lib/python3.10/codecs.py

However, this is only true in the first time you compile the Python code. To avoid re-compiling the code, the following executions will use a cached version that will be in:

cde-root/usr/lib/python3.10/__pycache__/codecs.cpython-310.pyc

But this caching mechanism does not seem to work with Sciunit that still tries to find the original Python source files (.py) instead of the cached compiled bytecode files (.pyc)

Solution (temporary until Python bytecode cache files are supported in Sciunit):

1. Disable the creation of the __pycache__ directory by define PYTHONDONTWRITEBYTECODE environment variable:

export PYTHONDONTWRITEBYTECODE=1

Note: disable cache means every time the .py source code is used, it must be re-compiled again which makes the execution a little slower.

2. If you have __pycache__ and encoding/__pycache__, delete them from your /usr/lib/python3.XX:

# rm -r /usr/lib/python3.10/__pycache__/
# rm -r /usr/lib/python3.10/encodings/__pycache__/

[moazreyad]

Solved by https://github.com/depaul-dice/provenance-to-use/pull/3

[sranasir]

Was not completed fixed. Fixed now by https://github.com/depaul-dice/provenance-to-use/pull/6