CustomPipeline is broken

[denisri]

When instantiating a CustomPipeline, we get an error:

  File "/casa/host/src/capsul/capsul/application.py", line 74, in executable
    return executable(definition, **kwargs)
  File "/casa/host/src/capsul/capsul/application.py", line 138, in executable
    result =  executable_from_json(definition, json_executable)
  File "/casa/host/src/capsul/capsul/application.py", line 268, in executable_from_json
    result = CustomPipeline(definition=definition,
  File "/casa/host/src/capsul/capsul/pipeline/pipeline.py", line 2392, in __init__
    super().__init__(definition=definition, autoexport_nodes_parameters=json_executable.get('export_parameters', False))
  File "/casa/host/src/capsul/capsul/pipeline/pipeline.py", line 243, in __init__
    self.pipeline_definition()
  File "/casa/host/src/capsul/capsul/pipeline/pipeline.py", line 2408, in pipeline_definition
    for name, ejson in self.json_executable.get('executables', {}).items():
  File "/casa/host/build/python/soma/controller/controller.py", line 322, in __getattribute__
    result = super().__getattribute__(name)
AttributeError: 'CustomPipeline' object has no attribute 'json_executable'

Apparently json_executabe is set in the instance in CustomPipeline.__init__, then it calls super().__init__(), which in turn calls pipeline_definition(), and here the json_executable attribute is gone. Does the constructor of Controller wipe out all existing attributes when its __init__() is called ? This is problematic... Is there a way to avoid it ?

See for instance the test:

python3 -m capsul.process.test.test_load_from_description

[sapetnioc]

I do not have this problem. The test is working for me. Serialization of pipelines also uses this json_executable so I should have had a problem when running TinyMorphologist.

I will push a quick made wip commit to be sure we are sync.

[denisri]

:o it works with python 3.9, but fails with python 3.10..................

[sapetnioc]

I tested test_load_from_description with Python 3.9 and 3.10 (using a Singularity image and a corresponding venv) and got no error in both cases.

[sapetnioc]

$ singularity pull docker://python:3.10
$ ./python_3.10.sif python -m venv venv-3.10
$ ./python_3.10.sif venv-3.10/bin/pip install -e src/soma-base
$ ./python_3.10.sif venv-3.10/bin/pip install -e src/capsul
$ ./python_3.10.sif venv-3.10/bin/pip install traits # by the way, this is missing in dependencies
$ ./python_3.10.sif venv-3.10/bin/python -m capsul.process.test.test_load_from_description
s....
----------------------------------------------------------------------
Ran 5 tests in 4.635s

OK (skipped=1)

[denisri]

Sorry I had disabled the test while working on #213. I have reactivated it, and for me it fails using python 3.10

[sapetnioc]

I just pulled Capsul and had no error with the above procedure (an no skipped test).

[denisri]

Strange. What can be different between our configs ? I still have the error...

[sapetnioc]

In case it might be useful to find the difference in our environments :

$ ./python_3.10.sif venv-3.10/bin/python --version
Python 3.10.2

$ ./python_3.10.sif venv-3.10/bin/pip freeze
-e git+https://github.com/populse/capsul@bb852d533cfb84a4fdb81dca060765629c72bbca#egg=capsul
lark-parser==0.12.0
populse-db==2.0.1
pydantic==1.9.0
python-dateutil==2.8.2
PyYAML==6.0
six==1.16.0
-e git+https://github.com/populse/soma-base@f4ffd4abae963cbcb57a561163dc323aa82fd60d#egg=soma_base
soma-workflow==3.0.0
traits==6.3.2
typing_extensions==4.1.1

[sapetnioc]

The commit displayed by pip freeze is the right one for the pydantic_controller branch (I did not know this feature by the way). I do no have files being edited. I do no see where the difference can be.

[denisri]

$ python3 --version
Python 3.10.2

$ pip3 freeze
alabaster==0.7.12
appdirs==1.4.4
argon2-cffi==21.1.0
astroid==2.9.3
atomicwrites==1.4.0
attrs==21.2.0
autopep8==1.6.0
Babel==2.8.0
backcall==0.2.0
bcrypt==3.2.0
beautifulsoup4==4.10.0
black==21.10b0
bleach==4.1.0
bounded-pool-executor==0.0.3
Brotli==1.0.9
bs4==0.0.1
certifi==2020.6.20
cfgv==3.3.1
chardet==4.0.0
ci-info==0.2.0
click==8.0.3
cloudpickle==2.0.0
colorama==0.4.4
commonmark==0.9.1
configobj==5.0.6
cryptography==3.4.8
cycler==0.11.0
Cython==0.29.28
decorator==4.4.2
defusedxml==0.7.1
diff-match-patch==20200713
distlib==0.3.4
docutils==0.17.1
entrypoints==0.4
et-xmlfile==1.0.1
etelemetry==0.3.0
fastcluster==1.2.3
filelock==3.6.0
flake8==4.0.1
fonttools==4.29.1
fs==2.4.12
future==0.18.2
gitdb==4.0.9
GitPython==3.1.27
h5py==3.6.0
h5py.-debian-h5py-serial==3.6.0
helpdev==0.7.1
html2text==2020.1.16
html5lib==1.1
identify==2.4.11
idna==3.3
imageio==2.4.1
imagesize==1.3.0
importlib-metadata==4.6.4
intervaltree==3.0.2
ipycanvas==0.11.0
ipyevents==2.0.1
ipykernel==4.10.1
ipython==7.31.1
ipython_genutils==0.2.0
ipywidgets==7.6.5
isodate==0.6.1
isort==5.6.4
jdcal==1.0
jedi==0.18.0
jeepney==0.7.1
Jinja2==3.0.3
joblib==0.17.0
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==7.1.2
jupyter-console==6.4.3
jupyter-core==4.9.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.2
keyring==23.5.0
kiwisolver==1.3.2
lark==1.1.1
lazy-object-proxy==0.0.0
logilab-common==1.8.2
lxml==4.8.0
lz4==3.1.3+dfsg
MarkupSafe==2.0.1
matplotlib==3.5.1
matplotlib-inline==0.1.3
mccabe==0.6.1
meld==3.20.4
mistune==0.8.4
mock==4.0.3
more-itertools==8.10.0
mpi4py==3.1.3
mpmath==0.0.0
mypy-extensions==0.4.3
mysqlclient==1.4.6
nbclient==0.5.6
nbconvert==6.3.0
nbformat==5.1.3
nbsphinx==0.8.7
nest-asyncio==1.5.4
networkx==2.4
nibabel==3.2.1
nipype==1.7.0
nodeenv==1.6.0
notebook==5.7.13
numpy==1.21.5
numpydoc==1.2
objgraph==3.5.0
openpyxl==3.0.9
packaging==21.3
pandas==1.3.5
pandocfilters==1.5.0
paramiko==2.8.1
parso==0.8.1
pathspec==0.9.0
pbr==5.8.0
pep8==1.7.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.0.1
pip-search==0.0.10
pkgconfig==1.5.5
platformdirs==2.5.1
plotly==5.4.0
pluggy==0.13.0
pqdm==0.2.0
pre-commit==2.17.0
prometheus-client==0.9.0
prompt-toolkit==3.0.28
prov==2.0.0
psutil==5.9.0
ptyprocess==0.7.0
py==1.10.0
pycairo==1.20.1
Pycluster @ http://bonsai.hgc.jp/~mdehoon/software/cluster/Pycluster-1.59.tar.gz
pycodestyle==2.8.0
pydantic==1.8.2
pydicom==2.2.2
pydot==1.4.2
pyflakes==2.4.0
Pygments==2.10.0
PyGObject==3.42.0
pylint==2.12.2
pyls-black==0.4.6
pyls-spyder==0.4.0
PyNaCl==1.5.0
pyparsing==2.4.7
PyQt5==5.12.1
PyQt5_sip==4.19.15
PyQtWebEngine==5.12.1
pyrsistent==0.18.1
python-dateutil==2.8.1
python-jsonrpc-server==0.4.0
python-language-server==0.36.2
python-lsp-jsonrpc==1.0.0
python-lsp-server==1.3.3
pytz==2021.3
PyWavelets==1.1.1
pyxdg==0.27
PyYAML==5.4.1
pyzmq==22.3.0
QDarkStyle==2.8.1
QtAwesome==1.1.1
qtconsole==5.2.2
QtPy==2.0.0
rdflib==6.0.1
requests==2.25.1
rich==12.0.0
roman==3.3
scikit-image==0.18.3
scikit-learn==0.23.2
scipy==1.7.1
SecretStorage==3.3.1
Send2Trash==1.8.1b0
simplejson==3.17.6
six==1.16.0
smmap==5.0.0
snowballstemmer==2.2.0
sortedcontainers==2.1.0
soupsieve==2.3.1
Sphinx==4.3.2
sphinx-gallery==0.10.1
spyder==4.2.1
spyder-kernels==1.10.2
SQLAlchemy==1.3.22
sympy==1.7.1
tenacity==6.3.1
terminado==0.13.1
testpath==0.5.0
textdistance==4.2.2
threadpoolctl==3.1.0
three-merge==0.1.1
tifffile==2022.2.9
toml==0.10.2
tomli==1.2.2
torch==1.8.0a0+unknown
torchvision==0.8.0a0
tornado==4.5.3
tox==3.21.4
tqdm==4.57.0
traitlets==5.1.1
traits==6.3.2
typing-extensions==3.10.0.2
ufoLib2==0.13.1
ujson==5.1.0
urllib3==1.26.5
virtualenv==20.13.0+ds
watchdog==2.1.6
wcwidth==0.2.5
webencodings==0.5.1
widgetsnbextension==3.5.2
wrapt==1.13.3
wurlitzer==3.0.2
xdg==5
xdot==1.2
xlrd==1.2.0
xlwt==1.3.0
xmltodict==0.12.0
zipp==1.0.0

especially we don't have the same version of pydantic and typing_extensions (I don't kinow what this last one is)

[sapetnioc]

I do not remember why but in Capsul, there is a minimum version of Pydantic set to 1.9.0

[sapetnioc]

I will try to downgrade pydantic to see if it is the source of the problem.

[sapetnioc]

I have the same problem with Pydantic 1.8.2

[sapetnioc]

Do you see a problem with forcing Pydantic >= 1.9.0 ?

[denisri]

OK it's actually the version of pydantic. Let's use pydantic >= 1.9. Thanks.

[denisri]

I was using the prerelease of Ubuntu 22.04 (through casa-distro container casa-dev-5.3alt.sif), and the apt install of pydantic ships version 1.8. So we have to upgrade in the images. I'm rebuilding them.

[denisri]

I have published new images, which fix the problem.