No module named 'pyln.client'

[darioAnongba]

Issue and Steps to Reproduce

Trying to start a Core Lightning node in a docker-compose setup on MacOS (arm64v8) and the CLNRest plugin (clnrest-port set) will fail with this error :

plugin-clnrest.py: Killing plugin: disabled itself: No module named 'pyln.client'

The doc says:

The plugin is built-in with Core Lightning but its python dependencies are not, and must be installed separately.
Install required packages with pip install -r plugins/clnrest/requirements.txt.

But after spending some time on this I came to the conclusion that this is a bug, or rather a missing item in the Docker image on arm64v8 because the image should ship with the necessary dependencies since we cannot exec into it to install whatever might be missing, especially not in a kubernetes or any other professional env.

Also, it is working fine on amd64 so this seems to be a bug specific to MacOS.

I can fix this myself if you consider that it is indeed a bug.

Version

Version v24.02-arm64v8

[ShahanaFarooqui]

@darioAnongba Thanks for reporting it. It is a bug as we already have a necessary step in the Dockerfile to install it:

RUN /root/.local/bin/poetry export -o requirements.txt --without-hashes --with dev
RUN pip3 install -r requirements.txt

Build logs also show that all clnrest requirements are being installed (including pyln-client):

Successfully installed aniso8601-9.0.1 asn1crypto-1.5.1 attrs-23.1.0 base58-2.1.1 bidict-0.22.1 bitarray-2.8.2 bitstring-4.1.2 blinker-1.6.3 certifi-2023.7.22 cffi-1.16.0 charset-normalizer-3.3.1 cheroot-8.6.0 click-8.1.7 clnrest-24.2 coincurve-18.0.0 crc32c-2.3.post0 cryptography-41.0.5 ephemeral-port-reserve-1.1.4 exceptiongroup-1.1.3 execnet-2.0.2 flake8-4.0.1 flaky-3.7.0 flask-2.3.3 flask-cors-4.0.0 flask-restx-1.1.0 flask-socketio-5.3.6 gevent-23.9.1 gevent-websocket-0.10.1 greenlet-3.0.0 grpcio-1.59.0 grpcio-tools-1.59.0 gunicorn-21.2.0 h11-0.14.0 idna-3.4 importlib-metadata-6.8.0 iniconfig-2.0.0 itsdangerous-2.1.2 jaraco-functools-3.9.0 jinja2-3.1.2 json5-0.9.14 jsonschema-4.19.1 jsonschema-specifications-2023.7.1 mako-1.2.4 markupsafe-2.1.3 mccabe-0.6.1 more-itertools-10.1.0 mypy-0.931 mypy-extensions-1.0.0 packaging-23.2 pluggy-1.3.0 protobuf-4.21.12 protobuf3-0.2.1 psutil-5.9.6 psycopg2-binary-2.9.9 py-1.11.0 pycodestyle-2.8.0 pycparser-2.21 pyflakes-2.4.0 pyln-bolt7-1.0.246 pyln-client-24.2 pyln-grpc-proto-0.1.1 pyln-proto-24.2 pyln-testing-24.2 pysocks-1.7.1 pytest-7.4.3 pytest-custom-exit-code-0.3.0 pytest-forked-1.6.0 pytest-test-groups-1.0.3 pytest-timeout-2.2.0 pytest-xdist-2.5.0 python-bitcoinlib-0.11.2 python-engineio-4.8.0 python-socketio-5.10.0 pytz-2023.3.post1 referencing-0.30.2 requests-2.31.0 rpds-py-0.10.6 setuptools-68.2.2 simple-websocket-1.0.0 six-1.16.0 tomli-2.0.1 typing-extensions-4.8.0 urllib3-2.0.7 websocket-client-1.6.4 werkzeug-3.0.1 wsproto-1.2.0 zipp-3.17.0 zope-event-5.0 zope-interface-6.1

However, the Dockerfile has been updated after 24.02.2 and I re-confirmed that clnrest works with the updated Dockerfile. So it should be fixed with the upcoming version 24.05 (releasing today).

Do you still need v24.02 image fixed?

[darioAnongba]

Thanks for the quick response. No there is no need to fix image v24.02 if there is a new release coming with a fix.

Just to clarify. The dependencies indeed seem to be correctly installed and the pyln-client present in the image. Running pip3 show pyln-client in the container by exec into it returns successfully:

Name: pyln-client
Version: 24.2
Summary: Client library and plugin library for Core Lightning
Home-page: 
Author: Christian Decker
Author-email: decker.christian@gmail.com
License: BSD-MIT
Location: /usr/local/lib/python3.9/dist-packages
Editable project location: /opt/lightningd/contrib/pyln-client
Requires: pyln-bolt7, pyln-proto
Required-by: clnrest, pyln-testing

The issue seems to be the usage of the module in the code. As running a simple test with:

python3 -c "import pyln.client; print('pyln.client is available')"

Shows the same error:

# python3 -c "import pyln.client; print('pyln.client is available')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pyln.client'

So the issue might be a bit different than missing dependencies. Rather non-accessible module.

[darioAnongba]

I see you jut released a version v24.05 but the image is missing on dockerhub. Is this voluntary? I tried building the image myself but sadly failed.

[ShahanaFarooqui]

It should be available by eod today.

[ShahanaFarooqui]

Editable project location: /opt/lightningd/contrib/pyln-client

So the issue might be a bit different than missing dependencies. Rather non-accessible module.

Thanks for the hint and saving my debugging time :). The issue is that pyln-client (and other packages) are installed in editable mode because pyln-client is listed in develop mode in pyproject.toml. But the project is unavailable at the editable project location.

pyln-client = { path = "contrib/pyln-client", develop = true }
...

Now, we have two solutions for the issue (apart from some other fix ups like uninstall & install again):

1: Install them in non-editable mode (develop = false)

RUN sed -i 's/develop = true/develop = false/' pyproject.toml

OR

2: Copy these path directories from builder stage to final stage's /opt/lightningd directory, so that it can be found at its "Editable project location". like:

COPY --from=builder contrib/pyln-client /opt/lightningd/contrib/pyln-client
COPY --from=builder contrib/pyln-proto /opt/lightningd/contrib/pyln-proto
COPY --from=builder contrib/pyln-grpc-proto /opt/lightningd/contrib/pyln-grpc-proto
COPY --from=builder contrib/pyln-testing /opt/lightningd/contrib/pyln-testing
COPY --from=builder plugins/clnrest /opt/lightningd/plugins/clnrest
COPY --from=builder plugins/wss-proxy /opt/lightningd/plugins/wss-proxy

We will not have this issue with newer Dockerfile because now we are explicitly installing these dependencies via plugin's requirements.txt in builder-python stage.

RUN pip3 install -r plugins/clnrest/requirements.txt && \
    pip3 install -r plugins/wss-proxy/requirements.txt && \
    pip3 cache purge

[darioAnongba]

I didn't try the version v24.05 but does this means that this version still has the bug?

Edit: I just tried and it seems to work on 24.05. I added "developer" in the conf file that I wasn't adding before so that might be useful? In any case, it seems to work fine.

[ShahanaFarooqui]

I didn't try the version v24.05 but does this means that this version still has the bug?

Edit: I just tried and it seems to work on 24.05. I added "developer" in the conf file that I wasn't adding before so that might be useful? In any case, it seems to work fine.

No, user doesn't need to add/remove anything for dependencies to work. So the bug doesn't persist anymore.

However, clnrest will only be enabled if use configures clnrest-port option otherwise it will be disabled by lightningd.

Closing the issue now.

[kroese]

@ShahanaFarooqui This new image reports as v24.05-modded. Shouldnt the official images have unmodded versions so that the database migrations can take place?

[ShahanaFarooqui]

It is strange, looking into it. I am confident that the build ran on the unmodded version but somehow images are still showing modded.

[ShahanaFarooqui]

@kroese Opened a PR #7376 with possible (?) solution but need further advise from the expert @cdecker :).