Error when starting client in CLN docker setup: libssl.so.3: cannot open shared object

[cryptoteun]

I run CLN in a docker container. Building the client is going well. But when I add the plugin to the plugin folder and do a plugin rescan I get the following error:

/root/.lightning/plugins/watchtower-client: error while loading shared libraries: libssl.so.3: cannot open shared object file: No such file or directory
INFO    plugin-watchtower-client: Killing plugin: exited before replying to getmanifest

Any clue how to fix this?

[sr-gi]

Umm, I guess that depends on what the CLN image is built on (what the root image of the CLN docker container).

Here are some solutions for it, but which one to pick will depend on what's the base image:

https://stackoverflow.com/questions/54124906/openssl-error-while-loading-shared-libraries-libssl-so-3

[cryptoteun]

Thanks, will check. Image is based on debian:bullseye-slim btw, so if someone knows a solution already :-)

[mariocynicys]

@cryptoteun OpenSSL always have to be a pain in the butt. Not sure If you can build the plugin for musl target (either inside docker or on bare metal). If you can then you can use that binary on any machine/container with the same arch and it's gonna work with no dependency on any library (OpenSSL or others).

[sr-gi]

I'm currently trying to get rid of OpenSSL ATM, I think the dependency comes from httpmock

[sr-gi]

So I created #194, but the OpenSSL dependency is still kept given reqwest builds on it. I'll try to replicate the issue on my local setup.

[sr-gi]

@cryptoteun can you provide the full list of steps to reproduce, from what Docker image you're using to how are you installing and linking the plugin?

[cryptoteun]

I run a test setup on a Ubuntu 22.04. I build the teos client on this machine.

I also run my CLN node in docker, via the official image: elementsproject/lightningd:v22.11.1 Since it is vps i use trustedcoin instead of a full node.

After the teos client was build i tried two things:

• Linkin watchtower-client to the plugin folder
• Move the file to the plugin folder

Both cases the file is recognized by CLN, but the error occur.

[mariocynicys]

@cryptoteun Where do you compile the plugin? Inside docker or outside?

If outside, maybe try to:

• compile inside docker
• compile outside docker with musl target
• maybe install openssl3 inside docker :/

[sr-gi]

After the teos client was build i tried two things:

• Linkin watchtower-client to the plugin folder
• Move the file to the plugin folder

@cryptoteun Where do you compile the plugin? Inside docker or outside?

If outside, maybe try to:

• compile inside docker
• compile outside docker with musl target
• maybe install openssl3 inside docker :/

I was wondering the same.

If you're compiling from outside, can you share how you're linking (or moving) the plugin? For how you're describing this it feels like the plugin does compile given the dependencies are meet on the outside but cannot run because they are not on the inside (it's a different target).

[cryptoteun]

Thanks I'll check after the weekends.

[cryptoteun]

• with musl target

I build indeed outside docker. When i try to compile I run from one error to the other. I'm not very familiar with rust... I think I need some help with compiling with musl target. @mariocynicys , can you point me in the right direction?

The cln docker is based on debian:bullseye-slim

[mariocynicys]

So this is the command to use to build for musl x64 target: cargo build --release --target=x86_64-unknown-linux-musl ~But don't mind it anyway, I just tried it and it failed and I remember it failing some time ago but forgot about it. It is an openssl issue as well (obviously).~

I would suggest using the same version of openssl inside and outside docker. And my guts tell me that if you install openssl 3 (which I think has libssl.so.3) inside docker, the plugin won't have that shared lib issue.

Edit: Following this https://github.com/rust-cross/rust-musl-cross/issues/33#issuecomment-781937298 (setting PKG_CONFIG_SYSROOT_DIR=/), I was able to build for musl successfully :D

I would suggest trying to install openssl 3 inside docker first though just to make sure it's not another issue we are having here.

[cryptoteun]

Ok, apparently Ubuntu 22.04 is using openssl3. But the CLN docker image (debian) is using openssl1.1

Based on you help I was able to conclude this. Now I have bulld the plugin on an older ubuntu server I had running. Moving this build to my node resulted in: plugin-watchtower-client: Watchtower client keys not found. Creating a fresh set 💯

Thanks for your patience and help!!

[sr-gi]

Sweet! I'll close this is there is nothing else pending.

Also, feel free to reach out in Discord so we can work out why CLN was not compiling on your setup in the first place.