search

grafana / opcua-datasource

An OPC UA datasource for reading from OPC UA servers (DA/HDA/AE) into Grafana directly
GNU Affero General Public License v3.0
54 stars 10 forks source link

Backend component crashes on recommended Docker image #44

Closed harvidsen closed 2 years ago

harvidsen commented 2 years ago

Hello! We are very excited about this plugin and really appreciate the work being put into this. Unfortunately I have encountered some difficulties running it in a production friendly way.

Problem

I am unable to run this plugin on the recommended Grafana Docker image. It is worth mentioning that the plugin also does not work on Grafana Cloud, which might be due to the same issue here. In both cases i see this

Screenshot from 2022-01-07 11-00-18

Reproduce

Run like so

docker run \
  -p 3000:3000 \
  --name=grafana \
  -e "GF_INSTALL_PLUGINS=grafana-opcua-datasource 1.1.4" \
  -e "GF_LOG_LEVEL=debug" \
  grafana/grafana-oss:8.3.3

go to your localhost and try to "Save and Test" a datasource with url opc.tcp://opcua.prediktor.com:4880. Here are some relevant lines from the logs

...
t=2022-01-07T12:57:39+0000 lvl=dbug msg="Loading plugin" logger=plugin.loader path=/var/lib/grafana/plugins/grafana-opcua-datasource/plugin.json
t=2022-01-07T12:57:40+0000 lvl=dbug msg="Plugin signature valid" logger=plugin.loader id=grafana-opcua-datasource
t=2022-01-07T12:57:40+0000 lvl=dbug msg="Plugin has valid signature" logger=plugin.signature.validator id=grafana-opcua-datasource
t=2022-01-07T12:57:40+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-opcua-datasource
t=2022-01-07T12:57:40+0000 lvl=dbug msg="starting plugin" logger=plugin.initializer pluginID=grafana-opcua-datasource path=/var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64 args=[/var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64]
t=2022-01-07T12:57:40+0000 lvl=dbug msg="plugin started" logger=plugin.initializer pluginID=grafana-opcua-datasource path=/var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64 pid=34
t=2022-01-07T12:57:40+0000 lvl=dbug msg="waiting for RPC address" logger=plugin.initializer pluginID=grafana-opcua-datasource path=/var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64
t=2022-01-07T12:57:40+0000 lvl=dbug msg="/var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory" logger=plugin.initializer pluginID=grafana-opcua-datasource
t=2022-01-07T12:57:40+0000 lvl=eror msg="Could not start plugin" logger=plugin.manager pluginId=grafana-opcua-datasource err="Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
t=2022-01-07T12:57:40+0000 lvl=dbug msg="plugin process exited" logger=plugin.initializer pluginID=grafana-opcua-datasource path=/var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64 pid=34 error="exit status 127"
...

Thoughts

It appears that the backend component is not entirely self contained. This works when using the old Ubuntu Grafana image, so it is likely a problem with the differing libc requirements.

alydersen commented 2 years ago

Thanks @harvidsen for your excellent issue reporting! @sleiban: Any thoughts on this?

sleiban commented 2 years ago

I suspect this has the same cause as #41. We are looking into it.

harvidsen commented 2 years ago

I suspect this has the same cause as #41. We are looking into it.

I agree. However, plugin signature issues due to ./dbs and ./logs directories being created as mentioned in #41 seems like a different problem. When deleting the MANIFEST file and running plugin as unsigned I still get the Plugin unavailable error.

I propose letting #41 consider plugin signature issues and this issue consider the shared library problem in the logs above:

...
t=2022-01-07T12:57:40+0000 lvl=dbug msg="/var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory" logger=plugin.initializer pluginID=grafana-opcua-datasource
...
srclosson commented 2 years ago

Lots has changed. I would try the latest 1.1.5 release, and respond back if this is still an issue?

harvidsen commented 2 years ago

Yes the issue is still there. Running

docker run -p 3000:3000 --name grafana -e "GF_INSTALL_PLUGINS=grafana-opcua-datasource 1.1.5" -e "GF_LOG_LEVEL=debug" -e "GF_PLUGIN_GRAFANA_OPCUA_DATASOURCE=/var/lib/grafana-opcua-datasource" grafana/grafana-oss:8.2.6

still results in the same behaviour. I switched to the latest image which is not in pre-release, and added configuration for the logging directory. I see the following relevant log lines

...
t=2022-02-17T08:14:04+0000 lvl=dbug msg="Loading plugin" logger=plugins path=/var/lib/grafana/plugins/grafana-opcua-datasource/plugin.json
t=2022-02-17T08:14:04+0000 lvl=dbug msg="Getting signature state of plugin" logger=plugins plugin=grafana-opcua-datasource isBackend=true
t=2022-02-17T08:14:04+0000 lvl=dbug msg="Verifying contents of plugin manifest" logger=plugins plugin=grafana-opcua-datasource
t=2022-02-17T08:14:04+0000 lvl=dbug msg="Plugin signature valid" logger=plugins id=grafana-opcua-datasource
t=2022-02-17T08:14:04+0000 lvl=dbug msg="Initial plugin loading done" logger=plugins
t=2022-02-17T08:14:04+0000 lvl=dbug msg="Found plugin" logger=plugins id=grafana-opcua-datasource signature=valid hasRoot=false
t=2022-02-17T08:14:04+0000 lvl=dbug msg="Plugin has valid signature" logger=plugins id=grafana-opcua-datasource
t=2022-02-17T08:14:04+0000 lvl=dbug msg="Attempting to add plugin" logger=plugins id=grafana-opcua-datasource
t=2022-02-17T08:14:04+0000 lvl=dbug msg="Registering backend plugin" logger=plugins.backend pluginId=grafana-opcua-datasource
t=2022-02-17T08:14:04+0000 lvl=dbug msg="Backend plugin registered" logger=plugins.backend pluginId=grafana-opcua-datasource
t=2022-02-17T08:14:04+0000 lvl=dbug msg="starting plugin" logger=plugins.backend pluginId=grafana-opcua-datasource path=/var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64 args=[/var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64]
t=2022-02-17T08:14:04+0000 lvl=dbug msg="plugin started" logger=plugins.backend pluginId=grafana-opcua-datasource path=/var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64 pid=34
t=2022-02-17T08:14:04+0000 lvl=dbug msg="waiting for RPC address" logger=plugins.backend pluginId=grafana-opcua-datasource path=/var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64
t=2022-02-17T08:14:04+0000 lvl=dbug msg="/var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory" logger=plugins.backend pluginId=grafana-opcua-datasource
t=2022-02-17T08:14:04+0000 lvl=eror msg="Failed to start plugin" logger=plugins.backend pluginId=grafana-opcua-datasource error="Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
t=2022-02-17T08:14:04+0000 lvl=dbug msg="plugin process exited" logger=plugins.backend pluginId=grafana-opcua-datasource path=/var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64 pid=34 error="exit status 127"
t=2022-02-17T08:14:04+0000 lvl=info msg="Registering plugin" logger=plugins id=grafana-opcua-datasource
t=2022-02-17T08:14:04+0000 lvl=dbug msg="Successfully added plugin" logger=plugins id=grafana-opcua-datasource
...

Using Dynamic Program Loader ldd on the backend executable gives 

bash-5.1$ ldd /var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64
    /lib64/ld-linux-x86-64.so.2 (0x7fcd94c54000)
    libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7fcd94c54000)
    libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7fcd94c54000)
Error loading shared library libstdc++.so.6: No such file or directory (needed by /var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64)
    libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7fcd94c54000)
    libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7fcd94c16000)
    libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7fcd94c54000)
Error relocating /var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64: _ZNSs6appendEPKcm: symbol not found
Error relocating /var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64: _ZSt20__throw_length_errorPKc: symbol not found
Error relocating /var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64: _ZSt7getlineIcSt11char_traitsIcESaIcEERSt13basic_istreamIT_T0_ES7_RSbIS4_S5_T1_ES4_: symbol not found
Error relocating /var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64: _ZNSt11logic_errorD1Ev: symbol not found
Error relocating /var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64: _ZNSt18basic_stringstreamIcSt11char_traitsIcESaIcEEC1ERKSsSt13_Ios_Openmode: symbol not found
Error relocating /var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64: _ZNSolsEi: symbol not found
....

and similar symbol not found messages below.

I can try adding libstc++6 with apk as root user in the container. And add the file to LD_LIBRARY_PATH

bash-5.1# apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
v3.14.3-108-g6a5ae2cc7d [https://dl-cdn.alpinelinux.org/alpine/v3.14/main]
v3.14.3-105-gf644c9d3b0 [https://dl-cdn.alpinelinux.org/alpine/v3.14/community]
OK: 14964 distinct packages available
bash-5.1# apk add libstdc++6
(1/1) Installing libstdc++6 (6.4.0-r12)
Executing glibc-bin-2.30-r0.trigger
/usr/glibc-compat/sbin/ldconfig: /usr/glibc-compat/lib/ld-linux-x86-64.so.2 is not a symbolic link

OK: 28 MiB in 35 packages
bash-5.1# cd /
bash-5.1# find / -name libstdc++.so.6
/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/libstdc++.so.6
bash-5.1# export LD_LIBRARY_PATH="/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0"

Then ldd works

bash-5.1# ldd /var/lib/grafana/plugins/grafana-opcua-datasource/gpx_opcua_linux_amd64
    /lib64/ld-linux-x86-64.so.2 (0x7fe4b5d64000)
    libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7fe4b5d64000)
    libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7fe4b5d64000)
    libstdc++.so.6 => /usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/libstdc++.so.6 (0x7fe4b5beb000)
    libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7fe4b5d64000)
    libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7fe4b5bd1000)
    libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7fe4b5d64000)

My guess is this has something to do with the build specification. Is there a convenient way to add the shared library libstdc++.so.6 to the linux system dependencies?

I apologize for not being able to help more! I am not familiar with dotnet applications or the grafana build system.

srclosson commented 2 years ago

Okay, you will need to use the ubuntu docker image. Not the alpine one, as the plugin was linked against the standard libc, and alpine uses something different. Also, you need to change the environment variable. Try this:

docker run -p 3000:3000 --name grafana -e "GF_INSTALL_PLUGINS=grafana-opcua-datasource 1.1.5" -e "GF_LOG_LEVEL=debug" -e "GF_PLUGIN_DATA_DIR=/var/lib/grafana/grafana-opcua-datasource" grafana/grafana-oss:8.2.6-ubuntu
harvidsen commented 2 years ago

Yes, I think that is also why it fails on Grafana Cloud. However, if you don't want to prioritize supporting both docker environments we can close this issue with the solution being "use ubuntu docker image instead".

Regarding the environment variable

"GF_PLUGIN_DATA_DIR=/var/lib/grafana/grafana-opcua-datasource"

should it not be 

"GF_PLUGIN_GRAFANA_OPCUA_DATASOURCE_DATA_DIR=/var/lib/grafana/grafana-opcua-datasource"

?

harvidsen commented 2 years ago

I see that GF_PLUGIN_DATA_DIR works with v1.1.5. However, it might be more intuitive according to the Grafana docs to change it to the GF_<SectionName>_<KeyName> format.

Anyways, I will close this issue since it seems like supporting the plugin on the alpine image won't be prioritized.

(I have managed a production friendly workaround on our NixOS servers, so I am happy :smile:)