ROCm / rocprofiler-compute

Advanced Profiling and Analytics for AMD Hardware
https://rocm.docs.amd.com/projects/omniperf/en/latest/
MIT License
135 stars 49 forks source link

grafana-cli: failed to verify certificate: x509: certificate signed by unknown authority #434

Closed xyuan closed 4 weeks ago

xyuan commented 1 month ago

I am trying to build the grafana UI locally on my macOS/ubuntu machine, I have always failed with the following message,

_311.0 logger=settings t=2024-09-27T22:57:21.272514922Z level=info msg=Path Logs path=/var/log/grafana
311.0 logger=settings t=2024-09-27T22:57:21.272517381Z level=info msg=Path Plugins path=/var/lib/grafana/plugins
311.0 logger=settings t=2024-09-27T22:57:21.272526131Z level=info msg=Path Provisioning path=/etc/grafana/provisioning
311.0 logger=settings t=2024-09-27T22:57:21.272539506Z level=info msg=App mode production
311.5 Error: ✗ Get  https://grafana.com/api/plugins/michaeldmoore-multistat-panel/versions: tls: failed to verify certificate: x509: certificate signed by unknown authority
------
failed to solve: process /bin/sh -c apt-get install -y tzdata systemd apt-utils npm vim net-tools &&     /usr/sbin/grafana-cli plugins install michaeldmoore-multistat-panel &&     chown root:grafana /etc/grafana &&     cd /var/lib/grafana/plugins/omniperf_plugin &&     npm install &&     npm run build &&     apt-get autoremove -y &&     apt-get autoclean -y &&     sed -i \"s/  bindIp.*/  bindIp: 0.0.0.0/\" /etc/mongod.conf &&     mkdir -p /var/lib/grafana &&     touch /var/lib/grafana/grafana.lib &&     chown grafana:grafana /var/lib/grafana/grafana.lib did not complete successfully: exit code: 1_

I have generated the certificate using openssl, and setup the ini file like,

_492;tls_skip_verify_insecure = true 493 ;tls_client_cert =/Work/omniperf/grafana/client.pem 494 ;tls_client_key = /Work/omniperf/grafana/client.key 495 ;tls_clientca = /Work/omniperf/grafana/ca.pem

after looking into the Dockfile, this error is from,

48 /usr/sbin/grafana-cli plugins install michaeldmoore-multistat-panel && \

anyone can help to resolve this issue, or help to generate these certificate/key files?

coleramos425 commented 1 month ago

Linking related issue #405

coleramos425 commented 1 month ago

@njobypet could you please assign to an engineer? Thank you.

vstempen commented 1 month ago

Can you please describe your system configuration? Are you building on macOS or Ubuntu? Which version? Docker container? Which image? VirtualBox? Thanks.

xyuan commented 1 month ago

I am building on macos by following this webpage, https://rocm.docs.amd.com/projects/omniperf/en/latest/install/grafana-setup.html using docker.

some minor changes in the dockfile to use arm distribution instead of the original ubuntum distribution. with this change, I am able to pass most, except the grafana-cli update the apis with tls certification failure.

vstempen commented 4 weeks ago

It's not a solution but just a workaround, but you can add --no-check-certificate key to wget and --insecure to curl to pass the build. We are still working the issue. It seems docker container cannot find SSL certificates, when running on macos. Even I implicitly dump a certificate from a web-site to a file and then specify it as a parameter. Will update you as soon as we find a proper solution.

xyuan commented 4 weeks ago

yes, I add -k to curl and wget, and both work fine. but it is difficult for me to figure out how to workaround the grafana-cli command. I tried to download these apps directly, and using ADD in the dockerfile, but it still failed. maybe I did wrong, will try it again.

vstempen commented 4 weeks ago

Hi, can you attach output of "openssl s_client -connect dl.grafana.com:443 -showcerts" form your macos.

vstempen commented 4 weeks ago

In my case it was endpoint security service that was not letting openssl to get proper certificate chain from server. It was replacing the proper certificate with some default company certificate during handshake. Disabling the service helped.

xyuan commented 4 weeks ago

Hi, can you attach output of "openssl s_client -connect dl.grafana.com:443 -showcerts" form your macos.

I wasn't sure whether it is allowed to post this information publically on github, though

vstempen commented 4 weeks ago

OK, can you make sure there are no security services running on macOS and try again.

xyuan commented 4 weeks ago

I made changes to the Dockfile to use 'ADD' the pre-download plugin, and disable the wget, and curl ssl security check, and it works,

19 ADD plugins/omniperf_plugin /var/lib/grafana/plugins/omniperf_plugin 20 ADD plugins/aceiot-svg-panel /var/lib/grafana/plugins/aceiot-svg-panel 21 ADD plugins/ae3e-plotly-panel /var/lib/grafana/plugins/ae3e-plotly-panel 22 ADD plugins/grafana-image-renderer /var/lib/grafana/plugins/grafana-image-renderer 23 ADD plugins/michaeldmoore-multistat-panel /var/lib/grafana/plugins/michaeldmoore-multistat-panel 24 ADD plugins/natel-plotly-panel /var/lib/grafana/plugins/natel-plotly-panel

and

29 RUN apt-get update && \ 30 apt-get install -y adduser libfontconfig1 musl wget && \ 31 wget -q https://dl.grafana.com/enterprise/release/grafana-enterprise_10.4.3_arm64.deb --no-check-certificate && \ 32 dpkg -i grafana-enterprise_10.4.3_arm64.deb && \ 33 apt-get install -y gnupg curl && \ 34 curl -fksSL https://www.mongodb.org/static/pgp/server-7.0.asc | gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg --dearmor && \ 35 echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" | tee /etc/apt/sources.list. d/mongodb-org-7.0.list && \ 36 apt-get update && \ 37 apt-get install -y mongodb-org

these pre-download plugins have to be at the grafana/plugin directory.

vstempen commented 4 weeks ago

I don't think we can re-distribute Grafana plugins. You solution works, but disabling network proxy services on macOS system should work too. I cannot reproduce the problem when network proxy services are disabled.

xyuan commented 4 weeks ago

no worries about it, at least it is working for us now, thanks

vstempen commented 4 weeks ago

Can we close the issue then? Thanks.

xyuan commented 4 weeks ago

yes, please