Bad Gateway Error 502 - Plugin setup

[cse-wdu]

Hi All,

I'm trying to add the osisoftpi-grafana plugin. Using version 4.2 of the plugin and version 9.3.2 grafana. Was originally working then when i upgraded to version 10 and then applied the enterprise license, adding the plugin results in a bad gateway error everytime.

The PI server / Grafana is on the same subnet / VLAN. There is no routing between them. There are no firewalls between them. Windows firewall has been disabled. I can see the request hitting the PI server and the server is providing the response. however, it isnt represented back in grafana for some reason. The grafana logs show the 502 error.

CORSOrigin has been configured on the PI server for the grafana address.

I've since reinstalled grafana onto a new vm - version 9.3.2 and the error is still here. Can't remember what the configs were before i upgraded from 9.3.2 to 10.

Both systems are windows machine and using basic authentication. They are VM's running server 2016/2019.

Was wondering whether the developers of this plugin could potentially provide some insights into what is happening here?

Any help would be highly appreciated. Been working with Grafana / Pi on this issue without much luck.

Screenshots below

Config for the gateway. It is authenticating fine - the PI server is providing a response - see pics below

Request from grafana to the PI API user

Authenticated user

Response by PI API server when saving/testing the plugin

Traffic from the plugin when saving the plugin. The requests result in 502 error. I'm able to copy / paste the URL into the browser and it works fine from the server

Configuration of the PI API server

Grafana logs showing broken connection

Any help highly appreciated.

Thanks

[coderReview]

Can you share a screenshot of the installed OSISoft plugin which shows the version? Looking at the configuration screenshot, it doesn't seem to be the latest version. Also, the plugin is not yet validated with Grafana 10.x, so I cannot guarantee it will work. Leave the timeout empty. Custom headers are not needed or used by the plugin.

Em sex., 28 de jul. de 2023 às 08:44, cse-wdu @.***> escreveu:

Hi All,

I'm trying to add the osisoftpi-grafana plugin. Using version 4.2 of the plugin and version 9.3.2 grafana. Was originally working then when i upgraded to version 10 and then applied the enterprise license, adding the plugin results in a bad gateway error everytime.

The PI server / Grafana is on the same subnet / VLAN. There is no routing between them. There are no firewalls between them. Windows firewall has been disabled. I can see the request hitting the PI server and the server is providing the response. however, it isnt represented back in grafana for some reason. The grafana logs show the 502 error.

CORSOrigin has been configured on the PI server for the grafana address.

I've since reinstalled grafana onto a new vm - version 9.3.2 and the error is still here. Can't remember what the configs were before i upgraded from 9.3.2 to 10.

Both systems are windows machine and using basic authentication. They are VM's running server 2016/2019.

Was wondering whether the developers of this plugin could potentially provide some insights into what is happening here?

Any help would be highly appreciated. Been working with Grafana / Pi on this issue without much luck.

Screenshots below

Config for the gateway. It is authenticating fine - the PI server is providing a response - see pics below [image: pic1] https://user-images.githubusercontent.com/89962643/256810373-fd77233e-8d05-49d4-8e53-4b7d3bb35bb8.png Request from grafana to the PI API user

[image: pic4] https://user-images.githubusercontent.com/89962643/256810395-019b6bda-3e1c-4917-b8a6-a96109ddef36.png Authenticated user

[image: pic3] https://user-images.githubusercontent.com/89962643/256810387-f78ca81c-2590-484d-a7c3-bda227ff3788.png Response by PI API server when saving/testing the plugin

[image: pic2] https://user-images.githubusercontent.com/89962643/256810381-3b1d8a1a-cff1-4b24-9cd9-a1f1b78d9b9b.png Traffic from the plugin when saving the plugin. The requests result in 502 error. I'm able to copy / paste the URL into the browser and it works fine from the server

[image: pic5] https://user-images.githubusercontent.com/89962643/256810398-9b38aa61-e6b8-4e6c-99b2-a49dd7e6ab48.png Configuration of the PI API server

[image: pic6] https://user-images.githubusercontent.com/89962643/256810404-0e8871c8-477e-4bcb-96de-0395a5404b57.png Grafana logs showing broken connection

[image: pic7] https://user-images.githubusercontent.com/89962643/256810708-0891817c-e539-4447-bb18-26b5df3638d4.png

Any help highly appreciated.

Thanks

— Reply to this email directly, view it on GitHub https://github.com/GridProtectionAlliance/osisoftpi-grafana/issues/133, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNGUQQ2VYN7KOYTR2N24PDXSOQYZANCNFSM6AAAAAA23MKR44 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[cse-wdu]

Hi @coderReview,

I've done a full reinstall and got 9.3.2 installed and version 4.2 of the osisoft plugin installed. Screenshots below show the details as requested. I'm in process of asking IT to help with restoring from a backup to a previous version to see if we can get it fixed.

Any help appreciated.

Thanks

Version of Plugin

Version of Grafana

Gateway Error

Console log

[cse-wdu]

Hi All,

Version with TLS verify turned off

[coderReview]

You can enable full debug in Grafana logs. That might give you a hint on what is going on.

[coderReview]

Is Grafana running on linux or windows ?

[cse-wdu]

Hi @coderReview,

It is a Windows installation. Please see the debug logs below. The server does not have access to the internet hence why external IP's are returning 502 (blocked by edge firewall). Typically 502 errors are the result of the PI server blocking the response, but i can see from the PI logs the server is returning the response.

logger=http.server t=2023-08-02T10:05:53.7397254+10:00 level=info msg="Successful Login" User=admin@localhost logger=context userId=1 orgId=1 uname=admin t=2023-08-02T10:05:54.66515+10:00 level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=192.168.237.102 time_ms=1 duration=1.1062ms size=0 referer= handler=/api/live/ws logger=context userId=1 orgId=1 uname=admin t=2023-08-02T10:05:54.7051365+10:00 level=error msg="not found" error="not found" remote_addr=192.168.237.102 traceID= logger=context userId=1 orgId=1 uname=admin t=2023-08-02T10:05:54.7051365+10:00 level=info msg="Request Completed" method=GET path=/api/recording-rules/writer status=404 remote_addr=192.168.237.102 time_ms=0 duration=895.2µs size=36 referer="http://grafana.COMPANY.com/?orgId=1" handler=/api/recording-rules/writer/ logger=context userId=1 orgId=1 uname=admin t=2023-08-02T10:06:04.7468761+10:00 level=info msg="Request Completed" method=POST path=/api/ma/events status=201 remote_addr=192.168.237.102 time_ms=85 duration=85.2945ms size=4 referer="http://grafana.COMPANY.com/?orgId=1" handler=/api/ma/events logger=context userId=1 orgId=1 uname=admin t=2023-08-02T10:06:05.5467926+10:00 level=error msg="Proxy request failed" err="read tcp 192.168.237.102:59434->34.120.177.193:443: wsarecv: An existing connection was forcibly closed by the remote host." logger=context userId=1 orgId=1 uname=admin t=2023-08-02T10:06:05.5467926+10:00 level=error msg="Request Completed" method=GET path=/api/gnet/plugins status=502 remote_addr=192.168.237.102 time_ms=141 duration=141.6275ms size=0 referer=http://grafana.COMPANY.com/connections/add-new-connection handler=/api/gnet/* logger=context userId=1 orgId=1 uname=admin t=2023-08-02T10:06:11.7889585+10:00 level=error msg="Proxy request failed" err="read tcp 192.168.237.102:60376->34.120.177.193:443: wsarecv: An existing connection was forcibly closed by the remote host."

PI Plugin starts here

logger=context userId=1 orgId=1 uname=admin t=2023-08-02T10:06:11.7889585+10:00 level=error msg="Request Completed" method=GET path=/api/gnet/plugins/gridprotectionalliance-osisoftpi-datasource status=502 remote_addr=192.168.237.102 time_ms=148 duration=148.9745ms size=0 referer=http://grafana.COMPANY.com/connections/datasources/edit/caa4129d-7a2f-494b-8639-d94cb3b40f6b handler=/api/gnet/ logger=data-proxy-log t=2023-08-02T10:06:44.901691+10:00 level=info msg="Proxying incoming request" userid=1 orgid=1 username=admin datasource=gridprotectionalliance-osisoftpi-datasource uri="/api/datasources/proxy/uid/caa4129d-7a2f-494b-8639-d94cb3b40f6b/dataservers?name=COMPANYPI" method=GET body= logger=data-proxy-log t=2023-08-02T10:06:44.9028032+10:00 level=info msg="Proxying incoming request" userid=1 orgid=1 username=admin datasource=gridprotectionalliance-osisoftpi-datasource uri="/api/datasources/proxy/uid/caa4129d-7a2f-494b-8639-d94cb3b40f6b/assetservers?path=\\COMPANYAF" method=GET body= logger=data-proxy-log t=2023-08-02T10:06:44.9059988+10:00 level=info msg="Proxying incoming request" userid=1 orgid=1 username=admin datasource=gridprotectionalliance-osisoftpi-datasource uri=/api/datasources/proxy/uid/caa4129d-7a2f-494b-8639-d94cb3b40f6b/ method=GET body= logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/uid/caa4129d-7a2f-494b-8639-d94cb3b40f6b/dataservers remote_addr=192.168.237.102 referer=http://grafana.COMPANY.com/connections/datasources/edit/caa4129d-7a2f-494b-8639-d94cb3b40f6b t=2023-08-02T10:06:45.5420517+10:00 level=error msg="Proxy request failed" err="read tcp 192.168.237.102:52240->192.168.237.24:443: wsarecv: An existing connection was forcibly closed by the remote host." logger=context userId=1 orgId=1 uname=admin t=2023-08-02T10:06:45.5420517+10:00 level=error msg="Request Completed" method=GET path=/api/datasources/proxy/uid/caa4129d-7a2f-494b-8639-d94cb3b40f6b/dataservers status=502 remote_addr=192.168.237.102 time_ms=640 duration=640.9455ms size=0 referer=http://grafana.COMPANY.com/connections/datasources/edit/caa4129d-7a2f-494b-8639-d94cb3b40f6b handler=/api/datasources/proxy/uid/:uid/ logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/uid/caa4129d-7a2f-494b-8639-d94cb3b40f6b/ remote_addr=192.168.237.102 referer=http://grafana.COMPANY.com/connections/datasources/edit/caa4129d-7a2f-494b-8639-d94cb3b40f6b t=2023-08-02T10:06:45.5435246+10:00 level=error msg="Proxy request failed" err="read tcp 192.168.237.102:52241->192.168.237.24:443: wsarecv: An existing connection was forcibly closed by the remote host." logger=context userId=1 orgId=1 uname=admin t=2023-08-02T10:06:45.5435246+10:00 level=error msg="Request Completed" method=GET path=/api/datasources/proxy/uid/caa4129d-7a2f-494b-8639-d94cb3b40f6b/ status=502 remote_addr=192.168.237.102 time_ms=638 duration=638.4834ms size=0 referer=http://grafana.COMPANY.com/connections/datasources/edit/caa4129d-7a2f-494b-8639-d94cb3b40f6b handler=/api/datasources/proxy/uid/:uid logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/uid/caa4129d-7a2f-494b-8639-d94cb3b40f6b/assetservers remote_addr=192.168.237.102 referer=http://grafana.COMPANY.com/connections/datasources/edit/caa4129d-7a2f-494b-8639-d94cb3b40f6b t=2023-08-02T10:06:45.8780827+10:00 level=error msg="Proxy request failed" err="read tcp 192.168.237.102:52239->192.168.237.24:443: wsarecv: An existing connection was forcibly closed by the remote host." logger=context userId=1 orgId=1 uname=admin t=2023-08-02T10:06:45.8780827+10:00 level=error msg="Request Completed" method=GET path=/api/datasources/proxy/uid/caa4129d-7a2f-494b-8639-d94cb3b40f6b/assetservers status=502 remote_addr=192.168.237.102 time_ms=975 duration=975.751ms size=0 referer=http://grafana.COMPANY.com/connections/datasources/edit/caa4129d-7a2f-494b-8639-d94cb3b40f6b handler=/api/datasources/proxy/uid/:uid/*

[coderReview]

Hello, What is the version of OSISoft PI and Piwebapi? I think there were issues with previous versions of Piwebapi.

Message ID: @.*** com>

[cse-wdu]

@coderReview - We havent changed PI / PI API versions between it working and then suddenly stop working when we applied the enterprise license. Though we may have had a patch applied.

[coderReview]

Check this https://github.com/GridProtectionAlliance/osisoftpi-grafana/issues/109. There are changes to the API in OSISoft. You can try doing some tests with the query generated by the Grafana plugin and use it against the PiWebApi directly. You can use Postman for that.

Message ID: @.*** com>

[cse-wdu]

The queries work fine from the browser and also via python requests. We actively use the web api internally with no issues. The only issue has been with Grafana unfortunately. Otherwise would've been easy to point towards PI Support.

[complacentsee]

I know I had similar issues during an upgrade and found that I either had a trailing / or was missing a trailing / from my address for URL in the configuration.