grafana / falconlogscale-datasource

Falcon LogScale data source for Grafana
Apache License 2.0
3 stars 1 forks source link

Cannot set timeout when using data source provisioning #236

Open van-vliet opened 7 months ago

van-vliet commented 7 months ago

What happened: We have a dashboard that uses the Logscale datasource to power some panels. Our querys toward Logscale time out after 1 minute. See screenshots: image image

The queries are heavy, so I want to increase the timeout.

We use provisioning to setup datasources. The official provisioning documentation mentioned a common timeout setting here. The plugin documentation mentions provisioning, but the example does not show the exact format for specifying the timeout.

I tried including the timeout in two different places in my datasource provisioning settings. I restarted Grafana between each attempt.

    - name: FalconLogscale
      type: grafana-falconlogscale-datasource
      url: https://cloud.humio.com
      uid: "<REDACTED>"
      jsonData:
        authenticateWithToken: true
        defaultRepository: <REDACTED>
        timeout: 300
      secureJsonData:
        accessToken: $__file{/etc/secrets/datasources/logscale-api-key}

and

    - name: FalconLogscale
      type: grafana-falconlogscale-datasource
      url: https://cloud.humio.com
      uid: "<REDACTED>"
      timeout: 300
      jsonData:
        authenticateWithToken: true
        defaultRepository: <REDACTED>
      secureJsonData:
        accessToken: $__file{/etc/secrets/datasources/logscale-api-key}

After performing the change, I get the same error as before. The stats show a total request time of 1 minute, which makes me believe the configuration I've applied has not changed anything.

What you expected to happen: I expected that the README file for this repo described the possible configuration values and showed an example of how to provision them.

I expected one of the two provisioning configurations I applied to have changed the HTTP timeout used towards Logscale. I expected this to allow my dashboard to load without timeout errors.

How to reproduce it (as minimally and precisely as possible):

  1. Provision a Logscale datasource using this plugin and the provisioning configuration
    - name: FalconLogscale
      type: grafana-falconlogscale-datasource
      url: https://cloud.humio.com
      uid: "<REDACTED>"
      timeout: 300
      jsonData:
        authenticateWithToken: true
        defaultRepository: <REDACTED>
      secureJsonData:
        accessToken: $__file{/etc/secrets/datasources/logscale-api-key}
  2. Create a dashboard that uses the Logscale datasource. Use one or more queries that are slow to respond.
  3. Load the dashboard and wait 1 minute.
  4. Observe that the dashboard fails to retrieve data. Press the little "inspect" button on a panel that failed to load Logscale data.
  5. In the "Stats" panel, notice that Total request time is 1.00 mins.

Screenshots

Anything else we need to know?: I looked through the code to find references to the correct name and placement of the the timeout property. I came across the plugin settings struct, where the timeout property is commented. This makes me wonder whether the plugin uses the value at all.

Environment:

bossinc commented 6 months ago

@van-vliet Thank you for creating this issue. You are correct that this plugin is not using the timeout value. The documentation you linked indicates that not all plugins support setting a timeout. LogScale queries do not timeout unless they are no longer being polled, so the query should run until completion. We will investigate this further and have a fix. Polling query docs