search

quickwit-oss / quickwit-datasource

Quickwit data source for Grafana
GNU Affero General Public License v3.0
41 stars 10 forks source link

Unable to properly install the 0.3.0 with grafana helm chart - it gets downloaded, installed, run and then overridden by the 0.2.4 from official plugin catalog #44

Closed kikokikok closed 7 months ago

kikokikok commented 7 months ago

Hi,

Following the documentation to download and run external grafana plugins, I was able to see it was properly downloaded at helm deployment time as shown in the logs:

2024-01-14T17:15:07.582415949Z ✔ Downloaded and extracted quickwit-quickwit-datasource v0.3.0 zip successfully to /var/lib/grafana/plugins/quickwit-quickwit-datasource
2024-01-14T17:15:07.582497033Z Please restart Grafana after installing or removing plugins. Refer to Grafana documentation for instructions if necessary.

The plugin seems to be properly initialized:

2024-01-14T17:15:09.377098358Z logger=local.finder t=2024-01-14T17:15:09.376027472Z level=debug msg="Loading plugin" path=/var/lib/grafana/plugins/quickwit-quickwit-datasource/plugin.json
2024-01-14T17:15:09.460300900Z logger=plugins.signature t=2024-01-14T17:15:09.460216857Z level=debug msg="Plugin signature valid" id=quickwit-quickwit-datasource
2024-01-14T17:15:09.460331025Z logger=plugin.signature.validator t=2024-01-14T17:15:09.460316358Z level=debug msg="Plugin has valid signature" id=quickwit-quickwit-datasource
2024-01-14T17:15:09.463630727Z logger=plugins.registration t=2024-01-14T17:15:09.463570434Z level=info msg="Plugin registered" pluginId=quickwit-quickwit-datasource
2024-01-14T17:15:09.463705436Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.46367231Z level=debug msg="starting plugin" path=/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64 args=[/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64]
2024-01-14T17:15:09.464339526Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.463979272Z level=debug msg="plugin started" path=/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64 pid=33
2024-01-14T17:15:09.464370859Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.464021939Z level=debug msg="waiting for RPC address" plugin=/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64
2024-01-14T17:15:09.471223265Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.471165681Z level=debug msg=Profiler enabled=false
2024-01-14T17:15:09.471238973Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.471221973Z level=debug msg=Tracing enabled=false propagation=
2024-01-14T17:15:09.471261057Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.471241223Z level=debug msg="Serving plugin" plugins="[stream diagnostics resource data]"
2024-01-14T17:15:09.471573977Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.471518185Z level=debug msg="using plugin" version=2
2024-01-14T17:15:09.471694520Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.471620519Z level=debug msg="plugin address" address=/tmp/plugin3134522783 network=unix
2024-01-14T17:15:09.472412611Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.472347777Z level=debug msg="Successfully started backend plugin process"
2024-01-14T17:15:09.472528362Z logger=plugin.loader source=external t=2024-01-14T17:15:09.47249507Z level=debug msg="Plugin source loaded" plugins=quickwit-quickwit-datasource duration=96.626891ms
2024-01-14T17:15:09.472544279Z logger=plugin.store t=2024-01-14T17:15:09.47251682Z level=info msg="Plugins loaded" count=56 duration=127.512258ms

Then for some reasons it gets stopped and the version 0.2.4 gets downloaded and installed instead

2024-01-14T17:15:16.996485963Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:16.996325795Z level=debug msg="Stopping plugin process"
2024-01-14T17:15:16.997524058Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:16.997064386Z level=debug msg="Plugin server exited"
2024-01-14T17:15:16.997639309Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:16.997593433Z level=info msg="plugin process exited" plugin=/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64 id=33
2024-01-14T17:15:16.997646309Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:16.9976196Z level=debug msg="plugin exited"
2024-01-14T17:15:16.997647851Z logger=plugins.deregister t=2024-01-14T17:15:16.997623767Z level=debug msg="Plugin unregistered" pluginId=quickwit-quickwit-datasource
2024-01-14T17:15:17.005295806Z logger=plugins.external.registration t=2024-01-14T17:15:17.005210972Z level=debug msg="Skipping HasExternalService call. The feature is behind a feature toggle and needs to be enabled."
2024-01-14T17:15:17.005348932Z logger=plugin.repository t=2024-01-14T17:15:17.005277264Z level=debug msg="Installing plugin from https://grafana.net/api/plugins/quickwit-quickwit-datasource/versions/0.2.4/download"
2024-01-14T17:15:17.473611708Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:17.473515582Z level=debug msg="Plugin decommissioned"
2024-01-14T17:15:22.683321927Z logger=installer.fs t=2024-01-14T17:15:22.683203551Z level=info msg="Downloaded and extracted quickwit-quickwit-datasource v0.2.4 zip successfully to /var/lib/grafana/plugins/quickwit-quickwit-datasource"
2024-01-14T17:15:22.683378386Z logger=local.finder t=2024-01-14T17:15:22.683345344Z level=debug msg="Loading plugin" path=/var/lib/grafana/plugins/quickwit-quickwit-datasource/plugin.json
2024-01-14T17:15:22.745257541Z logger=plugins.registration t=2024-01-14T17:15:22.745192832Z level=info msg="Plugin registered" pluginId=quickwit-quickwit-datasource
2024-01-14T17:15:22.745341250Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.745293625Z level=debug msg="starting plugin" path=/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64 args=[/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64]
2024-01-14T17:15:22.746285135Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.74624576Z level=debug msg="plugin started" path=/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64 pid=46
2024-01-14T17:15:22.746304427Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.746279843Z level=debug msg="waiting for RPC address" plugin=/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64
2024-01-14T17:15:22.753273709Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.75317925Z level=debug msg=Profiler enabled=false
2024-01-14T17:15:22.753461378Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.753392502Z level=debug msg=Tracing enabled=false propagation=
2024-01-14T17:15:22.753466253Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.753433836Z level=debug msg="Serving plugin" plugins="[stream diagnostics resource data]"
2024-01-14T17:15:22.753762422Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.753711214Z level=debug msg="using plugin" version=2
2024-01-14T17:15:22.753804881Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.753788089Z level=debug msg="plugin address" address=/tmp/plugin3988986528 network=unix
2024-01-14T17:15:22.754194135Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.75416401Z level=debug msg="Successfully started backend plugin process"
2024-01-14T17:15:22.754250344Z logger=plugin.loader source=external t=2024-01-14T17:15:22.754233344Z level=debug msg="Plugin source loaded" plugins=quickwit-quickwit-datasource duration=71.002793ms

After that I have the Datasource creation automation as shown here

2024-01-14T17:15:22.802052308Z logger=datasources t=2024-01-14T17:15:22.802005141Z level=debug msg="Received command to add data source" url=http://quickwit-searcher.quickwit.svc.cluster.local:7280/api/v1/otel-traces-v0_7/jaeger
2024-01-14T17:15:22.802058683Z logger=datasource t=2024-01-14T17:15:22.80201985Z level=debug msg="Applying default URL parsing for this data source type" type=jaeger url=http://quickwit-searcher.quickwit.svc.cluster.local:7280/api/v1/otel-traces-v0_7/jaeger
2024-01-14T17:15:22.847466455Z logger=datasources t=2024-01-14T17:15:22.847396913Z level=debug msg="Received command to add data source" url=http://quickwit-indexer.quickwit.svc.cluster.local:7280/api/v1/
2024-01-14T17:15:22.847472955Z logger=datasource t=2024-01-14T17:15:22.847409996Z level=debug msg="Applying default URL parsing for this data source type" type=quickwit-quickwit-datasource url=http://quickwit-indexer.quickwit.svc.cluster.local:7280/api/v1/

And here are the values.yml meaningful customization

grafana:
  grafana.ini:
    plugins:
      allow_loading_unsigned_plugins: quickwit-quickwit-datasource
  plugins:
    - "https://github.com/quickwit-oss/quickwit-datasource/releases/download/v0.3.0/quickwit-quickwit-datasource-0.3.0.zip;quickwit-quickwit-datasource"
  env:
    GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: quickwit-quickwit-datasource
    GF_LOG_LEVEL: debug
    GF_DEFAULT_APP_MODE: development

And the datasources:

  1. Jaeger:

    {
"name": "Jaeger",
"type": "jaeger",
"typeName": "Jaeger",
"typeLogoUrl": "public/app/plugins/datasource/jaeger/img/jaeger_logo.svg",
"access": "proxy",
"url": "http://quickwit-searcher.quickwit.svc.cluster.local:7280/api/v1/otel-traces-v0_7/jaeger",
"user": "",
"database": "",
"basicAuth": false,
"isDefault": false,
"jsonData": { "timeout": -2 },
"readOnly": false
}

  2. Quickwit

    {
"name": "Quickwit",
"type": "quickwit-quickwit-datasource",
"typeName": "Quickwit",
"typeLogoUrl": "public/plugins/quickwit-quickwit-datasource/img/logo.svg",
"access": "proxy",
"url": "http://quickwit-indexer.quickwit.svc.cluster.local:7280/api/v1/",
"user": "",
"database": "",
"basicAuth": false,
"isDefault": false,
"jsonData": {
"index": "otel-logs-v0_6",
"logLevelField": "level",
"logMessageField": "body.message",
"timeField": "timestamp_nanos",
"timeOutputFormat": "unix_timestamp_nanos"
},
"readOnly": false
}

    Any idea of what I might be doing wrong here ?

Thank you !

fmassot commented 7 months ago

Hi, strange; I just tried to start a new Grafana instance, and it worked. Maybe it's related to the source that you create automatically. In the new 0.3.0 version, you don't have to set the timeField and timeOutputFormat.

Let me share my config; I'm using the Grafana helm chart 7.0.21, and here are my values.yaml

grafana:
  resources:
    requests:
      cpu: 10m
      memory: 128Mi

  grafana.ini:
    date_formats:
      default_timezone: UTC

  deploymentStrategy:
    type: Recreate

  persistence:
    type: pvc
    enabled: true
    accessModes:
      - ReadWriteOnce
    size: 10Gi

  datasources:
    datasources.yaml:
      apiVersion: 1
      datasources: []

  dashboards: {}

  plugins:
    - https://github.com/quickwit-oss/quickwit-datasource/releases/download/v0.3.0/quickwit-quickwit-datasource-0.3.0.zip;quickwit-quickwit-datasource

I will try later with your config.

kikokikok commented 7 months ago

Merci for the prompt response !

I am using the kube-prometheus-stack latest version 55.8.1 The grafana sub-chart is referenced as 7.1.x

Only other difference I know of is an nginx ingress with a ClusterIssuer and cert-manager configured

fmassot commented 7 months ago

I updated the Grafana chart to 7.2, and it worked :/

The plugin 0.3.0 is signed so normally, you don't need the GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS nor allow_loading_unsigned_plugins.

kikokikok commented 7 months ago

I updated the Grafana chart to 7.2, and it worked :/

The plugin 0.3.0 is signed so normally, you don't need the GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS nor allow_loading_unsigned_plugins.

I didn't even think about trying to upgrade... I'll try that. For the unsigned, yes I figured it out once switching to debug grafana.

Thanks !

kikokikok commented 7 months ago

I was not able to make it work with grafana 7.2.1 subchart unfortunately. Would be curious to know if given the same packaged chart anyone could confirm or be more successful maybe which would be a sign that there might be a side effect of something else. FWIW, I am running on a local k3s running over a LimaVM on a arm64 architecture.

So I have rebundled the 55.8.1 to upgrade the grafana subchart to 7.2.x which translates into 7.2.1 kube-prometheus-stack-55.8.2.tgz

values.yaml is set with this option only ` plugins:

fmassot commented 7 months ago

@kikokikok the 0.3.0 is now officially released: https://grafana.com/grafana/plugins/quickwit-quickwit-datasource/

Let me know if this is still not working.

kikokikok commented 7 months ago

Thanks @fmassot I do confirm it works now. Interestingly enough I kind of figured out how to make it work also with the one downloaded from github. The trick is to restart grafana pod after plugin initializaton. I haven't looked further into it since you released it to the Grafana plugin catalog.

Thanks for the help !