Grafana warning msg="Could not create user agent" error="invalid user agent format"

[bahree]

Is there an existing issue for this?

• [X] I have searched the existing issues

What happened?

I just upgraded to the latest version (1.31.1) and not sure if this is related to that or if it has been for a while. Everything seems to be running OK but in docker I see a ton of these warnings.

grafana-1 | logger=plugin.context t=2024-11-03T15:17:05.894593108Z level=warn msg="Could not create user agent" error="invalid user agent format"

Expected Behavior

Given this is just a warning, I don't know if I should just ignore it or if there is a issue?

Steps To Reproduce

Goto any dashboard in Grafana

Relevant log output

grafana-1    | logger=plugin.context t=2024-11-03T15:16:23.551745422Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:23.553000485Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:23.553021765Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:23.556027669Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:23.562634752Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:23.806344093Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=live t=2024-11-03T15:16:24.007699806Z level=info msg="Initialized channel handler" channel=grafana/dashboard/uid/Y8upc6ZRk address=grafana/dashboard/uid/Y8upc6ZRk
grafana-1    | logger=plugin.context t=2024-11-03T15:16:24.183948854Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:29.400740522Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:29.400861311Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:29.405167549Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:29.405169016Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:29.453738319Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:29.587118493Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:29.595441147Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:29.602657338Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=live t=2024-11-03T15:16:29.800020584Z level=info msg="Initialized channel handler" channel=grafana/dashboard/uid/zm7wN6Zgz address=grafana/dashboard/uid/zm7wN6Zgz
grafana-1    | logger=plugin.context t=2024-11-03T15:16:29.961103218Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:29.961105111Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:30.017793316Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:30.035193074Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:30.510691838Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:35.753099527Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:35.753195794Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:35.759887537Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:35.760594549Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:35.780539525Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:36.012338187Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:36.017633019Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:36.017913772Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:16:36.019312402Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=infra.usagestats t=2024-11-03T15:16:41.093220738Z level=info msg="Usage stats are ready to report"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:00.601414158Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:00.606765943Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:00.606865982Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:00.612947449Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:00.625463106Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=live t=2024-11-03T15:17:00.827218337Z level=info msg="Initialized channel handler" channel=grafana/dashboard/uid/kOuP_Fggz address=grafana/dashboard/uid/kOuP_Fggz
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.00806124Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.033405329Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.039703493Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.055058555Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.140273217Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.245245022Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.245943806Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.249839969Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.266545043Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.33832449Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.452290235Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.464515532Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.489257306Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.49051619Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.547767296Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.656992947Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.68023559Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:01.698372648Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:03.350867118Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:03.357027509Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:03.357853671Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:05.657895616Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:05.658049472Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:05.658777809Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:05.664768062Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:05.701441782Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:05.859854816Z level=warn msg="Could not create user agent" error="invalid user agent format"
grafana-1    | logger=plugin.context t=2024-11-03T15:17:05.894593108Z level=warn msg="Could not create user agent" error="invalid user agent format"

Screenshots

No response

Additional data

No response

Type of installation

Docker

Version

v1.31.1

[JakobLichterfeld]

Thanks for reporting. Can you please share your docker-compose file (without your secrets)?

[bahree]

It really hasn't changed other than bumping up the postgres version. Shared below. I don't use MQTT, etc so have those removed.

services:
  teslamate:
    image: teslamate/teslamate:latest
    restart: always
    depends_on:
      - database
    environment:
      - ENCRYPTION_KEY=${API_TOKEN_ENCRYPTION_KEY}
      - DATABASE_USER=${TM_DB_USER}
      - DATABASE_PASS=${TM_DB_PASS}
      - DATABASE_NAME=${TM_DB_NAME}
      - DATABASE_HOST=database
      - DISABLE_MQTT=true
      - CHECK_ORIGIN=https://${FQDN_TM}
      - TZ=${TM_TZ}
    volumes:
      - ./import:/opt/app/import
    ports:
      - "4000:4000"

  database:
    image: postgres:17
    restart: always
    environment:
      - POSTGRES_USER=${TM_DB_USER}
      - POSTGRES_PASSWORD=${TM_DB_PASS}
      - POSTGRES_DB=${TM_DB_NAME}
    volumes:
      - teslamate-db:/var/lib/postgresql/data

  grafana:
    image: teslamate/grafana:latest
    restart: always
    environment:
      - DATABASE_USER=${TM_DB_USER}
      - DATABASE_PASS=${TM_DB_PASS}
      - DATABASE_NAME=${TM_DB_NAME}
      - DATABASE_HOST=database
      - GRAFANA_PASSWD=${GRAFANA_PW}
      - GF_SECURITY_ADMIN_USER=${GRAFANA_USER}
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PW}
      - GF_AUTH_ANONYMOUS_ENABLED=false
      - GF_SERVER_DOMAIN=${FQDN_TM}
      - GF_SERVER_ROOT_URL=https://${FQDN_TM}/grafana  # Correct root URL for Grafana
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
      - DATABASE_SSL_MODE=disable
    volumes:
      - teslamate-grafana-data:/var/lib/grafana
    ports:
      - "4001:3000"  # Expose Grafana on port 4001, but Grafana listens internally on port 3000

volumes:
  teslamate-db:
  teslamate-grafana-data:

[JakobLichterfeld]

It really hasn't changed other than bumping up the postgres version.

And you followed the https://docs.teslamate.org/docs/maintenance/upgrading_postgres and not just bump to 17, right?

Even if it seems unrelated, for security reasons I suggest adding according to the docs:

cap_drop:
  - all

Regarding your issue: I can not reproduce it on my end. I suggest you do a full backup, delete your Grafana volume and spin it up again as I assume you edited something in the past.

[b89dan]

I get the same error: logger=plugin.context t=2024-11-07T14:21:39.374989634Z level=warn msg="Could not create user agent" error="invalid user agent format" Looking around i see a similar issue is solved in grafana 11.3.0: https://github.com/grafana/grafana/issues/95006 Current Teslamate build uses grafana 11.2.3, so i guess we have to wait for the bump and see.

[JakobLichterfeld]

Looking around i see a similar issue is solved in grafana 11.3.0: grafana/grafana#95006 Current Teslamate build uses grafana 11.2.3, so i guess we have to wait for the bump and see.

Unfortunately Grafana 11.3 does still have the annoying fit to zoom not working bug, that's why we rolled back 3 months ago, see #4299 for Grafana 11.3.0 progress

@swiffer Can you reproduce the issue by OP?

[swiffer]

next version of teslamate upgrades grafana to 11.2.3 which should fix the issue.

[nookseal]

is there a problem if we stay 11.2 and warning message keep publishing?

[swiffer]

no, the warning can be ignored until the release is out - only problem -> log entries

[JakobLichterfeld]

fixed by #4338

[evilmumi]

fixed by #4338

but not for Grafana v11.3.1 (64b556c137= or?

[JakobLichterfeld]

fixed by #4338

but not for Grafana v11.3.1 (64b556c137= or?

Fixed for the recommended Grafana

[swiffer]

Happens only with securit releases which neither the now released 11.2.3 nor 11.3.1 are.

Grafana v11.2.3 (b37c88c4ec) Grafana v11.3.1 (9225f4a1cb)

(x64 both having different version identifiers to what you are using).

Please use the grafana latest docker image bundled with v1.32.0.