quickwit-oss / quickwit

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
https://quickwit.io
Other
7.88k stars 320 forks source link

Getting a `failed to load IMDS session token` on a non-AWS environment #5435

Open fulmicoton opened 4 days ago

fulmicoton commented 4 days ago

observed on scaleway with the following config

version: 0.7

metastore_uri: s3://qw-data-xxxxxx
default_index_root_uri: s3://qw-data-xxxxxx

storage:
  s3:
    access_key_id: xxxxxxx
    secret_access_key: yyyyyyyy
    region: fr-par
    endpoint: https://s3.fr-par.scw.cloud
    force_path_style_access: ${QW_S3_FORCE_PATH_STYLE_ACCESS:-false}
    disable_multi_object_delete: false
    disable_multipart_upload: false

indexer:
  enable_otlp_endpoint: ${QW_ENABLE_OTLP_ENDPOINT:-true}

jaeger:
  enable_endpoint: ${QW_ENABLE_JAEGER_ENDPOINT:-true}

As reported by @ineumann

^[[2m2024-09-16T09:47:44.250Z^[[0m ^[[33m WARN^[[0m ^[[2maws_config::imds::region^[[0m^[[2m:^[[0m failed to load region from IMDS ^[[3merr^[[0m^[[2m=^[[0mfailed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))
^[[2m2024-09-16T09:47:45.252Z^[[0m ^[[33m WARN^[[0m ^[[2maws_config::imds::region^[[0m^[[2m:^[[0m failed to load region from IMDS ^[[3merr^[[0m^[[2m=^[[0mfailed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))
^[[2m2024-09-16T09:49:22.920Z^[[0m ^[[31mERROR^[[0m ^[[2mquickwit_proto::error^[[0m^[[2m:^[[0m gRPC transport error: Timeout expired ^[[3mcode^[[0m^[[2m=^[[0mCancelled ^[[3mrpc^[[0m^[[2m=^[[0m"publish_splits"
^[[2m2024-09-16T09:49:22.921Z^[[0m ^[[31mERROR^[[0m ^[[2mquickwit_actors::spawn_builder^[[0m^[[2m:^[[0m actor-failure ^[[3mcause^[[0m^[[2m=^[[0mfailed to publish splits

Could be similar to https://github.com/delta-io/delta-rs/pull/2817/files

idrissneumann commented 3 days ago

Hi.

This dashboard might help to have more details (last occurrences of the error, some stats): https://grafana.comwork.io/public-dashboards/801a34eb04a5462d968e5f2d3a5b3e49

Thanks