AWS + Dynamo store mode, Dynamo tables could not be found

[alexsorkin]

I'm getting following error, once enabling aws-dynamo store mode: failed to flush user" err="ResourceNotFoundException: Requested resource not found"

NOTE: This works with default bolt shipper... But indexes are created in S3.

Config: ` schema_config: configs:

• from: 2018-10-24 store: aws-dynamo object_store: s3 schema: v11 index: prefix: period: 24h row_shards: 32

  storage_config: aws: s3: s3:/// http_config: idle_conn_timeout: 90s response_header_timeout: 0s dynamodb: dynamodb_url: dynamodb://

  boltdb_shipper: active_index_directory: /tmp/loki/boltdb-shipper-active cache_location: /tmp/loki/boltdb-shipper-cache cache_ttl: 24h shared_store: s3 filesystem: directory: /tmp/loki/chunks

  limits_config: enforce_metric_name: false reject_old_samples: true reject_old_samples_max_age: 168h

  table_manager: retention_deletes_enabled: true retention_period: 672h

  compactor: working_directory: /tmp/loki/boltdb-shipper-compactor shared_store: s3

  memberlist: abort_if_cluster_join_fails: false max_join_backoff: 1m max_join_retries: 10 min_join_backoff: 1s

  distributor: ring: kvstore: store: memberlist

  ingester: lifecycler: ring: kvstore: store: memberlist replication_factor: 1 final_sleep: 0s chunk_idle_period: 1m chunk_retain_period: 30s `

[muhammadn]

Can you try with this instead?

schema_config:
  configs:
  - from: 2020-10-24
    store: aws
    object_store: s3
    schema: v11
    index:
      prefix: index_
      period: 24h
      tags: {}

 storage_config:
  aws:
    s3: s3://us-east-1/rocky-dev-loki
    dynamodb:
      dynamodb_url: dynamodb://us-east-1

Reference: https://medium.com/codex/setup-loki-storage-to-aws-s3-and-dynamodb-in-kubernetes-with-kiam-4c223a0b7995

Also check the documents of loki. It's the same. store should be aws and not aws-dynamo

https://grafana.com/docs/loki/latest/configuration/examples/#aws-basic-configyaml

[alexsorkin]

Hi,

Could it be if I am already having the objects in S3 created with bolt shipper, and than I switched to aws store mode... The compactor service can't actually find the corresponding tables for existing objects?

I mean, I configured it first like in examples with boltdb-shipper as store type. Then tried to switch to aws-dynamo AND aws. Both produced the error.

[muhammadn]

@alexsorkin the error ResourceNotFoundException: Requested resource not found has nothing to do with boltdb-shipper.

It has to do with dynamodb. read here: https://stackoverflow.com/questions/40192304/aws-dynamodb-resource-not-found-exception

[muhammadn]

@alexsorkin Have you tried using access key and secret? (if you're not using IRSA, which i suspect, since your current configuration will only work with IRSA)

https://grafana.com/docs/loki/latest/configuration/examples/#aws-basic-configyaml

As mentioned (again) referenced from the document, you should try like this:

schema_config:
  configs:
  - from: 2020-05-15
    store: aws
    object_store: s3
    schema: v11
    index:
      prefix: loki_
storage_config:
  aws:
    s3: s3://access_key:secret_access_key@region/bucket_name
    dynamodb:
      dynamodb_url: dynamodb://access_key:secret_access_key@region

NOTE: dynamodb_url: dynamodb://access_key:secret_access_key@region

[alexsorkin]

Hi, unfortunately I had to abandon this solution.

The traces being produced are malformed and the performance haven't met our requirements. This solution could be used only for one collector producer. Distributed traces are not supported at all.

[muhammadn]

@alexsorkin Where do you view the distributed traces? Is it in Jaeger-UI or Kiali?

jaeger-objectstorage supports multiple collectors, which was solved with the latest version. Older versions have a problem with performance, until we rewrote it to use loki to save and retrieve the data. Internally we use loki - if it works with loki, it will work with jaeger-objectstorage.

I've tried to provide extensive documentation but if you need more support you can contact us. We've already deployed jaeger-objectstorage in production and since the code is opensource, you are free to do whatever you want with it.