search

kafkaesque-io / pulsar-helm-chart

Helm Chart for an Apache Pulsar Cluster
https://helm.kafkaesque.io
Apache License 2.0
31 stars 22 forks source link

Trying to configure aws s3 offload: no error but nothing is created in my bucket #87

Closed jbtruffault closed 3 years ago

jbtruffault commented 3 years ago

Dears, I tried to configure pulsar to use a storageOffload. Everything start and works fine: I can send data to my topics and get it afterward.

This is an extract of my values.yaml (inspired from https://github.com/kafkaesque-io/pulsar-helm-chart/blob/master/helm-chart-sources/pulsar/values.yaml).

I tried to set managedLedgerOffloadAutoTriggerSizeThresholdBytes to 0 to trigger offload instantly. Nevertheless, nothing appears in my buckets.

Did I forget something ? should I change something ? 

 345 storageOffload:
 346   ## General
 347   ## =======
 348   bucket: sw-pulsar
 349   region: us-east-2
 350   maxBlockSizeInBytes: "64000000"
 351   readBufferSizeInBytes: "1000000"
 352   ## The following are default values for the cluster. They can be changed
 353   ## on each namespace.
 354   managedLedgerOffloadDeletionLagMs: "14400000"
 355   managedLedgerOffloadAutoTriggerSizeThresholdBytes: "0" # disabled
 356
 357   # For AWS S3
 358   # ======
 359   # You must create an IAM account with access to the bucket and
 360   # generate keys for that account.
 361   #
 362   driver: aws-s3
 363   accessKey: mykey
 364   accessSecret: myaccesskey #pragma: allowlist secret

Regards

jbtruffault commented 3 years ago

I found the solution: when the AWS region is not us-east-1, we have to add a line in the configuration: serviceEndpoint: https://s3.<region>.amazonaws.com