search

lightbend / cloudflow

Cloudflow enables users to quickly develop, orchestrate, and operate distributed streaming applications on Kubernetes.
https://cloudflow.io
Apache License 2.0
321 stars 90 forks source link

Checkpointing configurations are not taking effect for flink streamlet #1257

Closed JustinPihony closed 2 years ago

JustinPihony commented 2 years ago

The below checkpointing configurations are not taking effect when tried to configure enabling the Rocksdb backend with Azure blob storage .

cloudflow.runtimes.flink.config {
  flink.state.backend = filesystem
  flink.state.checkpoints.dir = "wasbs://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path>"
  flink.fs.azure.account.key.<your-azure-acount>.blob.core.windows.net = "<blob-access-key>"
}

The current workaround is to do it via code:

getContext().env().setStateBackend(new EmbeddedRocksDBStateBackend(true));
getContext().env().getCheckpointConfig().setCheckpointStorage("wasbs://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path>");
getContext().env().getCheckpointConfig().setCheckpointStorage(new FileSystemCheckpointStorage(file:///tmp/checkpoint-dir));
franciscolopezsancho commented 2 years ago

As mentioned yesterday, since we don't use Flink Operator we can't pass any configuration to Flink. Only a subset described here. The configuration requested in this issue is not available.