[ENHANCEMENT] Expose AWS enabled properties for Operate, Tasklist, Optimize

ThorbenLindhauer commented 2 months ago

Describe the use case:

We need to fix bugs with Operate/Tasklist/Optimize around authentication with AWS Opensearch. This requires us to expose new configuration properties (see below) that must be configurable through the Helm chart. The fix needs to be implemented on main/8.6 and backported to 8.5.

Support case: https://jira.camunda.com/browse/SUPPORT-23119
Incident channel: https://app.slack.com/client/T0PM0P1SA/C07FZMBJ03G

Describe the enhancement/feature:

Map the Helm chart configuration global.opensearch.aws.enabled to:
- camunda.operate.opensearch.awsEnabled application property of Operate (ticket that introduces this: https://github.com/camunda/camunda/issues/20939)
- camunda.tasklist.opensearch.awsEnabled application property of Tasklist (ticket that introduces this: https://github.com/camunda/camunda/issues/21070)
- CAMUNDA_OPTIMIZE_OPENSEARCH_AWS_ENABLED environment variable for Optimize (ticket that introduces this: https://github.com/camunda/camunda-optimize/issues/14030)

Desired outcome and acceptance tests:

### Tasks
- [ ] https://github.com/camunda/camunda-platform-helm/pull/2232

hamza-m-masood commented 2 months ago

@ralfpuchert Just to confirm, if the user set global.opensearch.aws.enabled to true then there is no need for basic auth because AWS IRSA will be used instead for auth correct? Also is this true across the board for all apps, including tasklist, and optimize?

hamza-m-masood commented 2 months ago

@ThorbenLindhauer It is still possible to use AWS IRSA with operate and tasklist without the need to setup a username and password. Only optimize is not officially supported with AWS IRSA.

ThorbenLindhauer commented 2 months ago

Hey @hamza-m-masood,

Just to confirm, if the user set global.opensearch.aws.enabled to true then there is no need for basic auth because AWS IRSA will be used instead for auth correct?

Correct

Also is this true across the board for all apps, including tasklist, and optimize?

Yes

It is still possible to use AWS IRSA with operate and tasklist without the need to setup a username and password. Only optimize is not officially supported with AWS IRSA.

Thanks, wasn't aware of the Optimize detail. I wonder now a little why the Optimize medic create a ticket, but I assume it makes sense and otherwise they'll figure it out during implementation.

hamza-m-masood commented 2 months ago

@ThorbenLindhauer thanks for the reply 👍 I will still add this option to all listed apps. It is good to have. I hope the app logs clear up a little bit with this option enabled. I am assuming these configs don't currently exist since the issues are still open. In that case, I will not merge my PR until all the above issues are closed, and I can comfortably test the changes.

ThorbenLindhauer commented 2 months ago

I am assuming these configs don't currently exist since the issues are still open.

Correct, once the issues are closed (and the CI has run), you will be able to use the 8.6 and 8.5 snapshots to test. Note that a proper patch release will only be available as part of the regular release train that targets the second Tuesday of September.

hamza-m-masood commented 2 months ago

@matthewBearCamunda @grlimacan I have a few questions for Optimize:

Currently, Optimize does not support migration through the initContainer when OpenSearch is enabled with basic auth. Is this still the case?
As I said, we currently do not officially support Optimize with IRSA enabled. Will Optimize support IRSA when the above issue you created is closed?
Will migration through the initContainer be supported for IRSA as well?

hamza-m-masood commented 2 months ago

I have completed the PR. I will pause this issue. I will resume when I can begin testing and get more clarity from the Optimize team.

matthewBearCamunda commented 2 months ago

@hamza-m-masood

This I'm unsure about. Our resident OpenSearch expert @grlimacan is on FTO and he'd be the one to answer this. I can dive into this more if this is a blocking question
IRSA is supported by Optimize. It was added here: https://github.com/camunda/camunda-optimize/issues/12696
What migration are you talking about here? Are you referring to a DB migration when Optimize changes versions, when DB schema changes could be happening?

hamza-m-masood commented 2 months ago

I will have to make a change in the official docs that IRSA is supported for Optimize now that it is confirmed by you. I will wait for @grlimacan for further info on this topic.

In regards to the migration I am referring to the script called upgrade/upgrade.sh through this initContainer: https://github.com/camunda/camunda-platform-helm/blob/main/charts/camunda-platform-alpha/templates/optimize/deployment.yaml#L39 I am not sure if this script is related to DB schema change.

grlimacan commented 2 months ago

I will have to make a change in the official docs that IRSA is supported for Optimize now that it is confirmed by you. I will wait for @grlimacan for further info on this topic.

In regards to the migration I am referring to the script called upgrade/upgrade.sh through this initContainer: https://github.com/camunda/camunda-platform-helm/blob/main/charts/camunda-platform-alpha/templates/optimize/deployment.yaml#L39 I am not sure if this script is related to DB schema change.

Hey @hamza-m-masood , we implemented the ticket mentioned above by @matthewBearCamunda but we have still not been able to validate that in a real IRSA environment, since we don't have the means to create and configure one. Are you able to confirm if the fix works properly?

As for the actual changes concerning the disabling of the AWS config, I have just implemented the fix and am awaiting review. I will let you know as soon as it's reviewed

Also migration is still not supported by OpenSearch, but we hope to have this done soon. I will keep you updated

hamza-m-masood commented 2 months ago

@grlimacan Thanks for replying. I think this thread in our ask-distro channel will be useful to you for testing with IRSA: https://camunda.slack.com/archives/C03UR0V2R2M/p1721729195556469

If not, I can test for you. No problem. 👍

grlimacan commented 2 months ago

Hi @hamza-m-masood , https://github.com/camunda/camunda-optimize/issues/14030 is already implemented and will be present in the upcoming alpha and also on 3.13.5

hamza-m-masood commented 2 months ago

I am currently trying to test with the snapshot images of tasklist, operate, and optimize

hamza-m-masood commented 2 months ago

I can confirm the following when testing with basic auth:

Optimize snapshot image is working with basic auth.
Operate pod is not in a healthy state. The logs do not show an error but the readinessProb fails.
Tasklist does not build the correct URL for OpenSearch. I get the following error:
```
io.camunda.tasklist.zeebe.PartitionHolder - Error occurred when requesting partition ids from Zeebe client: io exception
....
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/127.0.0.1:26500
```
Seems like tasklist is using localhost as it's way to connect to zeebe gateway. I'm relatively sure that zeebe gateway is configured correctly because Operate would also be complaining if there was a problem with zeebe gateway.

hamza-m-masood commented 2 months ago

Operate and Tasklist are not able to connect to zeebe but I do see the data present on the OpenSearch database. For that reason I am happy to go ahead and merge this PR after review. I have tested with basic auth.

camunda / camunda-platform-helm

[ENHANCEMENT] Expose AWS enabled properties for Operate, Tasklist, Optimize #2215