make kinesis input cloud compatible and encrypt secret access key

Graylog2 / graylog-plugin-integrations

A collection of open source Graylog integrations that will be released together.

Other

14 stars 14 forks source link

make kinesis input cloud compatible and encrypt secret access key #1326

Closed AntonEbel closed 1 year ago

AntonEbel commented 1 year ago

Notes for Reviewers

enable AWS Kinesis/CloudWatch input in cloud
encrypt secret access key
use new FAILING state to signal errors

Part of Graylog2/graylog-plugin-cloud#1091

To test this, make sure that you run a cloud setup and that you enable the cloud_inputs feature, e.g. by setting the following Java system property: -Dgraylog.feature.cloud_inputs=on

/nocl

danotorrey commented 1 year ago

Sorry for the delay on the review. I should be able to review soon.

AntonEbel commented 1 year ago

Currently the cloud_inputs feature flag in the UI is broken, in the meantime this PR can be used with the fix.

danotorrey commented 1 year ago

The code LGTM, and I have tested the following aspects successfully:

Migration of previous unencrypted secret.
Running input in on-prem/non-cloud environment.

But, in Cloud, when trying to save the input at the setup wizard, the following error occurs at the very end of the setup wizard:

We were unable to save your Input, please try again in a few moments.
MORE INFO
Additional Information: Cannot construct instance of `org.graylog.integrations.aws.resources.requests.AWSInputCreateRequest$Builder`, problem: Missing required properties: global at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 297]

Also, I think the intention is to always run this input globally on Cloud, so perhaps we should default the UI Global Input option to true (currently defaults to false).

AntonEbel commented 1 year ago

Thanks @danotorrey for the review and the good catch. I removed the global setting from the UI and the request as it was not used in the backend anyway. With this PR we always create the Kinesis input as a global input. The issue with the input not being global is described here and we have changed it for other inputs as well. If the user wants to run it only on a single node, he can change the global settings afterwards (on-prem only). Creating the Kinesis input should work now.

danotorrey commented 1 year ago

@AntonEbel Sorry, I was out sick yesterday. I will retest this right away today. Should have results here in a few.