Enable AWS CloudTrail input in cloud

thll commented 1 year ago

Makes the AWS CloudTrail input compatible with a hosted cloud environment.

:cloud: + on-prem: Disables all inputs except the CloudTrail input
:cloud:: Disables AWS instance lookup processor
:cloud:: Disables default AWS authentication chain (env, properties, ~/.aws...)
:cloud: + on-prem: Makes secret key an encrypted field.

/jenkins-pr-deps Graylog2/graylog2-server#14271 TODO:

[x] Clarify if this is really a hard requirement for cloud
[x] Disable unrelated functionality of plugin (Configuration, Message Processor, etc...)
[x] Create new encrypted fields for credentials
[x] Decide how to handle the deprecated secret key field.
[x] Check if dependencies can be upgraded
[x] Disable instance lookup processor in Forwarder

bernd commented 1 year ago

@thll Our cloud build currently doesn't include the AWS plugin. So I guess that needs to change?

thll commented 1 year ago

@thll Our cloud build currently doesn't include the AWS plugin. So I guess that needs to change?

@bernd: I thought it did, since about two weeks ago: https://github.com/Graylog2/graylog-project-internal/pull/89. Or are we talking about different things?

bernd commented 1 year ago

@thll Our cloud build currently doesn't include the AWS plugin. So I guess that needs to change?

@bernd: I thought it did, since about two weeks ago: Graylog2/graylog-project-internal#89. Or are we talking about different things?

The cloud builds don't use the manifests. It's Git submodule based. I can add the AWS plugin, no problem. Is this for the 5.0 and 4.3 builds or just for the 5.0 one?

thll commented 1 year ago

The cloud builds don't use the manifests. It's Git submodule based. I can add the AWS plugin, no problem. Is this for the 5.0 and 4.3 builds or just for the 5.0 one?

Thanks @bernd, that would be great! 5.0 should be sufficient.

bernd commented 1 year ago

The cloud builds don't use the manifests. It's Git submodule based. I can add the AWS plugin, no problem. Is this for the 5.0 and 4.3 builds or just for the 5.0 one?

Thanks @bernd, that would be great! 5.0 should be sufficient.

Okay, I guess this change is targeted for 5.1. So we only need to include the AWS plugin once we have a 5.1 build. Currently, we only do 4.3 (stable), and 5.0 (dev) builds.

https://github.com/Graylog2/graylog-cloud-builds/issues/11

thll commented 1 year ago

I've noticed that when one of the now deprecated inputs has been created before, the Forwarder inputs page crashes with the following error:

UI error

Cannot read properties of undefined (reading 'name')

Stack Trace:

TypeError: Cannot read properties of undefined (reading 'name')
    at ForwarderInputListItem (http://localhost:8080/plugin.org.graylog.plugins.enterprise.EnterprisePlugin.js:208158:119)
    at renderWithHooks (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:14803:18)
    at mountIndeterminateComponent (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:17482:13)
    at beginWork (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:18596:16)
    at HTMLUnknownElement.callCallback (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:188:14)
    at Object.invokeGuardedCallbackDev (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:237:16)
    at invokeGuardedCallback (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:292:31)
    at beginWork$1 (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:23203:7)
    at performUnitOfWork (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:22154:12)
    at workLoopSync (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:22130:22)
    at performSyncWorkOnRoot (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:21756:9)
    at eval (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:11089:24)
    at unstable_runWithPriority (webpack://__/./node_modules/react-dom/node_modules/scheduler/cjs/scheduler.development.js?:653:12)
    at runWithPriority$1 (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:11039:10)
    at flushSyncCallbackQueueImpl (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:11084:7)
    at flushSyncCallbackQueue (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:11072:3)
    at discreteUpdates$1 (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:21893:7)
    at discreteUpdates (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:806:12)
    at dispatchDiscreteEvent (webpack://__/./node_modules/react-dom/cjs/react-dom.development.js?:4168:3)

Component Stack:

    in ForwarderInputListItem (created by ForwarderInputsManagement)
    in ul (created by styled.ul)
    in styled.ul (created by EntityList)
    in EntityList (created by ForwarderInputsManagement)
    in ListBase (created by ListWithOwnState)
    in ListWithOwnState (created by PaginatedList)
    in PaginatedList (created by ForwarderInputsManagement)
    in ForwarderInputsManagement (created by InputProfilePage)
    in div (created by Col)
    in Col (created by InputProfilePage)
    in div (created by Row)
    in Row (created by Styled(Row))
    in Styled(Row) (created by InputProfilePage)
    in DocumentTitle (created by InputProfilePage)
    in InputProfilePage
    in Unknown
    in Unknown (created by Context.Consumer)
    in Route (created by AppRouter)
    in Switch (created by AppRouter)
    in div (created by Grid)
    in Grid (created by Styled(Grid))
    in Styled(Grid) (created by PageContentLayout)
    in WithGlobalNotifications (created by PageContentLayout)
    in div (created by styled.div)
    in styled.div (created by PageContentLayout)
    in PageContentLayout (created by AppRouter)
    in Route (created by AppRouter)
    in Switch (created by AppRouter)
    in div (created by styled.div)
    in styled.div (created by Context.Consumer)
    in RuntimeErrorBoundary (created by Context.Consumer)
    in ReportedErrorBoundary (created by Context.Consumer)
    in div (created by styled.div)
    in styled.div (created by Context.Consumer)
    in ScratchpadProvider (created by Context.Consumer)
    in App (created by AppRouter)
    in Route (created by AppRouter)
    in Switch (created by AppRouter)
    in RouterErrorBoundary (created by AppRouter)
    in Router (created by AppRouter)
    in AppRouter (created by LoggedInPage)
    in StreamsProvider (created by ConnectStoresWrapper[StreamsProvider] stores=streams)
    in ConnectStoresWrapper[StreamsProvider] stores=streams (created by LoggedInPage)
    in LicenseCheckProvider (created by GlobalContextProviders)
    in ErrorBoundary (created by GlobalContextProviders)
    in InvestigationDrawerProvider (created by GlobalContextProviders)
    in ErrorBoundary (created by GlobalContextProviders)
    in GlobalContextProviders (created by LoggedInPage)
    in Fe (created by GraylogThemeProvider)
    in GraylogThemeProvider (created by ThemeAndUserProvider)
    in CurrentUserPreferencesProvider (created by ThemeAndUserProvider)
    in StaticTimezoneProvider (created by CurrentUserDateTimeProvider)
    in CurrentUserDateTimeProvider (created by UserDateTimeProvider)
    in UserDateTimeProvider (created by ThemeAndUserProvider)
    in CurrentUserProvider (created by ThemeAndUserProvider)
    in ThemeAndUserProvider (created by LoggedInPage)
    in QueryClientProvider (created by DefaultQueryClientProvider)
    in DefaultQueryClientProvider (created by LoggedInPage)
    in LoggedInPage
    in Suspense
    in ErrorBoundary
    in Unknown (created by AppFacade)
    in AppFacade
    in Fe (created by GraylogThemeProvider)
    in GraylogThemeProvider
    in CustomizationProvider

thll commented 1 year ago

I've noticed that when one of the now deprecated inputs has been created before, the Forwarder inputs page crashes

I create https://github.com/Graylog2/graylog-plugin-enterprise/issues/4798 for this. I think it should be addressed before merging this PR.

Graylog2 / graylog-plugin-aws

Enable AWS CloudTrail input in cloud #674