4.1 Forwarder Documentation

[danotorrey]

Modify the existing Cloud Forwarder documentation to target the new 4.1 Forwarder.

In the upcoming 4.1 release of Graylog, we will be migrating the Cloud Forwarder and making it available in the core on-premise Graylog Server product (Enterprise customers only). Currently, we are just calling this the “Forwarder”.

So, starting in Graylog 4.1, we will maintain this one Forwarder tool that can be used for both Cloud and on-premise Graylog installations. The main benefit for on-premise customers, is that they can forward data from other networks or locations to their central Graylog installation. The functionality remains the same for Cloud installations.

So, with the new 4.1 Forwarder, we will need to modify the existing Forwarder documentation to describe the general 4.1 Forwarder instead.

A few specifics on the modifications needed to the existing Forwarder docs:

• Product name: The name will change from "Cloud Forwarder" to just "Forwarder"
• Most of the implementation of the Forwarder (how it works internally) will remain the same.
• We will release one version of the forwarder that works with both Cloud and on-premise. So the installation (tar, os packages, docker should work for both). I think the OS packages/repos should remain mostly the same.
• The Forwarder will be versioned with a simple Major.Minor version scheme (eg. 4.1, 4.2). So, starting out, we will release 4.0. The major version will always be compatible with the same major version of Graylog. More info about Forwarder versioning here. We should include an explanation about the compatibility of versions in the docs.
• Since the Forwarder will be compatible with Graylog Cloud and Graylog On-premise, I think the documentation will need to provide instructions for both setups. The only thing that is different will be how they are configured. For on premise, they will need to provide their own server hostname/IP and TLS certs.
• TLS is automatic with the existing Cloud Forwarder, and will continue to be for Cloud in the new Forwarder. But, for on-premise, the customer will need to provide their own cert and key on the server/client side. See the cert and key properties in the wiki. forwarder_grpc_tls_trust_chain_cert_file in the Forwarder and forwarder_grpc_tls_private_key_file and forwarder_grpc_tls_trust_chain_cert_file on the server-side.
• The Forwarder setup wizard will change slightly with the new 4.1 Forwarder. It will need to be able to provide the configuration for both Cloud and on-premise settings depending on which is running. We need to look at this more to understand what it will look like.

Development on the 4.1 Forwarder is still in-progress, and as it continues, we will probably find a few more details to include in the documentation.

[danotorrey]

Hi @dulanism, I have made some notes here about the thinking on documentation changes for the 4.1 Forwarder in case you have time to work on it in the next weeks. Development on the Forwarder will continue, and some parts might change. I will be back in June 1, and will check back in then! :)

[devdanylo]

@dulanism @danotorrey please find below the essential functional diff between Cloud and On-Prem:

Aspect	Cloud	On-Prem
Server-side Service Configuration	The configuration is still done via the config file. Nothing has changed here. Except one minor thing: a new property to configure TLS private key password was introduced, but I believe it should not be reflected in the public documentation.	To avoid 1) having the Forwarder server-side services running by default and 2) restarting the server after making changes in the config file, it was decided to migrate away from the config file and to introduce a new Input type, which is supposed to facilitate Forwarder server-side service configuration and lifecycle management. The name for the new Input is Forwarder. The legacy one was renamed to Cluster-to-Cluster forwarder. This change entailed certain modifications in the Wizard: 1) Now the Wizard must verify if there is at least one running input 2) Populate the Forwarder (client) configuration snippet according to the Input's config values. So in other words one additional step was introduced to make sure that there is a running server-side service to connect to. See screenshots below. NOTE that some server-side configuration properties are still to be provided via the config file (not expected to change frequently): grpc_graceful_shutdown_timeout, grpc_max_message_size, heartbeat_timeout_interval, forwarder_user (see https://github.com/Graylog2/graylog-plugin-cloud/wiki/Forwarder#forwarder-configuration-properties-server-side). Not sure if it should be a part of the public document.
Menu Layout	Nothing should change here.	The Forwarder menu item resides under Enterprise

[dulanism]

Just making a note here that I'm making a few minor edits to what will be the old enterprise data forwarder. Really just the name, and the graphic which is out of place.