[Custom Packages] Moving to Input packages

[jamiehynds]

Our 'custom packages' currently state that changing the dataset will send the data to a different index. However, these packages create a template such as logs-udp.generic matching logs-udp.generic-* indices. This contradicts the text saying that changing the dataset will send the data to a different index.

Currently, the data itself will go into the new destination, however the initial component template is made to match the incorrect pattern.

In order to resolve this, our custom packages need to be convert to input packages (i.e. type: input)

We need to test the user experience when upgrading from a current custom package to the new input package. Are there breaking changes involved and to what extend? Depending on the experience, we may need to discuss deprecation of the current packages and shipping new v2 input packages (but hopefully that can be avoided). Fleet team will need to be engaged to discuss breaking changes. We can also co-ordinate efforts with other teams who own certain inputs.

List of packages to convert:

• [x] CEL
• [ ] httpjson (consider depreaction here to avoid confusion over CEL vs httpjon)??
• [ ] AWS (S3 and Cloudwatch inputs)
• [x] Azure Blob Storage
• [x] Google Cloud Storage
• [x] Google Pub/Sub
• [x] HTTP Endpoint
• [x] Journald https://github.com/elastic/integrations/pull/5890
• [ ] Kafka Log
• [x] Log
• [ ] TCP
• [ ] UDP
• [x] Windows https://github.com/elastic/integrations/pull/8010

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

We need to test the user experience when upgrading from a current custom package to the new input package.

Agree. The most important thing is that the upgrade requires no user intervention and that the Fleet stream ID does not change so that input state is not lost. Please do a test for this using one of the stateful inputs and verify that the existing state continues to be used after the upgrade.

[andrewkroh]

The blog post at https://www.elastic.co/blog/customize-data-ingestion-elastic-input-packages can be edited to include the converted packages when they are done. @ishleenk17 can probably aid in that edit if we let her know.

[ishleenk17]

The blog post at https://www.elastic.co/blog/customize-data-ingestion-elastic-input-packages can be edited to include the converted packages when they are done. @ishleenk17 can probably aid in that edit if we let her know.

Sure, let me know when these are migrated. We can add it to the blog.