search

opensearch-project / opensearch-catalog

The OpenSearch Catalog is designed to make it easier for developers and community to contribute, search and install artifacts like plugins, visualization dashboards, ingestion to visualization content packs (data pipeline configurations, normalization, ingestion, dashboards).
Apache License 2.0
21 stars 19 forks source link

[Schema] Update for AWS WAF Integration #198

Open Utkarsh-Aga opened 1 month ago

Utkarsh-Aga commented 1 month ago

Currently, when one uses the AWS WAF Integrations, the Index is considered to have the following mapping and the dashboard, visualizations are created based on that.

However, if we check the sample logs of the WAF then it seems to be quite different from the fields defined in the above mapping and fields like httpRequest are not within the aws.waf object. So, if one tries to send the default WAF logs to the OpenSearch, then this native integration cannot be used correctly.

Would like to propose that either an option to define the initial mapping of the Index should be provided [or the mapping should fetched when the Index is selected] or consider updating the schema of the Integration which matches the schema of the default WAF logs.

Swiddis commented 1 month ago

Thanks for the issue!

From the description it sounds like the logs aren't being mapped to OTEL. The current native integrations all have a specific format for cross-compatibility, so uploading the logs directly won't really work. Ideally we would have a way to specify the initial mapping or otherwise support raw fields directly at the install step as you suggest, but we haven't had the resources to implement that. The other alternative is to find a tool to convert the logs to OTEL -- last I heard this was in progress but still hasn't been finished.

In the meantime the supported fix would be to write a custom integration by hacking the dashboard fields[^1]. The integrations are designed to be hackable to support arbitrary formats that way. I wish I had a more complete answer on-hand, an RFC in this direction would be welcome.

[^1]: I made an editor to make this easier, as part of a project that involved doing just that.