Check changelogs of Vector between 0.26 and 0.31

[siegfriedweber]

Check the changelogs of Vector if there are any benefits or obstacles of upgrading to a newer version.

[siegfriedweber]

Version 0.28.2

Removal of metadata functions

The metadata functions are not used but the .metadata field of the Vector agent logs could also be affected (see e.g. https://github.com/stackabletech/operator-rs/blob/0.44.0/src/product_logging/framework.rs#L966-L967).

Version 0.29.1

:warning: There are no breaking changes listed in the changelog which should affect us, but I tried this version with Superset and several things were broken:

• Only the superset-init-db container was forwarded to the aggregator. The logs of the superset and vector containers were not gathered or forwarded.
• The log_schema.host_key seems to be ignored. The pod name is stored in the field host instead of pod.

Version 0.30.0

Changelog - Bugfixes

• The elasticsearch sink now uses the correct API to automatically determine the version of the downstream Elasticsearch instance (when api_version = "auto").

Version 0.31.0

Removal of various deprecated internal metrics

The processed_events_total metric was removed which is used in all logging integration tests (see e.g. https://github.com/stackabletech/zookeeper-operator/blob/23.4.1/tests/templates/kuttl/logging/test_log_aggregation.py#L15-L16). This metric must probably be replaced with component_received_events_total.

Log Namespacing

Log namespaces are still in the beta phase but can be enabled in this release. Log metadata is then organized in different namespaces which avoids name conflicts and makes the events cleaner.

Changelog - Enhancements

• A from_unix_timestamp function was added to VRL to decode timestamp values from unix timestamps. This deprecates the to_timestamp function, which will be removed in a future release.
• The internal_metrics and internal_logs sources now shutdown last in order to capture as much telemetry as possible during Vector shutdown.

[lfrancke]

Thank you! So, in summary: We cannot just upgrade, we need to change and test.

Did you see any enhancements that would be useful for us or any security vulnerabilities that have been fixed that we could be affected by?

[siegfriedweber]

There are no improvements that are really significant for us. The release notes mention bug fixes but I do not know if any of them are security related. However, I would upgrade in the medium term anyway because if there is a security update in the future which we need, then we still have to spend the time. And the security update is guaranteed to be published when the persons who know Vector, are on vacation.

[lfrancke]

I wonder who that could be.

Agreed.

Please take it on whenever you want. I'll leave it in your hands. You can put it on the board at your pleasure.