Log files produced by Hive aren't processed by Vector

zaultooz commented 4 months ago

Affected Stackable version

23.11.0

Affected Apache Hive version

3.1.3

Current and expected behavior

Currently none of the produced log statements in the hive.log4j2.xml is picked up by Vector and send to the aggregator, resulting in no logs from Hive appearing in our logging solution.

I would expect Vector would transform and propagate the logs similar to other Stackable components.

Possible solution

Looking a bit into the matter, it seems the files_log4j2 in vector.yaml cm doesn't take multi-line into account. The issue was resolved by changing the file entry in vector.yaml so it looks like this:

files_log4j2:
    type: file
    include:
        - /stackable/log/*/*.log4j2.xml
    line_delimiter: "\r\n"
    multiline:
        mode: halt_before
        start_pattern: ^<Event
        condition_pattern: ^<Event
        timeout_ms: 1000

I am unsure if there are other Stackable components using log4j2 currently so don't know if this is a suitable solution or it would be better to format everything from Hive one line.

Additional context

No response

Environment

Client Version: v1.28.9 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.9

Would you like to work on fixing this bug?

None

siegfriedweber commented 4 months ago

This should work in principle.

log4j2 emits the log events with the following line endings:

\n
  <Event ...>\n
    <Instant .../>\n
    <Message>...</Message>\n
  </Event>\r\n

Therefore, the Vector configuration used by the operator should split the "multiline" events correctly:

sources:
  files_log4j2:
    type: file
    include:
      - /stackable/log/*/*.log4j2.xml
    line_delimiter: "\r\n"

The integration tests pass: https://github.com/stackabletech/hive-operator/tree/23.11.0/tests/templates/kuttl/logging

We also successfully tested stack traces in log events.

Could you please check the following points?

Is the discovery ConfigMap of the Vector aggregator added to the Hive cluster specification and the Vector agent enabled?

---
apiVersion: hive.stackable.tech/v1alpha1
kind: HiveCluster
spec:
  clusterConfig:
    vectorAggregatorConfigMapName: vector-aggregator-discovery
  metastore:
    roleGroups:
      default:
        config:
          logging:
            enableVectorAgent: true

Is the Vector aggregator ConfigMap configured correctly?

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: vector-aggregator-discovery
data:
  ADDRESS: vector-aggregator:6000

Does the Vector agent (the side-car container of the Hive metastore pod) log the following line?

INFO source{component_kind="source" component_id=files_log4j2 component_type=file component_name=files_log4j2}:file_server: vector::internal_events::file::source: Found new file to watch. file=/stackable/log/hive/hive.log4j2.xml

zaultooz commented 4 months ago

Thanks for taking the time to response. I can see that you are indeed correct.

After removing the custom settings I added and restarted hive, I can confirm that it still works so I will close the issue.

The issue I was seeing before must have been caused by something else. Originally when I spotted the missing logs, I couldn't see any of the new log statements getting propagated but I guess all it needed was a restart.

Again thanks.

stackabletech / hive-operator