search

prestodb / presto

The official home of the Presto distributed SQL query engine for big data
http://prestodb.io
Apache License 2.0
15.96k stars 5.35k forks source link

[native] Merge coordinator and native process logs in e2e tests. #23100

Closed amitkdutta closed 3 months ago

amitkdutta commented 3 months ago

Description

Merging worker and coordinator logs in same output.

Motivation and Context

Often we need to see worker logs while running e2e tests. This is hard to do in CI and testing systems as worker log is burried inside. This makes debugging hard and often impossible if worker process never even starts. Here merging both logs in same output to see worker logs along with cooridnator logs. While the log gets mixed, its very easy to separate between java and c++ format, plus any issue is printed immidiately.

Impact

None

Test Plan

Ran locally and observed worker logs showing up in console. Example:

2024-06-28T01:25:24.341-0500    INFO    Service Thread  com.facebook.airlift.stats.JmxGcMonitor Major GC: application 0ms, stopped 181ms: 255.07MB -> 244.36MB
2024-06-28T01:25:24.341-0500    INFO    Service Thread  com.facebook.airlift.stats.JmxGcMonitor Major GC: application 0ms, stopped 181ms: 255.07MB -> 244.36MB
2024-06-28T01:25:24.341-0500    INFO    Service Thread  com.facebook.airlift.stats.JmxGcMonitor Major GC: application 0ms, stopped 181ms: 255.07MB -> 244.36MB
I20240627 23:25:25.761371 3974755 TaskManager.cpp:539] Starting task 20240628_062522_00000_zfxjc.1.0.0.0 with 16 max drivers.
I20240627 23:25:25.820855 3974755 TaskManager.cpp:539] Starting task 20240628_062522_00000_zfxjc.2.0.0.0 with 16 max drivers.
== NO RELEASE NOTE ==
amitkdutta commented 3 months ago

TaskManagerTest.buildSpillDirectoryFailure failure is tackled in TaskManagerTest.buildSpillDirectoryFailure

amitkdutta commented 3 months ago

Even though the logs get merged when I ran on Mac, it does not seems to get merged when running in CI https://app.circleci.com/pipelines/github/prestodb/presto/17428/workflows/667c8525-6e07-45b4-997d-213898975be1/jobs/68743

arhimondr commented 2 months ago

@amitkdutta Do you know whether both, stdout and stderr are captured by Circle CI? Is it possible that only stderr is captured? Have you tried removing the .redirectErrorStream(true) flag and check the logs?