search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.12k stars 3.81k forks source link

roachprod: output start and end timestamp to ssh logs #100151

Open smg260 opened 1 year ago

smg260 commented 1 year ago

We currently preserve ssh debug logs in the case of any failure, however there is no way to see what time the command actually ran. The timestamp encoded in the log file currently, only indicates when the command was initialised, not when it was executed.

There is no easy way to show timestamps alongside the debug log output. A process substitution solution worked in bash, but not zsh. Additionally, it relies on enriching the existing error output of the command, on which some parts of the code may already depend.[1]

A compromise would be to emit start and end timestamps to the ssh debug file, which operates outside of stdout/stderr.

[1] A subset of read accesses of RunResultDetails.Stderr image

Jira issue: CRDB-26277

blathers-crl[bot] commented 1 year ago

cc @cockroachdb/test-eng

srosenberg commented 1 year ago

Is this a general enhancement or is there an example of a recent failure where the timestamps would have been crucial during triage?

smg260 commented 1 year ago

General enhancement. It would be nice to correlate commands run over ssh to a more specific time range; it can be somewhat deduced from the corresponding command log, and test.log, but useful not to have to look at multiple files just for a timestamp.

Low priority, but easy fix.