cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.12k stars 3.81k forks source link

backup: export failed errors on retry are too noisy #134779

Open shailendra-patel opened 2 days ago

shailendra-patel commented 2 days ago

On drt-scale During incremental backup there are many failed export request error message. The incremental backup did complete successfully in 6 min. Logs link on datadog.

failed export request ‹/Table/113/4/151{397/76453-404/56910}›
 trace:
     0.000ms      0.000ms    === operation:backupccl.ExportRequest
     0.000ms      0.000ms    [dist sender send: {count: 1, duration 491ms}]
     0.000ms      0.000ms    [/cockroach.roachpb.Internal/Batch: {count: 1, duration 491ms}]
     0.000ms      0.000ms    [admissionWorkQueueWait: {count: 1, duration 491ms}]
   490.651ms    490.651ms    structured:‹{"@type":"type.googleapis.com/cockroach.util.admission.admissionpb.AdmissionWorkQueueStats","waitDurationNanos":"0.490573770s","queueKind":"kv-elastic-cpu-queue","deadlineExceeded":true,"workPriority":"bulk-normal-pri"}›

This issue is request to change log severity from error to warning. Here is the code which needs to be changed.

More details in thread.

Jira issue: CRDB-44234

blathers-crl[bot] commented 2 days ago

cc @cockroachdb/disaster-recovery