search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.07k stars 3.8k forks source link

roachtest: jepsen run an ntp server on controller node to enable clock skew tests #74401

Closed aliher1911 closed 1 year ago

aliher1911 commented 2 years ago

Currently clock skew tests are disabled: https://github.com/cockroachdb/cockroach/blob/21a286f2b2978b1cc117596865cd680dd268cf74/pkg/cmd/roachtest/tests/jepsen.go#L38-L42 Because of #35599.

Tests rely directly on ntp.ubuntu.com which starts throttling requests pretty quickly making tests flaky.

It should be easy to install an ntp package to the controller node only and point tests to use it since we could pass its IP to the test from roachtest. We need to check how it would interact with chrony or to disable chrony on controller.

Jira issue: CRDB-12064

aliher1911 commented 2 years ago

This could be done pretty easily by passing ntp server to test and then adding 'allow' to chronyd configuration on controller. No need to run ntpd as chronyd has ntpd functionality that is not enabled by default. But easier solution could be to just use pool.ntp.org instead. We should try that first and only proceed with this approach if using pool still fails. (see #35599)

aliher1911 commented 1 year ago

This is resolved as we now use ntp that doesn't throttle us.