Validator client performing all duties 1 second early

Description

My lighthouse validator client is performing its duties one second early according to the logs. This appears to be causing it to attest to the wrong block when a block is received between 3 and 4 seconds into the slot.

Version

Lighthouse was manually built with cargo build --profile maxperf.

$ lighthouse --version
Lighthouse v4.2.0-c547a11
BLS library: blst
SHA256 hardware acceleration: true
Allocator: system
Profile: maxperf
Specs: mainnet (true), minimal (false), gnosis (false)

$ rustup show
Default host: x86_64-unknown-linux-gnu
rustup home:  /home/ptd/.rustup

stable-x86_64-unknown-linux-gnu (default)
rustc 1.69.0 (84c898d65 2023-04-16)

$ uname -a
Linux atlas 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux

Present Behaviour

Here is a sample list of log messages (content stripped to avoid leaking information).

May 26 14:59:35.399 INFO Published validator registrations to the builder network
May 26 14:59:41.001 INFO Connected to beacon node(s)
May 26 14:59:41.001 INFO Some validators active
May 26 14:59:53.000 INFO Connected to beacon node(s)
May 26 14:59:53.000 INFO Some validators active
May 26 15:00:05.000 INFO Connected to beacon node(s)
May 26 15:00:05.000 INFO Some validators active
May 26 15:00:17.001 INFO Connected to beacon node(s)
May 26 15:00:17.001 INFO Some validators active
May 26 15:00:29.000 INFO Connected to beacon node(s)
May 26 15:00:29.000 INFO Some validators active
May 26 15:00:39.140 INFO Successfully published attestations
May 26 15:00:41.001 INFO Connected to beacon node(s)
May 26 15:00:41.001 INFO Some validators active
May 26 15:00:51.125 INFO Successfully published attestations
May 26 15:00:53.000 INFO Connected to beacon node(s)
May 26 15:00:53.000 INFO Some validators active
May 26 15:01:05.000 INFO Connected to beacon node(s)
May 26 15:01:05.001 INFO Some validators active
May 26 15:01:17.000 INFO Connected to beacon node(s)
May 26 15:01:17.000 INFO Some validators active
May 26 15:01:29.000 INFO Connected to beacon node(s)

Expected Behaviour

Since all validator activity is driven by the slot clock I would expect all messages to be timestamped shortly after a multiple of 4 seconds. In particular I would expect Successfully published attestations messages to be emitted shortly after 4 seconds into the slot, rather than shortly after 3 seconds into the slot.

Steps to resolve

Updated to latest Rust and Lighthouse. Updated system software and rebooted.

sigp / lighthouse