Fixes for STR-522

[delbonis]

Description

Something unclear is broken with the CSM stuff that's preventing us from correctly advancing the chain. This PR is incomplete and is trying to investigate and fix those issues.

Type of Change

• [x] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature/Enhancement (non-breaking change which adds functionality or enhances an existing one)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] Documentation update
• [x] Refactor

Checklist

• [ ] I have performed a self-review of my code.
• [ ] I have commented my code where necessary.
• [ ] I have updated the documentation if needed.
• [ ] My changes do not introduce new warnings.
• [ ] I have added tests that prove my changes are effective or that my feature works.
• [ ] New and existing tests pass with my changes.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 0% with 253 lines in your changes missing coverage. Please review.

Project coverage is 57.39%. Comparing base (0b0be55) to head (afa7267). Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/consensus-logic/src/csm/worker.rs	0.00%	242 Missing :warning:
crates/state/src/sync_event.rs	0.00%	8 Missing :warning:
crates/consensus-logic/src/csm/config.rs	0.00%	3 Missing :warning:

@@            Coverage Diff             @@
##             main     #406      +/-   ##
==========================================
- Coverage   57.56%   57.39%   -0.17%     
==========================================
  Files         263      264       +1     
  Lines       27716    27792      +76     
==========================================
- Hits        15954    15951       -3     
- Misses      11762    11841      +79     

Files with missing lines	Coverage Δ
bin/strata-client/src/helpers.rs	0.00% <ø> (ø)
bin/strata-client/src/l1_reader.rs	0.00% <ø> (ø)
...rates/consensus-logic/src/csm/client_transition.rs	61.93% <ø> (ø)
crates/consensus-logic/src/csm/ctl.rs	0.00% <ø> (ø)
crates/consensus-logic/src/csm/message.rs	0.00% <ø> (ø)
crates/consensus-logic/src/csm/state_tracker.rs	46.40% <ø> (ø)
crates/consensus-logic/src/duty/worker.rs	0.00% <ø> (ø)
crates/consensus-logic/src/fork_choice_manager.rs	0.00% <ø> (ø)
crates/consensus-logic/src/l1_handler.rs	0.00% <ø> (ø)
crates/consensus-logic/src/sync_manager.rs	0.00% <ø> (ø)
... and 4 more

... and 4 files with indirect coverage changes

[delbonis]

Here's what I've done so far:

• reorganized the CSM modules a bit, split out some exec behavior so we might be able to refactor it some more in the future
• improved logging with span so that we can see from the logs more easily what the CSM is trying to do
• added sync event retry and exponential backoff so if it really is a transient issue we'll just retry it in a second or a few
• if there's ever an error processing a CSM message we quit the CSM worker, which crashes the client that we can identify more clearly instead of leaving it in some weird broken state

[delbonis]

The actual issue was resolved, but these are still useful changes that are in service of longer term goals, so we should still wrap this up.

[delbonis]

Will address the other comments.