feat(zebra-db): Split low level database code inside `zebra-state`

[oxarbitrage]

Motivation

We want to re-using RocksDB for our scanner work, because we already have well-tested low level interfaces to it.

Suggested Changes

We want to move low-level database code into:

• https://github.com/ZcashFoundation/zebra/blob/main/zebra-state/src/service/finalized_state/disk_db.rs
• https://github.com/ZcashFoundation/zebra/blob/main/zebra-state/src/service/finalized_state/disk_format.rs
• and their submodules
• the state version code from config into disk_db/upgrade/version.rs
• the old versions cleanup task from service into disk_db/upgrade/cleanup.rs
• the generic format upgrade code from zebra_db.rs (but not the zebra-state-specific read and write methods on ZebraDb)
• check for anything else at the service, finalized state, or zebradb levels

And move anything specific to zebra-state outside those modules:

• the Zebra-specific parts of upgrade.rs
• anything specific to the column families, validation, or custom tasks in zebra-state

Related Work

This was mentioned in https://github.com/ZcashFoundation/zebra/issues/7904#issuecomment-1802586921 and earlier comments on that ticket.

[teor2345]

@arya2 this ticket is related to some ideas you had about making format upgrades generic.

The simplest change we could do is to pass a format upgrade function and a format check function to the upgrade task in the split out crate.

But how much more work would it be to refactor each upgrade into an upgrade trait, and then pass a list of upgrade trait objects to the upgrade task?

I guess we could do that later, it's not required to do the split.

[arya2]

But how much more work would it be to refactor each upgrade into an upgrade trait, and then pass a list of upgrade trait objects to the upgrade task?

I don't think it would be too significant, but it would involve a lot of code movement that could be difficult to review alongside this change.

I guess we could do that later, it's not required to do the split.

It would be good to do it before the next format upgrade, I opened https://github.com/ZcashFoundation/zebra/issues/7932

[teor2345]

But how much more work would it be to refactor each upgrade into an upgrade trait, and then pass a list of upgrade trait objects to the upgrade task?

I don't think it would be too significant, but it would involve a lot of code movement that could be difficult to review alongside this change.

I was expecting this change to happen in 2-3 PRs:

• optional: change upgrades into a trait
• move zebra-state specific code out of the files that are going to be moved to zebra-db
• create the zebra-db crate

[mpguerra]

But how much more work would it be to refactor each upgrade into an upgrade trait, and then pass a list of upgrade trait objects to the upgrade task?

I don't think it would be too significant, but it would involve a lot of code movement that could be difficult to review alongside this change.

I was expecting this change to happen in 2-3 PRs:

* optional: change upgrades into a trait

This is #7932

* move zebra-state specific code out of the files that are going to be moved to zebra-db

This can be this issue (#7926)

* create the zebra-db crate

I've split this out into another checklist item in #7728

[mpguerra]

Is this still relevant even if we don't do any more work on the scanner?