We want to avoid caching them when they're not needed and ideally allow for pruning them.
Keeping these indexes behind a feature flag or only adding them when enabled in the configuration would require Zebra to concurrently support multiple db formats if/when another db format version bump is required.
If Zebra starts with the indexer feature, then shuts down and restarts without the feature, the db format version on disk will still match the one in code, but the indexes won't be available in the state cache.
Possible Design
Zebra could add build metadata to the db format version to indicate whether the state cache has an index of spending transaction ids.
It could then remove the build metadata from the on-disk db format version when it starts without the feature or indexing enabled in the configuration, and always run the db format upgrade for adding the spending transaction id indexes when it's missing from the on-disk db format version's build metadata.
Zebra could also track which blocks should have the indexes in a new column family so it can start where it left off instead of traversing the entire chain up to the initial tip height if necessary.
Motivation
The indexes added in https://github.com/ZcashFoundation/zebra/pull/8895 use more storage than expected and are only required for Zebrad nodes that are supporting a light client server.
We want to avoid caching them when they're not needed and ideally allow for pruning them.
Keeping these indexes behind a feature flag or only adding them when enabled in the configuration would require Zebra to concurrently support multiple db formats if/when another db format version bump is required.
If Zebra starts with the indexer feature, then shuts down and restarts without the feature, the db format version on disk will still match the one in code, but the indexes won't be available in the state cache.
Possible Design
Zebra could add build metadata to the db format version to indicate whether the state cache has an index of spending transaction ids.
It could then remove the build metadata from the on-disk db format version when it starts without the feature or indexing enabled in the configuration, and always run the db format upgrade for adding the spending transaction id indexes when it's missing from the on-disk db format version's build metadata.
Zebra could also track which blocks should have the indexes in a new column family so it can start where it left off instead of traversing the entire chain up to the initial tip height if necessary.