[Spark] Protocol version downgrade in the presence of table features

[andreaschat-db]

Which Delta project/connector is this regarding?

• [x] Spark
• [ ] Standalone
• [ ] Flink
• [ ] Kernel
• [ ] Other (fill in here)

Description

This PR adds support for protocol versions downgrade when table features exist in the protocol. The downgraded protocol versions should be the minimum required to support all available table features. For example, Protocol(3, 7, DeletionVectors, RowTracking) can be downgraded to Protocol(1, 7, RowTracking) after removing the DV feature.

How was this patch tested?

Added new UTs in DeltaProtocolVersionSuite. Furthermore, existing UTs cover a significant part of the functionality. These these are the following:

• Downgrade protocol version on table created with (3, 7).
• Downgrade protocol version on table created with (1, 7).
• Protocol version downgrade on a table with table features and added legacy feature.
• Protocol version is not downgraded when writer features exist.
• Protocol version is not downgraded when reader+writer features exist.
• Protocol version is not downgraded when multiple reader+writer features exist.

Does this PR introduce any user-facing changes?

Yes. Dropping a table feature from a table with multiple features may now result to a Protocol versions downgrade. For example, Protocol(3, 7, DeletionVectors, RowTracking) can now be downgraded to Protocol(1, 7, RowTracking).

[felipepessoto]

Is the description correct? Reader v1 doesn’t support DV

[larsk-db]

Is the description correct? Reader v1 doesn’t support DV

I think the intent of "can be downgraded" was an implicit "after removing the DV feature".