Cloning from Dremio v25.1.0 to pre-v25.1.0 fails with "Invalid value found at: format.ignoreOtherFileFormats"

[dullbouy]

Dremio introduced a new API parameter "ignoreOtherFileFormats" in v25.1.0 when promoting parquet datasets.

See: https://docs.dremio.com/current/reference/api/catalog/table/#parameters-of-the-format-object

If Dremio-Cloner is used to clone from a 25.1.0 cluster to a pre-v25.1.0 cluster the promotion of parquet based PDS's will fail with the following error :

 errorMessage: Invalid value found at: format.ignoreOtherFileFormats moreInfo:
2024-11-18T11:52:49.9939332Z ERROR:2024-11-18 11:52:49,993:_promote_pds: Error promoting PDS:

[deane-dremio]

I do not believe the description of the issue is accurate. Isn't this more the case that the customer is using Cloner to DOWNGRADE from a v25.1.0+ version to a pre-25.1.0 version, so what they are doing is first reading the v25.1.0+ data which DOES contain the ignoreOtherFileFormats field, then when we try to write that data into a pre-v25.1.0 cluster it fails because the pre-v25.1.0 cluster has no idea about the existence of the ignoreOtherFileFormats field?

[dullbouy]

You are correct Deanne, I had it back to front, I have now corrected the decription

[mxmarg]

I tried a quick fix on this branch, which tries to remove the offending field: https://github.com/dremio-professional-services/dremio-cloner/tree/v25.2-to-v25.1-backwards-compat