After some digging I found out that this is indeed the reason why we did not notice this breaking change. Testing with other docker images I get the following results
paritypr/substrate:master-ae864e6a broken
paritypr/substrate:8dbe4ee8 broken
paritypr/substrate:master-eda98954 works
paritypr/substrate:master-ad54bc36 works
Commit 8dbe4ee8 introduced the CheckMetadataHash extension
RFC-0078 Merkelized Metadata proposes to include a metadata hash in the extrinsic signature to improve trust in the protocol: https://polkadot-fellows.github.io/RFCs/approved/0078-merkleized-metadata.html
Open questions