We run this on all versions of Cyrus that have any kind of JMAP support, but JMAP::TestSuite matches the currently-developing spec, which older experimental releases may no longer match, but also might not be upgradable to match.
We probably need some mapping of tests to Cyrus minimum version numbers, so that Cassandane can selectively suppress the ones it knows will not pass. We already do this for releases that do not support JMAP at all, but the "supports JMAP, experimentally" releases will eventually become stale and no longer pass tests, and we shouldn't be failing CI just for old experimental features.
The one example so far is that 3.2 now fails JMAPTestSuite.t:Email:get:header-header-field-name as of the last few weeks, but I'm sure there will be more over time...
We run this on all versions of Cyrus that have any kind of JMAP support, but JMAP::TestSuite matches the currently-developing spec, which older experimental releases may no longer match, but also might not be upgradable to match.
We probably need some mapping of tests to Cyrus minimum version numbers, so that Cassandane can selectively suppress the ones it knows will not pass. We already do this for releases that do not support JMAP at all, but the "supports JMAP, experimentally" releases will eventually become stale and no longer pass tests, and we shouldn't be failing CI just for old experimental features.
The one example so far is that 3.2 now fails
JMAPTestSuite.t:Email:get:header-header-field-nameas of the last few weeks, but I'm sure there will be more over time...