search

haskell-compat / base-compat

A compatibility layer for base
http://hackage.haskell.org/package/base-compat
19 stars 12 forks source link

Travis: Consider only testing the latest point releases of each major GHC version #65

Closed RyanGlScott closed 4 years ago

RyanGlScott commented 4 years ago

Currently, base-compat's Travis setup tests no fewer than 32 (!) different configurations. This seems somewhat excessive, especially given that there are only 11 different major versions of GHC represented among the various configurations. This means that 21 of the configurations simply test different minor releases.

While there is some value in ensuring that base-compat works with every minor release of GHC, it causes the CI turnaround time to be quite high. Moreover, most of the CPP used in base-compat only depends on the major version of GHC being used (it might be all of the CPP, in fact, but I haven't tested this). Therefore, the amount of value we gain from testing every minor release is quite low compared to the significant build times they add in each CI run.

I propose that we simply test against the latest point release of each major GHC version. What do others think?

sjakobi commented 4 years ago

Is the motivation to improve developer ergonomics for you?

Also, has CI ever been useful in detecting a change between minor versions?

RyanGlScott commented 4 years ago

Is the motivation to improve developer ergonomics for you?

Yes. That, and I have to maintain a large collection of hacks to support old minor versions of GHC due to old compiler bugs:

https://github.com/haskell-compat/base-compat/blob/9bf3fc67d262b22e145b47bed4f8bf739000bc38/cabal.haskell-ci#L1-L18

Also, has CI ever been useful in detecting a change between minor versions?

There are a handful of changes to base that were introduced in minor versions, and we might consider backporting those changes to base-compat (see, for instance, the base-4.5.1.0 and base-4.7.0.2 sections in #24). If we did do that, it might be beneficial to test the corresponding versions of GHC that these minor releases of base are shipped with. At present, however, I don't believe testing all minor releases of GHC pays its weight.

sjakobi commented 4 years ago

Seems reasonable! :+1:

phadej commented 4 years ago

If someone writes a small script which searchs out for all the MIN_VERSION_base used, so one can audit that GHC versions on the both side of the conditional one are tested, then I'd be happy.

BTW, disabling base-compat-batteries can be done using jobs-selections: any and tweaking tested-with in base-compat-batteries.

RyanGlScott commented 4 years ago

If someone writes a small script which searchs out for all the MIN_VERSION_base used, so one can audit that GHC versions on the both side of the conditional one are tested, then I'd be happy.

I'm not sure how fancy of a script you're envisioning, but this crude one-liner tells you how many occurrences of each variant of MIN_VERSION_base are currently used in base-compat:

$ grep -rh "MIN_VERSION_base" base-compat/src/ base-compat-batteries/src/ | sort | uniq -c
      1 #elif MIN_VERSION_base(4,10,0)
      1 # if !(MIN_VERSION_base(4,10,0))
      1 # if MIN_VERSION_base(4,10,0)
      5 #if !(MIN_VERSION_base(4,10,0))
     11 #if MIN_VERSION_base(4,10,0)
      2 #if MIN_VERSION_base(4,10,0) && !(MIN_VERSION_base(4,11,0))
      1 #  if MIN_VERSION_base(4,10,0) && !(MIN_VERSION_base(4,12,0))
      4 #if MIN_VERSION_base(4,10,0) && !(MIN_VERSION_base(4,12,0))
      7 #if !(MIN_VERSION_base(4,11,0))
      1 #if MIN_VERSION_base(4,11,0)
      4 #if MIN_VERSION_base(4,12,0)
      1 # if !(MIN_VERSION_base(4,13,0))
      1 #if !(MIN_VERSION_base(4,4,0))
      1 #if !(MIN_VERSION_base(4,4,0)) || MIN_VERSION_base(4,8,0)
      2 #if MIN_VERSION_base(4,4,0) && !(MIN_VERSION_base(4,8,0))
      1 #if !(MIN_VERSION_base(4,4,0)) || MIN_VERSION_base(4,9,0)
      2 #if MIN_VERSION_base(4,4,0) && !(MIN_VERSION_base(4,9,0))
      3 #if !(MIN_VERSION_base(4,5,0))
      1 #if !(MIN_VERSION_base(4,5,0)) && !(MIN_VERSION_base(4,9,0))
      1 # if !(MIN_VERSION_base(4,6,0))
      6 #if !(MIN_VERSION_base(4,6,0))
      4 #if MIN_VERSION_base(4,6,0)
     12 #if !(MIN_VERSION_base(4,7,0))
     11 #if MIN_VERSION_base(4,7,0)
      1 #if MIN_VERSION_base(4,7,0) && !(MIN_VERSION_base(4,8,0))
      1 #if !(MIN_VERSION_base(4,7,0)) || MIN_VERSION_base(4,9,0)
      2 # if !(MIN_VERSION_base(4,8,0))
     13 #if !(MIN_VERSION_base(4,8,0))
     13 #if MIN_VERSION_base(4,8,0)
      2 # if MIN_VERSION_base(4,9,0)
     10 #if !(MIN_VERSION_base(4,9,0))
     27 #if MIN_VERSION_base(4,9,0)
      1 #if MIN_VERSION_base(4,9,0) && !MIN_VERSION_base(4,10,0)
      1 #if MIN_VERSION_base(4,9,0) && !(MIN_VERSION_base(4,11,0))
      1 #if MIN_VERSION_base(4,9,0) && !(MIN_VERSION_base(4,13,0))

There's some duplication in there, but the point I want to make shines through: every use of MIN_VERSION_base at present uses 0 as the tertiary version number.

phadej commented 4 years ago

@RyanGlScott so the hacks are actually more like,

base-compat-batteries doesn't work with GHC-7.8.1

and not that we have some code written differently specifically for GHC-7.8.1 (or some other minor version)?

RyanGlScott commented 4 years ago

Right, the issues I have to work around are exclusively old GHC bugs, not API differences in base-compat{-batteries}. For instance, I run into https://gitlab.haskell.org/ghc/ghc/issues/8978 when building base-compat-batteries with GHC 7.8.1, which does not affect any other GHC release.

RyanGlScott commented 4 years ago

It sounds like there is a consensus to proceed with this idea, so I have implemented this in #66.