search

exercism / julia

Exercism exercises in Julia.
https://exercism.org/tracks/julia
MIT License
68 stars 68 forks source link

GHA workflows - are we doing too much CI testing? #787

Open colinleach opened 2 months ago

colinleach commented 2 months ago

The Julia track currently tests 3 versions of Julia against 3 OS's, which seems unusual compared with similar tracks. This is the relevant part of exercise-tests.yml:

  test:
    name: Julia ${{ matrix.julia-version }} - ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        julia-version: ["1.6", "1", nightly]
        os: [ubuntu-22.04, windows-2022, macos-14]
        exclude:
          - julia-version: 1.6
            os: macos-14

Python and R both seem fine with just using ubuntu-22.04. Even C# uses only ubuntu-20.04, despite its Windows heritage.

I'm particularly keen to get rid of the Windows/Julia 1.6 combination, which is glacially slow (15-20 minutes, compared with <5 min for all the others). That should go away in any case later this year, when 1.6 is due to be dropped as the LTS version, but maybe Windows and Mac are now inappropriate?

I don't know the history of this choice, but perhaps it made more sense when Julia was less mature and stable, so there could be important differences between platforms.

Thoughts?

depial commented 2 months ago

I'm unaware of the reasons for testing all the OSs, but did notice the issue with Windows a while ago (if I remember correctly it hangs longest when loading dependencies).

We did have an issue with MacOS and Julia 1.6 a few months ago which @cmcaine believed was due to 1.6 not supporting ARM versions of MacOS. There's a bit more detail in PR #733 if you want to have a look.

colinleach commented 2 months ago

We did have an issue with MacOS and Julia 1.6 a few months ago

Presumably, that explains the exclude: in the testing. It still leaves me wondering why we test Mac and Windows, when other tracks don't.

I asked about this on Discord yesterday but with no replies so far.

BNAndras commented 1 month ago

The likely answer to me is the boring one. Most tracks don’t bother testing multiple platforms since they don’t have to. Exercism exercises generally don’t go in directions where platform differences may become significant like I/O or networking. Linux is the default platform to test since most if not all test runners run Linux.