Parallel main matrix

[centreboard]

Run main build multiple OS and Python versions

- 3.6 and 3.9 as min/max supported version.
- Windows, macOS and Linux
- Cache python site-packages on Windows for install speed
- On Linux/macOS just cache pip wheels

[kneasle]

We can expand the build to run in parallel on Linux + Windows (and even macOS) + different python versions.

I think it might be worth supporting macOS over lots of different Python versions, since I think discrepancies are far more likely to occur between OSes than Python's minor versions. I think perhaps 2 py versions x 3 OSes seems reasonable? Also for some reason our original build CI run is permanently marked as 'expected' for this PR - perhaps GH is just confused by the fact that we changed it...

Also we could perhaps do more strict checks between merging PRs and the code being released - I think the PyPI action is the wrong place, but perhaps we could run a bigger matrix each time master is committed to?

We could cache the python location (though probably not for all 6 runs without being close to 5GB limit) https://medium.com/ai2-blog/python-caching-in-github-actions-e9452698e98d

Perhaps if we did this just for Windows to improve the speed? I think too much caching could lead to us missing things that only happen on fresh python installs...

[centreboard]

I don't have permission to see where build is set as a required checks for PRs to master.

macOS 3.9 is really slow to install numpy when not cached, we'll have to keep submitting PRs to stop the caches expiring :D

Using legacy 'setup.py install' for wrapt, since package 'wheel' is not installed.

Do we need to do pip install wheel?

[centreboard]

In a total of 3 runs I've had a doc test timeout on macOS 3.6 once and another time on macOS 3.9 And now one on Windows 😞

[kneasle]

I don't have permission to see where build is set as a required checks for PRs to master.

It's in the security settings I think (which might only be accessible to me). If the issue is just getting this PR merged, then I can bypass the protection on a per-PR basis. I figured it was probably good practice to protect the master branch like this, although if it's more annoyance than it's worth then I'm happy to just reset the setting...

Do we need to do pip install wheel?

Ooh possibly - this does seem to work, but it's probably worth not using any legacy things for future proofing.

[kneasle]

In a total of 3 runs I've had a doc test timeout on macOS 3.6 once and another time on macOS 3.9 And now one on Windows :disappointed:

Hmmm. Perhaps we could increase the timeout as part of #175?

[centreboard]

/rebase

[centreboard]

Both mac runs had errors, I suspect it doesn't like the number of simultaneous connections

 >>> README.md:140: python ./run-wheatley 238915467 --method "Plain Bob Major" -I 1.0
RETURN CODE: 1
OUTPUT:
Traceback (most recent call last):
  File "/Users/runner/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/site-packages/urllib3/connection.py", line 159, in _new_conn
    conn = connection.create_connection(
  File "/Users/runner/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/Users/runner/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/runner/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/Users/runner/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/runner/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/http/client.py", line 1253, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/runner/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/http/client.py", line 1299, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/runner/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/http/client.py", line 1248, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/runner/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/http/client.py", line 1008, in _send_output
    self.send(msg)
  File "/Users/runner/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/http/client.py", line 948, in send
    self.connect()
  File "/Users/runner/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/Users/runner/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/site-packages/urllib3/connection.py", line 171, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x109ff1100>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/runner/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/Users/runner/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 724, in urlopen
    retries = retries.increment(
  File "/Users/runner/hostedtoolcache/Python/3.9.4/x64/lib/python3.9/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='methods.ringing.org', port=80): Max retries exceeded with url: /cgi-bin/simple.pl?title=Plain+Bob+Major&fields=title%7Cpn%7Cstage (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x109ff1100>: Failed to establish a new connection: [Errno 61] Connection refused'))

[centreboard]

Hmm, I'm tempted to just drop the macOS runs for now as we don't have someone who can debug them

[kneasle]

Hmm, I'm tempted to just drop the macOS runs for now as we don't have someone who can debug them

I agree - I don't think we should drop support for macOS completely but the minority of users use the CLI version and a minority of those use macOS so I don't think it's a particularly big deal. At any rate, macOS and Linux tend to generally work the same since they're both ultimately Unix systems.

Ironically, I suspect that at least 80% of Wheatley's users are running it on Linux without knowing it (i.e. Wheatley itself is running on the RR servers) so I think Ubuntu and Windows should cover nearly everyone.

[centreboard]

To be clear I just meant don't add macOS runs to this matrix, which is the previous situation where they just ran on Linux.

I think you might need to bypass checks to merge this and then edit the required checks afterwards?

[kneasle]

Yep I can merge without the checks finishing. I'll do that now.