chore(mailman): gate non-breaking releases on GNU Mailman 3 #2031

Closed vytas7 closed 2 years ago

This pull request adds a GNU Mailman 3 CI gate that is proposed to be run on every master merge. Upon every run, Mailman Core is cloned from GitLab, its tox.ini patched to add an additional falcon env where we simply uninstall Falcon and reinstall it from local source (effectively building a cythonized PEP 517 wheel).

~~A successful run is demoed here: https://github.com/falconry/falcon/runs/5279630719?check_suite_focus=true~~

The following concessions had to be made, because of the way the things are, my incompetence, or a combination thereof:

~~I couldn't use @maxking 's Docker image, because GitHub Actions seem to be very hostile to setting a non-root USER.~~
I wished to run tests for a stable release of Mailman instead of the latest master commit. However, it seems that some tests inside the 3.3.5 tree are failing, so I abandoned this idea for the time being.

Edit: now using the Falcon flavour of @maxking's image. An example of the latest incarnation: https://github.com/falconry/falcon/runs/5351987534?check_suite_focus=true.

It is looking unlikely at this point that we'll be able to resurrect the Hug gate (Hug is still on Falcon 2.0), but I have left the scripts and definitions in question intact for now. (Because HOPE dies last as they say!)

Closes #259

Codecov Report

Merging #2031 (35f9067) into master (d2c1306) will not change coverage. The diff coverage is n/a.

@@            Coverage Diff            @@
##            master     #2031   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           63        63           
  Lines         6707      6707           
  Branches      1238      1238           
=========================================
  Hits          6707      6707

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update d2c1306...35f9067. Read the comment docs.

@maxking Agreed re benefits of a specialized Docker image. I would be happy to try such an image in the case it isn't too much effort for you to create one.

Alternatively, tips are welcome on how to marry Actions to a container employing USER. Quick googling didn't help much, rather confirmed what I suspected: https://docs.github.com/en/actions/creating-actions/dockerfile-support-for-github-actions#user.

I would be happy to try such an image in the case it isn't too much effort for you to create one.

https://github.com/maxking/mailman-ci-runner/pkgs/container/mailman-ci-runner-falcon

I added an USER root at the end of the Dockerfile and re-built the same image. Let me know if that helps with GH Actions.

@vytas7 I think part of the problem is that due to switching user, the $PATH isn't set correctly, so because of /usr/local/bin/ not being on $PATH.

You might be able to fix that in the tox env, but I think it should really be fixed in the container image. I should've tried the image before giving it out, so my bad :( I'll try to add it to the image this evening. You might be able to run quick test by adding export PATH=/usr/local/bin:$PATH in commands.

Nevermind, looks like you've got them kicking already with moving the checkout to the final command.

No worries @maxking. Against all the odds, something is running there :crossed_fingers: https://github.com/falconry/falcon/runs/5309572521?check_suite_focus=true

Hm, same problems as I had on my home workstation, IPv6 doesn't work it seems. And some other failures that I haven't seen before :thinking:

One suggestion would be to configure mailman to bind to 127.0.0.1 instead of localhost and see if that stops mailman from trying to bind to ipv6.

https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/config/schema.cfg#L418

The default is localhost, so it depends on the entries in/etc/hosts I suppose.

https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/testing/testing.cfg#L31 is where we can make the change upstream. If this fails locally for you, can you add hostname: 127.0.0.1 locally and see if that fixes tests for you? We can get this upstream too, atleast in the testing config.

And how about other two failures, is that just fallout from the former?

Update: I tried locally and it really didn't help, just introduced 127.0.0.1 where the tests tried to assert localhost:

======================================================================
ERROR: test_mailman_command_without_subcommand_prints_help (mailman.bin.tests.test_mailman.TestMailmanCommand)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/mock.py", line 1325, in patched
    return func(*newargs, **newkeywargs)
  File "/tmp/falcon/.ecosystem/mailman/src/mailman/bin/tests/test_mailman.py", line 70, in test_mailman_command_without_subcommand_prints_help
    help_commands = list(
  File "/tmp/falcon/.ecosystem/mailman/src/mailman/bin/tests/test_mailman.py", line 71, in <genexpr>
    line.split()[0].strip()
IndexError: list index out of range

======================================================================
ERROR: LayerSuite
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/falcon/.ecosystem/mailman/src/mailman/testing/layers.py", line 243, in setUp
    cls.smtpd.start()
  File "/tmp/falcon/.ecosystem/mailman/src/mailman/testing/mta.py", line 177, in start
    super().start()
  File "/tmp/falcon/.ecosystem/mailman/.tox/py38-nocov/lib/python3.8/site-packages/aiosmtpd/controller.py", line 210, in start
    raise self._thread_exception
  File "/tmp/falcon/.ecosystem/mailman/.tox/py38-nocov/lib/python3.8/site-packages/aiosmtpd/controller.py", line 176, in _run
    srv: AsyncServer = self.loop.run_until_complete(self.server_coro)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server
    raise OSError(err.errno, 'error while attempting '
OSError: [Errno 99] error while attempting to bind on address ('::1', 9025, 0, 0): cannot assign requested address

======================================================================
ERROR: LayerSuite
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/falcon/.ecosystem/mailman/src/mailman/testing/layers.py", line 297, in setUp
    cls.smtpd.start()
  File "/tmp/falcon/.ecosystem/mailman/src/mailman/testing/mta.py", line 177, in start
    super().start()
  File "/tmp/falcon/.ecosystem/mailman/.tox/py38-nocov/lib/python3.8/site-packages/aiosmtpd/controller.py", line 210, in start
    raise self._thread_exception
  File "/tmp/falcon/.ecosystem/mailman/.tox/py38-nocov/lib/python3.8/site-packages/aiosmtpd/controller.py", line 176, in _run
    srv: AsyncServer = self.loop.run_until_complete(self.server_coro)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server
    raise OSError(err.errno, 'error while attempting '
OSError: [Errno 99] error while attempting to bind on address ('::1', 9465, 0, 0): cannot assign requested address

======================================================================
ERROR: LayerSuite
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/falcon/.ecosystem/mailman/src/mailman/testing/layers.py", line 352, in setUp
    cls.smtpd.start()
  File "/tmp/falcon/.ecosystem/mailman/src/mailman/testing/mta.py", line 177, in start
    super().start()
  File "/tmp/falcon/.ecosystem/mailman/.tox/py38-nocov/lib/python3.8/site-packages/aiosmtpd/controller.py", line 210, in start
    raise self._thread_exception
  File "/tmp/falcon/.ecosystem/mailman/.tox/py38-nocov/lib/python3.8/site-packages/aiosmtpd/controller.py", line 176, in _run
    srv: AsyncServer = self.loop.run_until_complete(self.server_coro)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server
    raise OSError(err.errno, 'error while attempting '
OSError: [Errno 99] error while attempting to bind on address ('::1', 9587, 0, 0): cannot assign requested address

======================================================================
FAIL: test_transaction_commit_after_successful_subcommand (mailman.bin.tests.test_mailman.TestMailmanCommand)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/mock.py", line 1325, in patched
    return func(*newargs, **newkeywargs)
  File "/tmp/falcon/.ecosystem/mailman/src/mailman/bin/tests/test_mailman.py", line 99, in test_transaction_commit_after_successful_subcommand
    self.assertEqual(mlist.volume, 6)
AssertionError: 5 != 6

======================================================================
FAIL: test_linked_nonmember_findable_after_posting (mailman.rest.tests.test_membership.TestNonmembership)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/falcon/.ecosystem/mailman/src/mailman/rest/tests/test_membership.py", line 756, in test_linked_nonmember_findable_after_posting
    self.assertEqual(
AssertionError: 'http://127.0.0.1:9001/3.0/addresses/nonmember@example.com' != 'http://localhost:9001/3.0/addresses/nonmember@example.com'
- http://127.0.0.1:9001/3.0/addresses/nonmember@example.com
?        ^^^^^^^^^
+ http://localhost:9001/3.0/addresses/nonmember@example.com
?        ^^^^^^^^^

======================================================================
FAIL: test_nonmember_findable_after_posting (mailman.rest.tests.test_membership.TestNonmembership)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/falcon/.ecosystem/mailman/src/mailman/rest/tests/test_membership.py", line 725, in test_nonmember_findable_after_posting
    self.assertEqual(
AssertionError: 'http://127.0.0.1:9001/3.0/addresses/nonmember@example.com' != 'http://localhost:9001/3.0/addresses/nonmember@example.com'
- http://127.0.0.1:9001/3.0/addresses/nonmember@example.com
?        ^^^^^^^^^
+ http://localhost:9001/3.0/addresses/nonmember@example.com
?        ^^^^^^^^^

----------------------------------------------------------------------
Ran 1472 tests in 216.973s

FAILED (failures=3, errors=4, skipped=1, expected failures=3)

This was my diff btw (bar tox.ini patches):

--- a/src/mailman/testing/testing.cfg
+++ b/src/mailman/testing/testing.cfg
@@ -29,6 +29,7 @@ incoming: mailman.testing.mta.FakeMTA
 configuration: python:mailman.testing.passlib

 [webservice]
+hostname: 127.0.0.1
 port: 9001
 workers: 1

Ok, so I've been trying to get the tests running inside a Github Codespace, hopefully that is close enough environment to Actions. The failing tests are failing because we prevent running of mailman commands as root by default (some users accidentally run as root and then cause a whole lot of permission issues), so running the tests as root triggered those code paths.

I seem to have fixed the issue with this MR but I saw another failure running as root, the fix for it is still running CI, but should be merged soon.

Finally, the ipv6 issue is difficult to fix since it is coming from aiosmtpd and moving to an IP address (127.0.0.1) breaks a bunch of other SSL related stuff we have for smtps testing. So, what I did was get rid of ::1 from /etc/hosts and that worked.

sed '/::1/d' /etc/hosts > /tmp/hosts && cat /tmp/hosts > /etc/hosts

(For some reason, i can't do in-place edit using sed in the container, it fails to write to a temp file, even though running as root should allow me to do that. I didn't dig further and just used a temp file instead)

After this, I was able to run the full test suite (from this MR)

----------------------------------------------------------------------
Ran 2095 tests in 467.698s

OK (skipped=1, expected failures=3)

Update: https://gitlab.com/mailman/mailman/-/merge_requests/967 is now merged, so master should pass tests.

Thanks a lot for digging into this @maxking, the tests pass now. The question is of course the maintainability of these hacks compared to simply using Actions' default env, and bumping the Python version when needed. But since you've put lots of good work into this it would be a shame to go back.

Could you publish that CI image onto the Docker hub? Or should I refer to it in the current way? (I had to look to its push actions to get the exact reference to :master etc on ghcr.io).

@vytas7 with the docker hub pull restrictions, I had to move all my images to ghcr.io actually for use in CI, because it will go over the limit pretty fast. The :master tag is probably the right one and you have the right pull URL. I updated the readme in the repo on instructions to pull the image, but I don’t know how to actually make https://github.com/maxking/mailman-ci-runner/pkgs/container/mailman-ci-runner-falcon/15817109?tag=master this page show up on the top of the package info for users.

About the hacks, yeah the sed hack is really not ideal and it did end up being more work that initially expected to get the container image and tests running. I wonder why do those issues not appear in default actions, I thought all of them run in containers 🤔

No problems re the ghcr.io URL as long as it is confirmed to be the intended form.

These containers are indeed not created equal, it seems that if you use a custom docker:// action, it is run via some sort of secondary containerization from inside a top level container (using GH stock images).

Thanks for all the effort on this @maxking! :+1:

falconry / falcon

chore(mailman): gate non-breaking releases on GNU Mailman 3 #2031

Codecov Report