MacOS Catalina test failures

[mikestillman]

I find that under MacOS Catalina, with apple clang (12.0.0), under the cmake build some tests are failing for me, even though they apparently pass the git actions.

(I will edit with results of autotools build when that has completed)..

1. cmake build using apple clang (after doing ninja M2-unit-tests, ctest):

• normal/commands.m2 -- fails after about 900 seconds
• normal/schenck-book-7.m2 --Error regular expression found in output. Regex=[--back trace--]
  • (This one is probably being handled in another git issue)
• normal/threads.m2 -- Error regular expression found in output. Regex=[--back trace--]
• all of the ComputationsBook/ chapters
• there are also some in quarantine, goals, engine that fail (but I expected these)

2. autotools build using apple clang

[mahrud]

• normal/commands.m2 -- fails after about 900 seconds
• normal/threads.m2 -- Error regular expression found in output. Regex=[--back trace--]

These two are known issues. The reason is simply that unlike the autotools tests where the input is simply piped into M2, ctest runs the tests using C exec, not through a shell, so there is no piping. The workaround I came up with was to use the M2 load function and read the test file, which meant anything that interrupts load interrupts the test. Also, they don't pass on git actions because I've excluded them: https://github.com/Macaulay2/M2/blob/afaa83b5c6062dcbfc7f383bf4786d17ed0a0c3d/.github/workflows/test_build.yml#L173

• normal/schenck-book-7.m2 --Error regular expression found in output. Regex=[--back trace--]

You figured this one out here: https://github.com/Macaulay2/M2/issues/1108#issuecomment-620309033. Probably #1472 would fix this, but I keep forgetting to do it.

• all of the ComputationsBook/ chapters

This one is kind of weird ... we need to generate the output files first using ninja M2-tests-ComputationsBook, then ctest -R ComputationsBook tests would read the output file, trim them, and take diff with the expected output. If you just run ninja M2-tests-ComputationsBook they should all pass.

• there are also some in quarantine, goals, engine that fail (but I expected these)

Yes: #1213

[mahrud]

Can this be closed?

[mikestillman]

Actually, I just did last night or night before a compilation (on branch based off of development from a few days ago), and I get the same errors (in particular, normal/commands.m2 is especially annoying as it takes 900 seconds to time out!). It would be nice to be able to run the tests to completion and have them pass. What do we still need to do to get that to happen?

[mahrud]

It would be nice, but I don't think that can happen. The problem is that those two tests are interactive in nature, so they are not compatible with the way CTest runs the tests using input("threads.m2")

I could do something else, like use runString inside CTest, but that would reduce the usefulness of CTest (e.g. for running tests in parallel, etc), so I don't want to do that.

I could also add a line or two in tests/CMakeLists.txt to skip those two files, but then one might forget that those two are being skipped, so I'd rather you use ctest -E "command|threads" instead. Last option would be to move those two tests into a directory named interactive, which CTest would skip. This has the same issue.

In summary: I can't think of how those two tests can be run through input.

[mahrud]

Closing in favor of #3421.