BarinderBanwait
commented
1 year ago This seems like a reasonable suggestion to me.
Checking the git blame, what I seem to have implemented 3 years ago were utility functions assert_if_magma and check_sage_compiles_and_extract_var in this file that simulate a user clicking the "Code to SageMath" or "Code to Magma" buttons, and then running those; these seem to only get used in CMF, BMF and HMF tests (in the test_download_sage and test_download_magma tests).
We could have a new kind of test, test_code_snippets that implements the above pseudocode of @JohnCremona ; adding this to the existing test_bmf.py etc would ensure they get run by the automated github actions.
[I'm sure that this has been proposed before, and may even have been implemented in part (@BarinderBanwait ?), but could not see an open issue.]
Issue #5396 prompted this.
It would not be hard to implement the following, and even automate it (at least in the case of the free software, Sage and gp):
For each languange L (=Sage, Magma, GP and others as appropriate); for each section of the LMFDB; come up with a set of test objects (not random, probably, since some will be too large), form the URL for the "download all code" link to language L for the object. Now run L on all the chunks of code in language L and check that it does not crash. For Sage and GP (via "sage -gp") this could probably be added to the automated GitHub actions. Otherwise, we could have separate scripts to download the code and run it locally, to be done at certain times such as a new release version of any of the languages.
An enhanced version would have a lot of "assert" statements in the downloaded code to check that the computed values agree with our data, but this would be more work since it requires more than just concatenating all the snippets in the right order, with comments.