Use official tests?

[Ziqi-Yang]

Typst place their tests here
It should be mentioned that the tests is not specialized for syntax, whereas it contains many tests that are used for testing Typst compiler's output(i.e. syntax is correct, but result is wrong or error). However, if we use the official tests, we don't need to racking our brain to create many tests and we can 70%(IMO) make sure we get the correct syntax processing result, (if we totally manage our owns, with the tests growing larger, there may be some duplicates, though they can be properly managed if divided into a proper structure). What we need to do is to filter the tests that is suitable for our syntax testing, while ignoring others
How to do that? For example, this is an unsuitable test file (for testing computing):

==================
official/compute/calc Test 48
==================

// Error: 18-19 number must not be zero
#range(10, step: 0)

---

We just ignore it. This is a suitable one:

==================
official/meta/bibliography-ordering Test 0
==================

#set page(width: 300pt)

@mcintosh_anxiety
@psychology25
@netwok
@issue201
@arrgh
@quark
@distress,
@glacier-melt
@issue201
@tolkien54
@sharing
@restful

#bibliography("/files/works.bib")

---

We can append Enabled tag to the title of this test, and use tree-sitter test -f "Enabled" to test those tests.

I write a simple script to convert the official tests into tree sitter capable ones. By executing the following command, we can get the converted tests under corpus/official folder.

./scripts/update_official_tests.py ~/projects/probe/typst ~/projects/probe/tree-sitter-typst-mine

If we have already created the corpus/official using this script, it will use git diff <test directory> and git diff --name-only <test directory> to get the diff and names of changed files between the latest commit of typst and the current one we are using for tests.

[uben0]

This script is nice to have 👌 I tested it and it works nicely.

I see several improvements:

• If no Typst path provided, tries to fetch the repo from github (using git clone)
• Remove 1 level of depth by regrouping in a single file, all tests from the same folder
• Name test as official/math/op/4 instead of official/math/op Test 4
• Automatically call tree-sitter test -f "official/" -u on the generated tests
• Automatically search for MISSING, EXPECTED or ERROR in the generated trees

What do you think?

[Ziqi-Yang]

Yes, the improvements should be beneficial. And I also think more filtering is needed (and with the ability to automatically generate .gitignore file). I'd like to give it a try later, to see whether the result is beneficial or on the other hand, increase the works we do.

[Ziqi-Yang]

After some more observation into the tests directory of typst repo. I believe I'm wrong. It is not a time-saving approach to directly use the official tests. While using official tests can improve the accuracy, it may greatly increase our workload. In accuracy part should be made up by the issues raised by community.

[uben0]

Verifying one by one the outputted trees is not on the table, because it is too long. But having this script able to detect ERROR, MISSING, or even if there is no segfault or whatever, is a really good thing.