tingerrr / typst-test

A test runner for typst projects.
https://tingerrr.github.io/typst-test/
MIT License
27 stars 2 forks source link

BUG: flaky CI tests when importing package #51

Closed Otto-AA closed 4 months ago

Otto-AA commented 4 months ago

Description

The tests randomly fail in the CI. This does not happen when previously running typst compile (which likely caches the required packages).

Steps to Reproduce the Problem

See: https://github.com/Otto-AA/tt-flaky-mre

Create multiple test cases importing linguify and let them run in github actions (I did not yet test it with other packages, but I guess it is the same. Could also be related to the use of WASM by linguify).

#import "@preview/linguify:0.4.1": linguify, set-database

Test

Expected Behavior

Test should not be flaky and should not depend on previously running typst compile.

Actual Behavior

Several errors occur seemingly random, sometimes the tests also pass.

Context

tingerrr commented 4 months ago

This is a byproduct of the lack of locking from concurrent package downloads, see #16, this is essentially fixed on main, but the rewrite is unfortunately not fully finished.

As you noted yourself, this can be fixed with caching, i.e. by running typst or by extension typst-test on a single test which depends on all packages required once before running the others:

- run: |
  typst-test run dummy-test
  typst-test run
Otto-AA commented 4 months ago

Thanks, that explains it. I guess I need to improve my issue searching skills ^^

Closing, as it's already fixed on main.