Moonsong-Labs / moonwall

Testing Framework for Substrate networks
https://moonsong-labs.github.io/moonwall/
GNU General Public License v3.0
10 stars 4 forks source link

Skipping Tests take so much time and require going through every cases #410

Open dr-manhattan-1987 opened 2 months ago

dr-manhattan-1987 commented 2 months ago

If I currently run moonwall test "D0103" then moonwall will waste time and CPU, going through every test file, add better ways to control which tests to run and provide good ways to skip or run tests.

Ex: moonwall test test-balances only run tests from test-balances dir, no need to traverse all dir in suites

Ex: moonwall test test-balances/test_specific_test.ts only run that specific test file Ex: moonwall test test-balances/test_specific_test.ts::D0103 only run that specific describe suite

I like how pytest handles it, jest is also not bad.

timbrinded commented 2 months ago

This is super annoying, and but is part of the underlying's vitest library behaviour - it loads all the test files first and that helps with its concurrency management. It's not a fix but i have the option --subDirectory / -d to limit it to only search in a specific directory.

dr-manhattan-1987 commented 2 months ago

Gotcha! I like the -d flag but I think it maybe better if instead of -d being dir string it can be array of dir string So, moonwall test -d test-balance -d test-block -> only tests test-balance and test-block

Although not ideal since it will still require scanning the subdir which can have lots of files and I can do something like this now:

moonwall test -d test-balance "D010301"

This will still scan the dir for all the files. But What if we add another flag -t that picks up the array of test file put it in a temp directory and run the test on it. This can be quick fix for this issue.

I also think there should be an option to manually configure concurrency Ex: In rust we can easily do cargo build --jobs 16 but here I can't configure it.

timbrinded commented 2 months ago

the subdir option is really not ideal. What i would prefer is if vitest didnt prescan everything but i am hoping they just fix it upstream eventually.