scotws / TaliForth2

A Subroutine Threaded Code (STC) ANS-like Forth for the 65c02
Other
86 stars 22 forks source link

Feature/parallel testing #226

Closed SamCoVT closed 5 years ago

SamCoVT commented 5 years ago

I've been working on this for a few days, and I think it's good enough that you should check it out.

This is a parallel version of the test suite that will use as many cores as you have to run the tests in parallel. On my 8-core system it drops the test time from 5:16 to 1:08 and on my 4-core system it drops to 2:02. To run it, simply run the ptest.sh script in the tests folder (or do a make ptests). A word of warning: this will peg your CPU at 100% for a minute or so.

Because core.fs was the longest test (at almost 40K!), I broke it up into core_a.fs, core_b.fs, and core_c.fs. If there is a better way to break up the core tests (eg. by function), I'm all for it. I just found a convenient separator and sliced it there. I also had to copy a few of the words defined in core_a into the top of core_b and core_c to make it all work properly. I didn't delete the core.fs file, but it is currently unused.

The script spawns a copy of py65mon for every test, with each test putting the results into a temporary file named "results_testname.txt" (eg. the tests from ed.fs go into "results_ed.txt"). Once all of the tests have completed, the script concatenates all of the tests together to make the results.txt file. I mention these temporary results files because they will still be there if you CTRL-C the tests, but running ptest.sh again will clean them up.

Currently, the only difference in results.txt between running ptest.sh and running talitest.py (as modified in this pull request) is the "bye" at the very end. ptest.sh has to put a bye on the end of every test that runs, so it strips them all out of the results.

scotws commented 5 years ago

This is sort of funny for me because I had actually thought about how you could probably run these tests in parallel, but then I thought about the hassle of threading on Python (Go ruins you on concurrency with other languages), got a headache, and wandered off to do something else. Of course, your solution of starting one instance per test with a simple shell script is the way to go. It also allows the user to just add a nice if things get too bogged down on the machine. Cool, good idea.

Two minor things we'll have to take care of sometime in the future:

First, if I understand the code correctly, we now have to maintain core.fs and the core_?.fs files in parallel, which sounds like an problem waiting to happen. My suggestion would be to have talitest.py work on your new core_?.fs files as well, then we can just delete that monster core.fs file. I can't think of a logical way to split the core routines either, but since we're not going to be adding much to them that I can see, this should be fine.

(I was wondering if we should put the Gforth tests in a separate file (sees impolite to have them listed under "Tali"), so we could reorganize the whole thing sometime in the future. Maybe we can put all of the tests in a subfolder, too, so the test code can be rewritten to "run any tests you find there" and we don't have to add stuff by hand to those lists anymore.)

Second, README needs to be updated, not only because it still talks about ALPHA and whatnot. We can now offer two ways for testing, parallel and the old way, and should explain that. I'd be grateful if you could handle this because you wrote the parallel code?

This is going to really improve turnaround times, thanks for this. Though I will miss the TV-hacker vibe of having code scroll down a terminal window in the background :-).