Add automatic testing method for Forth words

[scotws]

Currently, Tali Forth words are tested by typing or copy & pasting commands from the documentation into the terminal and comparing this to the output. This is time consuming and error-prone. What is needed is some script or program that will start Tali (through py65mon at least, through other programs like minicom if possible) and then send tests and automatically test responses.

[SamCoVT]

I'm making good progress trying to get John Hayes' test suite working. I found it at: https://www.taygeta.com/forth.html Many of the links are old/broken, but the test software was still available: ftp://ftp.taygeta.com/pub/Forth/Applications/ANS/tester.fr ftp://ftp.taygeta.com/pub/Forth/Applications/ANS/core.fr

It looks like they are free to use/distribute as long as the copyright notice is left intact. That seemed compatible with your distribution of Tali. My thought is to leave these files mainly alone (although I have lowercased them to work with Tali) and put any Tali-specific tests in a separate file. I also changed the extensions to .fs because that's what you were using.

Tali passes most of the tests, and many of your untested words are in here. I need to work on it a bit more, but when I get it all set (and hopefully automated), I'll issue a pull request.

[scotws]

@SamCoVT Cool, thanks! I've found this link as well: http://www.forth200x.org/documents/html/testsuite.html which refers to http://soton.mpeforth.com/flag/anstests/index.html which in turn links to the GitHub page https://github.com/gerryjackson/forth2012-test-suite/tree/master in case any of that helps.

[SamCoVT]

@scotws Those links are helpful. I found the forth200x version, but it was way too complicated for me to figure out what it was doing, mostly because it had optional floating point support set by environment variables. It didn't look like anything that would run on a 65C02 with no OS. With some digging, I found the original version these were all based on, and that was simple enough that I could see what it was doing. It also looked like it might run on Tali with a good lowercasing (fortunately my editor is very good at changing the case of words/files).

My current plan is to continue with the older version that I am using (because it has the shortest path to getting working on Tali) and then investigate what tests might be useful from the forth2012 test suite. They have done a good job breaking up the functions into different files, so that we can easily ignore things we don't have (like files and an environment). I will probably modify the older version to use forth2012's T{ and }T convention (even though we don't have conflicts with { and } like they do) so that it will be easier to use some of their tests.

I've still been running the tests manually, but I'm almost at the point to start automating it. I want to send all of the forth to the simulator and log its entire output to a file. That will help with regression testing because we'll be able to go back and see if a previous version was passing a particular test. Then I can grep through the result file for Tali errors (like unknown words, running a compile-only word, etc), and the two errors from the test program ("WRONG NUMBER OF RESULTS:" or "INCORRECT RESULT:").

[SamCoVT]

33 Pull Request for Initial Test Suite

With your recent updates to Tali (most notably the immediate \ word), I have been able to run the entire test suite. There are only a handful of issues found with these tests now (many of the lines are the same issue over and over), which is impressive given the number of tests performed.

[scotws]

Closed with #33