ABS tests

[odino]

Currently, we only test ABS' lexer, parser, evaluator and ability to build cross-platform At some point, we should definitely have a set of scripts to ensure BC is not broken when releasing a new version of ABS.

What I'm thinking is something like:

• having a folder that contains example ABS scripts with expected output
• during each Travis build, loop through that folder and do abs $script, then compare the result

This would allow understanding whether some feature is introducing backward-incompatible changes as well as let us really test ABS cross-platform.

[ntwrick]

@odino: Take a look at the Bash Automated Testing System (BATS). This might be useful for automating the testing of our ABS scripts in tests/*.abs.

See https://github.com/bats-core/bats-core https://testanything.org/ https://opensource.com/article/19/2/testing-bash-bats

I have created a POC for testing these scripts on branch bats-testing. Check it out and take a look at my notes and test cases in tests/test-abs.bats. In particular, note that the TAP format output will be what you get when automation is running instead of the interactive format seen in a terminal via $ bats tests/test-abs.bats. You can force TAP format output in the terminal by using $ bats --tap tests/test-abs.bats to see what the automation will see.

I'm not sure how you integrate this with Travis etc. in github, but it looks promising for driving external ABS script tests including using ABS Init Files, OS env variables, and sourced ABS scripts. Still not an answer for testing the ABS interactive REPL mode, but it seems to handle testing the scripting mode pretty well.

And, another thought: maybe we could re-implement this using ABS instead of bash. This might give us better cross-platform testing capability as BATS is pretty-much Linux bash specific. The downside is that if we break ABS++ then ABS-- has to be stable for testing to work.

[odino]

I'll have a look, thanks :)

On Sat, Feb 23, 2019 at 8:08 PM Erich (Rick) Arnold < notifications@github.com> wrote:

@odino https://github.com/odino: Take a look at the Bash Automated Testing System (BATS). This might be useful for automating the testing of our ABS scripts in tests/*.abs.

See https://github.com/bats-core/bats-core https://testanything.org/ https://opensource.com/article/19/2/testing-bash-bats

I have created a POC for testing these scripts on branch bats-testing. Check it out and take a look at my notes and test cases in tests/test-abs.bats. In particular, note that the TAP format output will be what you get when automation is running instead of the interactive format seen in a terminal via $ bats tests/test-abs.bats. You can force TAP format output in the terminal by using $ bats --tap tests/test-abs.bats to see what the automation will see.

I'm not sure how you integrate this with Travis etc. in github, but it looks promising for driving external ABS script tests including using ABS Init Files, OS env variables, and sourced ABS scripts. Still not an answer for testing the ABS interactive REPL mode, but it seems to handle testing the scripting mode pretty well.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/abs-lang/abs/issues/95#issuecomment-466665047, or mute the thread https://github.com/notifications/unsubscribe-auth/AAUC5AYtFTvKqpn2zxW8BD9AsiI6GXY_ks5vQWeEgaJpZM4Zj5Rw .

-- Alex Nadalin http://www.linkedin.com/in/alessandronadalin www.odino.org www.twitter.com/odino