search

adoptium / temurin-build

Eclipse Temurin™ build scripts - common across all releases/versions
Apache License 2.0
996 stars 243 forks source link

introduce bash unit-tests #3741

Open judovana opened 2 months ago

judovana commented 2 months ago

hello! Was there any attempt to include tests at least to temurin-builds? Looking to https://github.com/adoptium/temurin-build/pull/3736 or https://github.com/adoptium/temurin-build/pull/3728 are excellent candidates... And I belive in past thre were many simialar pathces introducing standalone single function.

I would vote to use my ancient and favourite: https://github.com/rh-openjdk/run-folder-as-tests as it offers ultimate freedom, to to run individual sets, driven by: https://github.com/lehmannro/assert.sh https://github.com/kward/shunit2 https://github.com/bmizerany/roundup https://github.com/sstephenson/bats https://shellspec.info/

wdyt?

From those, I have strong exepreince only with run-folder-as-tests, but as it can wrap all others, it may be good start. I have also some experience with shunit2 which works quite well, but (simiarly as assert.sh) I would not use it as standalone solution. I have no experience with bats (which is quite recomended over web, but rep is archived) and with roundup. On roundup (acording to documentation) its sandbox is cool. But even the run-folder-as-tests is running eah unit as standalone process.

One of the conditionnal steps may be extracting various build.sh, makeAnyJdk.sh, prepareWorkSpace.sh to build.sh, makeAnyJdk.sh, prepareWorkSpace.sh and buildFunctions.sh, makeAnyJdkFunctions.sh, prepareWorkSpaceFunctions.sh.

karianna commented 2 months ago

GPT-4 says:

There isn't a single de facto standard for unit testing shell scripts as the choice often depends on the specific needs of a project, the complexity of the shell scripts, and the preferences of the development team. However, several popular frameworks and tools are commonly used in the industry. Here are a few notable ones:

  1. shUnit2: An xUnit based unit test framework for Bourne based shell scripts, shUnit2 is one of the older and more established frameworks. It provides a straightforward way to write and run tests, making it a popular choice for many developers.

  2. Bats (Bash Automated Testing System): Bats is a TAP-compliant testing framework for Bash scripts. It provides a simple way to write tests right in Bash, using special @test syntax. Bats also supports test setup and teardown, making it suitable for more complex test scenarios.

  3. ShellSpec: A full-featured BDD (Behavior Driven Development) unit testing framework for shell scripts. ShellSpec supports various shells (not just Bash) and offers a rich set of features, including mocking and coverage reporting. Its syntax is inspired by RSpec, making it familiar to Ruby developers.

  4. bash_unit: A simple Bash script to automate testing of Bash scripts, applications, and shell libraries. It's less complex than some of the other options, making it easy to get started with minimal setup.

  5. Roundup: Another simple framework for writing and running tests for shell scripts. Roundup provides an easy-to-understand syntax and is designed to be portable and easy to deploy.

Each of these tools has its strengths and is suitable for different kinds of projects and preferences. For simpler scripts or projects where Bash is not the primary focus, shUnit2 or bash_unit might be sufficient. For projects requiring more sophisticated testing features, such as mocking or BDD-style tests, ShellSpec or Bats could be more appropriate.

The best approach is often to evaluate a few of these frameworks based on the specific requirements of your project and choose the one that best fits your needs.

karianna commented 2 months ago

2 looks interesting to me because we already have TAP-compliant testing elsewhere

judovana commented 2 months ago

If the tap is generated by suite, it may be disadvantage, as in aqa-tests, the tap files are global. So if the tests would pass, it would be one huge tap file with one test temurin-build passed/failed. Eg I had to disable individual tap files in churn testsuite, becasue it was contraproductive with the main aqa-test tap file, which needed just one tap-file with one test churn, from churn, not details. In addition tap format is ancient, and despite I really like it, it is error prone. Modern xunit xml files are much less error prone. There is nice article of https://medium.com/wemake-services/testing-bash-applications-85512e7fe2de comparing all except shellSpec

The https://github.com/rh-openjdk/run-folder-as-tests can speak tap files to. and is really surprising powerful as top level runner.

I would agree that bats is probably best, but it do not speak xunit - which is really much more reliable then tap, and is already decommissioned project:(

With that gone, I would go with run-folder-as-tests and using individual frameworks as sub-suites, generate both tap and xml , and in individual sub suites, use any of the offered frameworks as needed. Tbh, shellspec is to abstract for me.