lukego
commented
7 years ago Good writeup! To me it makes sense to keep snabb minimal but to be more liberal with related tooling e.g. tests, benchmarks, log analysers, etc. It's definitely important to nail down the dependencies but this can naturally by done by making them run under Hydra.
So 👍 from me on writing tests suites and related tools using the technologies that you consider most appropriate and providing a nix expression to make them run with the right dependencies. This way we can always run the tests under the Hydra CI with consistent dependency versions, lazy people like me can use nix to install the dependencies automatically in development environments, and people can also use any other methods they like personally e.g. apt-get, yum, docker, make install, etc.
Sound reasonable?
Further braindump...
I have been pondering this quite a bit lately because I am working on Snabb-related development tools that I want to write in R and Pharo - domain-specific tools for statistical analysis and interactive visualization - and I will need a solution for making these easy to run including dependencies. I see nix as the solution here but I am working out the exact details.
For one simple example, I have an R program called timeliner that reads a Snabb "shm" folder and produces statistics and visualizations. The basic source code for this program looks like:
#!/usr/bin/env Rscript
... R code ...and the issue is that it only actually runs if you have a suitable version of R installed and also suitable versions of all the R libraries that my code depends on. This is potentially a problem because I really have no idea which Linux distros (let alone OSX) satisfy that criteria. So a simple solution is to make a timeliner-nix wrapper script that uses the nix-shell shebang trick so that nix provides the necessary dependencies. This is a simple two-liner:
#!/usr/bin/env nix-shell
#!nix-shell -i Rscript -p 'with pkgs; with rPackages; [ R dplyr ggplot2 bit64 ]'
... R code ...... and so if you are in doubt about the dependencies you can just install nix and run the timeliner-nix wrapper to automatically get the same versions that I am using. If you are sure you have the dependencies right then you can run timeliner directly instead and you don't need nix.
That's not as convenient as deploying Snabb - one binary with no dependencies - but seems like a reasonable compromise for optional development tools.
teknico
TL;DR: at Igalia we'd like to also use another language for writing tests, probably Python.
Current status
In addition to unit tests in Lua code, integration tests are mostly written using Bash shell scripts.
In the
nextbranch there are currently 100 shell scripts, totaling around 3100 lines. In thesrc/program/lwaftrdirectory of our fork alone there are currently 22 shell scripts, totaling around 1200 lines. Most of those scripts are used for tests, called bymake testvia 18selftest.shfiles (stats courtesy of loccount).Our situation
While we keep adding and enhancing integration tests, our dissatisfaction with Bash shell scripts grows. The tests now include sizable amounts of logic, and we frequently find ourselves fighting the various peculiarities and limitations of the language.
We are at a point where we feel it would be more productive to start using a more consistent tool for this job.
Alternatives
After evaluating the alternatives, we are considering two candidates: Lua and Python.
Lua would be the natural choice, being the main language of the project. However its features for handling subprocesses are not very good, and that is obviously the main thing happening in integration tests. Also, libraries would have to be added for specific features we need, like JSON encoding/decoding.
Python is a strong candidate: it has robust subprocess handling, an extensive standard library and many of us know it well. Its syntax is slightly more verbose than shell scripts for running system commands, but that could be corrected by using a small, one-file library.
Feasibility
Our tests are currently run both manually and automatically, via
snabb-botand Hydra. As far as I know there is no formal list of test dependencies: the closest thing is the list in a Dockerfile.Ensuring the availability of another interpreter in all the environments tests are run should not be a problem.
Impact on the project
We are not proposing to establish any kind of policy for future tests, much less rewriting the current ones: we would like to have the option to also use another language in tests. The current structure would remain in place: the
selftest.shscripts would call other test scripts, whatever the language they are written in.Request for comment
Do you think it would be acceptable to have tests written in another language too? Would you benefit from this proposal? Would you like to suggest other languages? Which ones, and why?
Please comment. Thank you.