alexAubin
commented
3 years ago Sooooooo it's somewhat starting to work and ready for feedback ... Still need to test all use cases on the battlefield
There's so many things I changed and I'm a bit tired to write an exhaustive list but here's a try ... :
- Ended up switching to LXD instead of vanilla LXC because I was fed up about all the "lol let's configure the network ourself with a bunch of iptable rules, this is so fun ! Also let's tweaks the rootfs manually all the time !". LXD is just way easier to work with (c.f. discussion below about having multiple package check in parallel for different test channels) ... Also this brings more consistency with ynh-dev and core ci, + possibly opens the possibly to install on non-Debian systems
- Cleaned the output to be more human-friendly and focus on the actual test logic
- Tried to reorganize everything such that it's less of a big clumsy, repetitive bash soup. Tests are now ~somewhat independent of themselves and with more work could really be ran as standalone-entities I guess (in terms of not-relying-on-7548-magic-global-variables-to-work). Or at least things should be slightly more readable ...
- Rewrote the end summary / level analysis using python
- Parsing check_process is a lot more simpler now ... I guess could be turned into python but meh
- Dropped notifications stuff
- Dropped a bunch of weird options that were apparently not that useful
- Dropped a bunch of weird mechanism about "false positive" that imho was not doing anything useful...
- App script progression is displayed in real time
- The configpanel / actions tests are probably heavily broken at this point ... not sure I want to fix those considering the corresponding code is hell + things are still likely to move soon + only a handful of apps actually implement them ... Considering disablign them
- About check_process config file, turns out there no point in providing lines for args
domain,path,is_publicandadmin... (just a random thing about the simplifying-check_process-format-some-day™ .. as discussed in an issue on the bugtracker, most stuff could be guessed from the manifest) - ...
About tests
To give you an idea, the test summary for wordpress looks like (forget about the actual OK/fail results, this is just to discuss the list of test being ran) :
Package linter: OK
Install (root): OK
Install (subdir): OK
Install (private): OK
Install (multi): fail
Backup/restore: OK
Upgrade: OK
Upgrade (00a1a6e7): fail
Change url: fail
Config/panel (actions): fail
Config/panel (config_p): fail
Install (subdir): OK In particular:
- no more tests about "removal", they are included in the install tests..
- no more tests about "public install" because all install tests are public by default (apart from private install ofc)
- upgrade tests are independent from each other
- less madness about multiple test series .. in fact only a handful of apps have multiple test series. In my example, the last "Install (subdir)" corresponds to an install from the alternative test serie (this should be made explicit in the display...). Now tests from extra test series are only considered for a) root/subdir/nourl installs, b) upgrades from commit != current. For example, no test about backup/restore or private install is performed in extra test series because it does feel worth in terms of cost vs. gain ...
About the configuration, LXD, CI perspectives
By default there's no config file anymore. Default values are defined are the top of lib/common.sh. Though you may still create a config file to override the default. In particular, one could imagine setting ARCH=arm and who knows ... if you rebuild the base image, it should pick up the ARM LXC and run everything with that arch (supposedly cross-architecture, meaning from an x64 machine). Absolutely not tested though...
Similarly, one could tweak the branch that's used for yunohost install and create a base image that's based on testing or unstable or next-debian... Theoretically, one could run multiple instances of package_check as long as LXC_BASE and LXC_NAME are named appropriately to not conflict with other stuff...
Possible other improvements
- [x] Fix display of other-test-serie tests ...
- [ ] Make sure Complete.log contains all the useful info for debug
- [x] Try running tests on a non-web app
- [ ] Having a
--single-test <some_test_name>option to run a single test instead of the entire test suite - [ ] Having a
--self-upgradeoption to interface with CI_package_check (replacement for the auto_upgrade script) ?
kay0u
So yeah I'm like at "only" 20hours of work on this and this is starting to look somewhat readable / evolvable ... I'm aiming to have a few separate pieces that can somewhat work independently from each other and that we can make evolve ... instead of a big pile of obscure global variables that magically change states in a completely incomprehensible, unnecessarily complex way. I'm just amazed that somehow the current code works so well...