meijer
commented
2 years ago Brought to board for acceptance vote, board acceptance decision reached 3 August 2021.
Closed monkeyiq closed 2 years ago
meijer
commented
2 years ago Brought to board for acceptance vote, board acceptance decision reached 3 August 2021.
There is one pending task for this milestone: to move the Savage process to a virtual machine on the Internet. This VM is kindly being provided by aarnet and I will move the process to that VM from my desktop machine once it is available. In the meantime I may run the Savage task on my desktop machine.
I have brought up my local selenium test vm to run things against and updated the tests to all run again and updated the test suite itself to be more robust against subtle timing issues which were causing a false failure to be reported in the suite run.
https://github.com/filesender/filesender/pull/1016 https://github.com/filesender/filesender/pull/1004 https://github.com/filesender/filesender/pull/1005 https://github.com/filesender/filesender/pull/1006 https://github.com/filesender/filesender/pull/1007
The tests are also a little more resilient to recent menu changes, for example: https://github.com/filesender/filesender/pull/1003
I also updated the main CI github actions job file to be able to work on the main repository and on the filesenderuici repository used by savage. The later repository selects the jobs to run only the UI tests with the sauce tunnel.
https://github.com/filesender/filesender/pull/1017 https://github.com/filesender/filesender/pull/1018
There were are series of tests of the web hooks that start savage, for example: https://github.com/filesender/filesender/pull/1023
Rather than allow savage or the filesenderuici user to make comments on the main repository I setup a github action to create a comment on PRs in the main repository with a link to the filesenderuici repository where selenium test results would be if the run was executed there. Not allowing such write access helps to mitigate issues if the savage/filesenderuici is somehow compromised. https://github.com/filesender/filesender/pull/1027
This resulted in a test of the linking in https://github.com/filesender/filesender/pull/1031
I may disable (or greatly extend the time frame for) the branch deletion in savage to ensure that test results stay around indefinitely on the filesenderuici repository.
Although it was not in the milestone itself, I played around for a while trying to run selenium in docker through github actions. This is recorded in PR 1002. https://github.com/filesender/filesender/pull/1002
The advantage of the setup in 1002 is that there would be no secret sauce labs key to protect as the test would run entirely in the same github actions virtual environment. The downside is that sauce labs have a nice web interface and none of that would be available from the base selenium docker vm in github actions. The closest I found was being able to perhaps capture video files and save them as artifacts in github actions. That would result in a huge amount of data on github actions which might not be too welcome there.
Longer term I think perhaps having both selenium in docker and the savage run sauce labs testing (using the same suite) would be handy. The docker run could always execute and would tell if none of the tests failed with the proposed change in the PR. The savage run could be used to investigate failed tests with nice video and interaction history. Though the savage run might not execute all the time because of 'risky' changes (relative to leaking the sauce labs secret) that were made.
This is one of the tests for the setup. The main PR on the primary filesender github repo https://github.com/filesender/filesender/pull/1044
which has a link to the push on the filesenderuici repo which itself links to the github actions on there with the selenium build https://github.com/filesenderuici/filesender/actions/runs/859064989
more explicitly https://github.com/filesenderuici/filesender/runs/2626423153?check_suite_focus=true