search

owncloud / QA

:collision: public test plans for owncloud components and apps
GNU Affero General Public License v3.0
14 stars 17 forks source link

make videos of UI tests available to developers #612

Open individual-it opened 5 years ago

individual-it commented 5 years ago

When developing UI tests or investigating bugs (in tests or app) its sometimes very helpful to see what the browser is doing, specially when the test works locally but fails in CI. We lose a lot of time just investigation those issues by guessing what the problem might be.

Highest priority is to have video recordings of failed tests, having all test runs recorded would be nice but not very important. Also to save storage the recordings could only be made when a specific switch is set in the .drone.yml. When the developer faces an issue, she could set that switch and get the recordings for analysis.

Some work was done by @patrickjahns on this topic: https://github.com/patrickjahns/vncrecordingserver

alternative we could use the recording feature of https://opensource.zalando.com/zalenium/#usage

CC @patrickjahns @phil-davis @PVince81

PVince81 commented 5 years ago

How much effort is needed for the video of failed tests ?

patrickjahns commented 5 years ago

@PVince81

How much effort is needed for the video of failed tests ?

From my initial research this is not an easy task to get done, as it involves figuring out aspects of: 1) get the recording done 2) have a storage location where recordings could be stored ( for a certain period of time ) 3) manage to upload the recording from the ci built 4) provide a url where a developer could possibly fetch these

Back then Zalenium didn't provide any possibility to trigger uploads from itself and was designed to be a central service that provisions browser containers by itself ( i.e. via kubernetes or similar ). This is different to how we currently utilize drone - it doesn't require a central service that manages other containers - it starts the containers by itself.

As @individual-it mentioned - I did some preliminary research how to built something relevant for drone and there is some prototype to connect to a selenium container ( and start/stop a recording ) - however this has some technical issues when a recording is too short. ( this seems to be a issue in the upstream vnc->mp4 converter ).

Short discussion with @individual-it showed also, that the zalenium project has evolved and we might utilize their upload capabilities, when we provide a service/server that implements a zalenium specific protocol.

All in all, the effort is not easy to be estimated in PD - it both contains research tasks and experimental tasks where we might need to overcome quite a few obstacles

patrickjahns commented 5 years ago

Forgot to mention, I also prototyped https://github.com/owncloud-ci/recorder

This can be used straight away when you run the tests locally. @individual-it so the question is, do we need to have recording in the ci system - or is it enough if developers can run the same tests via drone exec - but they provide a parameter like RECORD=true drone exec

individual-it commented 5 years ago

@patrickjahns the real problem is when running tests in CI and stuff fails for unknown reason. When running them with local drone exec I can simply find the IP of the docker container and VNC into it to see what happens

patrickjahns commented 5 years ago

Instead of zalenium - we could have a look at the underlying image https://github.com/elgalu/docker-selenium

also see: https://github.com/elgalu/docker-selenium/blob/master/docs/videos.md