search

DSpace / dspace-angular

DSpace User Interface built on Angular.io
https://wiki.lyrasis.org/display/DSDOC8x/
BSD 3-Clause "New" or "Revised" License
131 stars 429 forks source link

E2e tests against the live rest demo are slowing down Travis CI jobs #453

Closed artlowel closed 5 years ago

artlowel commented 5 years ago

Our e2e tests run against the demo rest api, and that isn't good practice. The data could change and unexpectedly break tests.

Despite of this it was working well enough up until recently. However in the last few weeks it seems Travis's connection to that demo server is slower than before, and it times out frequently on branches that didn't have any issues. I was never able to reproduce these timeouts locally, they happen only on Travis. I was able to work around this issue by increasing the timeouts to 10 minutes (the default was 30 seconds). But this of course means that we slow down all Travis jobs.

And we're not really using the e2e tests at the moment. We have only 3 e2e-spec files, the last of which was created in October 2017. And the contents of them is usually trivial. So I propose we no longer include them in Travis jobs for now. I'll create a PR to disable them.

Afterwards, if we decide we want to really want to use e2e tests and write more of them, we should first figure out a way to run them independently of any other server, either by mocking all rest interactions, or by spinning up a DSpace 7 rest server in docker, with predictable test data, and a specified commit id. So I would leave this ticket open until we do.

tdonohue commented 5 years ago

As discussed in today's DSpace 7 meeting, the current (minimal) end to end (e2e) tests have been temporarily disabled until a better solution can be found. https://wiki.duraspace.org/display/DSPACE/2019-08-08+DSpace+7+Working+Group+Meeting

This ticket will remain open as we look for a way to run e2e tests that does not depend on the demo REST API instance.

One potential solution that could use further investigation is determining whether Travis CI could use our Docker compose scripts (https://github.com/DSpace-Labs/DSpace-Docker-Images/) to spin up a backend for e2e testing.

Some resources related to Travis CI + Docker Compose:

This is looking for volunteers

terrywbrady commented 5 years ago

I hope to have some time to work on this. In case I do not, I wanted to share some thoughts here for discussion.

I presume that this process will run against a published version of dspace/dspace and will default to dspace/dspace:dspace-7_x-jdk8.

The docker-compose files could be pulled from DSpace-Labs/DSpace-Docker-Images or a clone of the compose files could reside in the dspace-angular repo as a testing resource.

Since speed is important, I presume that test data will be pre-populated using a SQL dump + and assetstore tar file. By default, I presume that we would start with the assets provided with the preview release. The database dump will be loaded as the database container is created.

Should the SQL dump and the assetstore files be downloaded each time that the tests are run?

Besides discovery, what optional services will be invoked by the end to end tests? Will the OAI service be needed? Will an authority index be needed?

Will the tests modify the contents of the volumes as the process runs? This may not matter, but it would be interesting to determine if the volumes could be cached.

tdonohue commented 5 years ago

@terrywbrady : Overall, I'd suggest we start simple in our approach, and we could build on that as needed in the future.

A cached SQL dump & assestore might work. Another option is to honestly just run a series of REST calls to create the necessary content prior to tests -- that may take slightly longer, but would avoid any need for storing a SQL dump or assestore. That said, on this topic I don't have a strong opinion, and am willing to go in whatever direction others feel is easier.

On the backend, Discovery is definitely needed. OAI and Authority could be optional initially. We might eventually have tests that exercise them, but not initially. So, they could be added later.

Tests would modify content (as we'd be testing modifications, submissions, etc). There would be some read-only e2e tests, but we'd also have tests to exercise admin functionality, submitter functionality, etc.

tdonohue commented 5 years ago

Closing, as this was fixed by #466