feat: allow to run e2e tests against existing server

tobiasdiez commented 8 months ago

Allow to pass the url of an already running server to setup and use this as the target instead of building and running a new server. This has two use cases:

Run tests against a deployment of your application (i.e. "real" end-to-end tests)
During local development, run tests against a already running dev server. This reduces the time to run tests and was wished for in eg https://github.com/nuxt/test-utils/issues/314#issuecomment-1983129269.

Usage:

setup({ endpoint: process.env.TEST_URL })

(or of course just using a hard-coded url)

tobiasdiez commented 7 months ago

@danielroe friendly ping. Is there anything a can improve in this PR?

danielroe commented 7 months ago

I love this and would like to get this or something like it merged, but just want to make sure we do it in the best way. Will review soon.

danielroe commented 7 months ago

@tobiasdiez Is the only use-case here avoiding restarting/rebuilding the server when running tests? (ie. if we can solve that issue, do you still think this is a needed feature?)

tobiasdiez commented 7 months ago

My use cases are:

Run the same tests against a deployed instance. So for each PR, I build the app, deploy it to an temporary sandbox that is close to the actual production deployment (in my case, Azure Functions). Then I would like to run all the e2e tests against that deployed instance to check that the PR is not breaking anything. This setup is essential for me since every other nuxt update (or to a minor extend other dependency changes) breaks the deployment in some way while passing all tests locally (i.e. the issues are specific to azure). I'm currently using supertest for this (see eg https://github.com/JabRef/JabRefOnline/blob/main/server/e2e.test.ts and https://github.com/JabRef/JabRefOnline/blob/main/.github/workflows/deploy-preview.yml), but it would be nice to migrate to test-utils.
And indeed, if you already run nuxi dev to start a dev server, make some changes, and then want to verify that your changes are not breaking anything, it would be nice if the already running dev server is used as the endpoint for the tests instead of spinning up a new dev server.

Personally, the first use case is actually more important for me.

adamdehaven commented 5 months ago

This PR seems ideal for my use-case as well. We build for the nitro preset cloudflare-module which doesn't seem to be compatible out of the box with @nuxt/test-utils, meaning I have to override the preset when running tests to node-server.

Being able to do a build separately, running the build via wrangler, and then pointing the tests to the URL speeds things up exponentially.

The only issue I encounter is that it cannot seem to fetch the manifest:

passing test manifest error

@danielroe I'd be open to other ways of speeding up/reusing a build, but this PR seems promising other than the manifest error above

danielroe commented 5 months ago

I think we'll go ahead with this approach. Would you be able to add a test? 🙏

adamdehaven commented 4 months ago

This is awesome and will be super useful for us.

Let me know if the Nuxt docs page needs to be updated and I’m happy to write something up and submit a PR

tobiasdiez commented 4 months ago

@danielroe Thanks for merging. I'll try to add tests when I find the time.

@adamdehaven Would be awesome if you could create a PR adding this option to the docs.

adamdehaven commented 4 months ago

PR to add the new host option, along with a usage example to the Nuxt docs: https://github.com/nuxt/nuxt/pull/28331

Barbapapazes commented 3 months ago

This is so cool! Thank you so much, I will give it a try asap.

nuxt / test-utils

feat: allow to run e2e tests against existing server #803