search

decaporg / decap-cms

A Git-based CMS for Static Site Generators
https://decapcms.org
MIT License
17.96k stars 3.05k forks source link

Tests: Add more complex e2e tests #3373

Open erezrokah opened 4 years ago

erezrokah commented 4 years ago

Context https://github.com/netlify/netlify-cms/issues/3332, https://github.com/netlify/netlify-cms/issues/3372 and https://github.com/netlify/netlify-cms/pull/3411

We've had several cases in the past when having out of dirty repos/out of sync forks will cause issues for editorial workflow and open authoring.

Editorial Workflow Examples

  1. Updating a PR that has an empty commit. We'll probably need to create an empty commit via GitHub API.
  2. Updating a PR that had been edited manually via GitHub, specifically added media files that are not mapped to fields.
  3. Update an existing entry in editorial workflow mode when that entry branch already exists.

Open Authoring Examples

Example scenario:

  1. Create a test repo with open authoring enabled.
  2. Login with a GitHub user that doesn't have access to the repo.
  3. Fork the repo via the CMS.
  4. Simulate an out of date fork by reverting a bunch of commits and force pushing an out of date default branch.
  5. Create an entry via the fork and move to Review.
  6. Verify the created PR.

Another example for https://github.com/netlify/netlify-cms/issues/3372:

  1. Create a test repo with open authoring enabled.
  2. Login with a GitHub user that doesn't have access to the repo.
  3. Fork the repo via the CMS.
  4. Simulate an out of date fork by reverting a bunch of commits and force pushing an out of date default branch.
  5. Reload the CMS and see that it offers the user to sync the fork.
  6. Sync the fork via the CMS.
  7. Validate success.

Another scenario (https://github.com/netlify/netlify-cms/issues/3506):

  1. Create a test repo with open authoring enabled.
  2. Login with a GitHub user that doesn't have access to the repo.
  3. Fork the repo via the CMS.
  4. Create a new entry.
  5. Move entry to In Review.
  6. Merge entry via GitHub API
  7. See that entry disappears for open authoring user and branch is deleted.
  8. Edit the same entry and repeat steps 5-7.

Repeat scenario for both merge commits, squash commits and rebase commits.

We should also test when the sync should fail (e.g. unresolved conflicts).

jperl commented 4 years ago

👋 Hello I maintain an open source library that generates Playwright code. I would be happy to submit PRs setting up e2e tests for netlify-cms, if you are cool with them using Playwright?

erezrokah commented 4 years ago

Hi @jperl, this sounds very cool. We already have a very extensive cypress setup. I don't think we would like to have two different types of testing frameworks so first we would need to replace all existing tests. Most of our tests use mocked recorded data and mock the browser fetch API, so most of our challenges were around that and not how to select items from the dom. Also, I was trying to figure out how the parallel testing works (we use Cypress parallel flag which load balances our tests and also provides a dashboard). Looks like https://github.com/qawolf/playwright-ci/blob/dd83f2d1a0bfa136c46f21a0f2c67b735439e8cb/static/github.hbs will run the tests on a single machine.

I probably missed some requirements, but please let me know what you think. As for the tests in this issue, most of them are about having "dirty" repos to test the CMS so the challenge there is not necessarily working with the dom, but finding a consistent way of generating the "dirty" status.

jperl commented 4 years ago

Thanks for the response :slightly_smiling_face:

I see you already have an extensive setup, so it probably would be too much work to port all of the existing tests. Re parallel testing: This can be done by using a GitHub Action matrix, where each job targets a subset of the tests.

I understand if you prefer to use Cypress for your tests, if you change your mind let me know!

erezrokah commented 4 years ago

This can be done by using a GitHub Action matrix, where each job targets a subset of the tests.

Thanks, we use a GitHub actions matrix, but Cypress takes care of splitting the tests between machines: https://github.com/netlify/netlify-cms/blob/8b0248bdc3f55dca451121d089b3274c8e72c09d/package.json#L28.

jperl commented 4 years ago

That is a great point! I added a ticket for creating a Jest scheduler to do this automatically.