search

ClassicPress / ClassicPress-v1

A copy of ClassicPress v1.x.
1 stars 1 forks source link

Strategy for quality assurance of ClassicPress core code #230

Closed scottybo closed 6 years ago

scottybo commented 6 years ago

@warrior79 As discussed in the committee, if you have a chance to create these issues over the next few days that would be brilliant!

ruiguerreiro79 commented 6 years ago

sure will do.

nylen commented 6 years ago

I think the next step here is a PR rather than more new issues...

nylen commented 6 years ago

I created https://github.com/ClassicPress/ClassicPress-e2e for these tests, let's get any code or issues going there (preferably some initial code that we can look at)

nylen commented 6 years ago

I'm not sure I understand this issue though, please feel free to re-open if it's still needed.

ruiguerreiro79 commented 6 years ago

The Goal of this GHI( Github issue ) is to brainstorm about what can be done to improve the QA (Quality assurance) of the code that is developed in ClassicPress core.

The Quality assurance is extremely vital because a good programmer isn't always a good tester.

There are several levels of testing but it requires a lot of human resources to implement all the levels.

At the moment there are the Travis CI unit testing that run when a PR(Pull request) is made in the ClassicPress core.

I'm suggesting here the creation of Acceptance tests that would run in an automated way. This doesn't discard the human testing but it can save a lot of time if we can find some bugs using the automated testing.

The definition (wikipedia) of an Acceptance test in software development is this one: "Formal testing with respect to user needs, requirements, and business processes conducted to determine whether a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system." Also called has end-user testing.

To achieve the automated acceptance tests we can use the Selenium Web Driver that enable us to control several browsers and mimic the user behaviour and validate if the software is behaving as we expect.

Any other thoughts? Ideas? Suggestions?

nylen commented 6 years ago

Sounds fine to me. You have access to the ClassicPress-e2e repo so that we can get started on this.

ruiguerreiro79 commented 6 years ago

My previous interactions with automated testing were done using Codeception but I was thinking of using tools like Katalon Studio that is very powerful and it has a large community behind.

It's free, it has Git connection to the repos and it's a more user friendly environment that testers can also participate in the creation of the test scenarios.

@nylen what is your opinion on that?

nylen commented 6 years ago

I have not tried either of these tools - when I need to automate a browser, I use puppeteer.

I think now is a great time to explore different options here and see what will work best for us going forward.

Mte90 commented 6 years ago

I use Codeception with wo-browser because I can use an unique solution for all the tests and be compatible with phpunit.

Anyway do new tests is not so easy for a test suite like the wordpress one with over 8000 tests...

ruiguerreiro79 commented 6 years ago

I never used puppeteer but from what I read it's seems that only works with Chrome/Chromium while Codeception or Katalon use the Selenium Webdriver that is already part of W3C https://www.w3.org/TR/webdriver1/ and can work with the major part of the browsers.

nylen commented 6 years ago

Our QA strategy should look something like this:

@ruiguerreiro79 let's close this, as most of the work to be done is elsewhere. If you or anyone else has anything to add, though, we can still discuss here or in Slack.

ruiguerreiro79 commented 6 years ago

Sure, let's follow up there.

nylen commented 6 years ago

Oh, and coming back to the discussion about browser automation tools:

For the migration plugin we can just use puppeteer. Most of the complexity there is what is going on with the server, and there should be no difference across browsers.

For ClassicPress core, we probably want to use something like selenium so that we can catch bugs that only occur in some browsers. We need to define our test cases better, though. This can proceed over on the ClassicPress-e2e repo.