QA Proposal: Test Scripts

btsfav commented 6 years ago

I'm in the process of creating a test script environment, which would allow us to test standard processes prior to a new live wallet release.

Plan is to create test scripts as we proceed in the wallet releases

Please give me layout feedback: https://docs.google.com/spreadsheets/d/1qtqUQlIkFRw5rLi8vSGITsuRSglH3RYXDZhjOE0I_gY/edit?usp=sharing

So basically add 1-2 days to a new wallet release for testing, anyone interested could copy the current test script sheet and do the testing.
Open a github issue under version number so that testers can submit the sheet link filtered to FAIL.
UI devs could browse through the logs and come back for feedback or try to reproduce

ThomasFreedman commented 6 years ago

Is your framework internel to the codebase or are you talking about some type of GUI poker to test the UI? The only feedback I can offer on what you posted for feedback is it doesn't yet convey the scope of what you're testing or specifically how those tests are executed. In other words, not enough info about the framework architecture you're implementing to run the tests.

btsfav commented 6 years ago

@ThomasFreedman test scripts are scripts that are written to test specific actions in software. ie if there is a new version, those whole set of tests are performed and ensure that everything is working as intended prior to release. it's a QA process

btsfav commented 6 years ago

I would write, maintain, perform and micro manage the feedback, if our UI devs want to implement a continuous improvement process

ThomasFreedman commented 6 years ago

@btsfav I have decades of software dev experience, so your reply provided no new info and didn't address any of my questions. I'm not saying that to be a dick, I am trying to understand your approach to testing.

There are tools to poke buttons on a GUI app, there's instrumentation and test frameworks that can be added to the code base, unit tests are such a testing framework.

However, if you're talking about an end-user test framework or approach, for example the POST (PowerOnSelfTest) code in older computer BIOSs, that's a totally different animal. I use POST diagnosics as an example b/c the end user can choose to run those or not long after it leaves the factory.

Is your aim is to provide tools the user can initiate to give a score about functional / operational status, or is it to help the devs produce code that is tested more thoroughly (increase % of codebase covered by unit tests) before it is even released?

Nothing I see here gives me a clue about your approach.

btsfav commented 6 years ago

@ThomasFreedman I think you lost me too. We're talking about UI here, not software dev. The focus is on user acceptance and bugs that may be implemented in the process of adding new features.

The goal is to create a test script (as mentioned) that needs to pass every test case prior to the release of a new version. It's to ensure nothing is broken and works as intended in a controlled and efficient manner

ThomasFreedman commented 6 years ago

Upon further review it looks like you are talking about a test plan or series of test plans to test the functionality of the UI. Is it simply a set of steps to perform with conventions for the outcome of the test. What the UI team does to verify their work.

You might consider an assertion model, as commonly used in unit tests. The only difference is you manually execute the steps by interacting with the GUI. There are automated tools to do that. The hard part of any testing approach is thinking through and then codifying a test algorithm for the piece of functionality you're trying to verify.

You only propose to test 4 areas (SETTINGS, LOGIN, VOTING, EXCHANGE), or were those only a starting point? Exchange is a huge body of functionality, doesn't that need to be broken down into specific functional tests? To what degree does such a testing target get decomposed?

btsfav commented 6 years ago

the test script is NOT final. It will grow with future versions, as it's extremely time consuming to write and cover all possible test cases in one go.

There is no chance of automating this, as we're dealing here with the end user, and they do stuff you can not predict. algorithm tests etc are for backend devs.

ThomasFreedman commented 6 years ago

Have you had a conversation with Bill Butler, SVK, J Calfe or other GUI devs about this?

btsfav commented 6 years ago

this is called proposal for a reason thomas :)

ThomasFreedman commented 6 years ago

Just trying to assess your level of understanding of how GUI apps are tested, what you know about state of the art. My knowledge is a bit dated, but there are definitely automation tools for testing GUIs.

Regarless whether the "algorithm" is manifested in a series of steps a human does or whether you write a test script for a tool that guides the poking of buttons and reads results from the screen, you still need to think thru HOW to test and that is where interpretation comes into play.

Before you can write a test to measure if something works you need to have a spec or doc that tells you how it should work. You can't write a test script without that.

wmbutler commented 6 years ago

@btsfav I was assuming you were going to use something like Cypress.io as a frontend test suite that could run with our automated builds. Is this where you were going?

btsfav commented 6 years ago

@wmbutler eventually yes, but I've never used tools for it. So I would start by defining test scripts and check them by hand.

describe('BitShares Register', function() { it('wallet bitshares org', function() { cy.visit('https://wallet.bitshares.org')

cy.get('a.button.primary').click()

}) })

cypress is definitely easy and fun to learn though, perhaps I could do this as well

wmbutler commented 6 years ago

I would suggest that you create an issue for each test and estimate your hours for each. Start with 1 test and let's see what the time commitment looks like.

btsfav commented 6 years ago

@wmbutler it's a cool tool and I played around with it in the last hours, however, here are the downsides I see:

Naming schemes in the wallet are confusing, ie. there are billions of buttons with no definite name = cypress won't find a specific one easily
For a lot of actions you'll need a user name, password and an asset = you'd have to adjust those scripts
Design changes are breaking the scripts
100% CPU bug etc. would not be caught by this

I'd still very much prefer a human readable test script.

It would probably not take longer than 30-60 mins at max test complexity to perform, if that's available budget money.

btsfav commented 6 years ago

https://github.com/bitshares/bitshares-ui/issues/1207

wmbutler commented 6 years ago

Automated tests are the best way forward. Start small and add more complicated items later. I'd rather invest resources in tasks that bring automation. Automated tests do not replace human testing but they definitely help improve reliability. Regarding the account names, make that part of the test:

Test 1: Create account name: frontend-test-agent-00001 Test 2: login with account frontend-test-aget-00001 etc...

Your test engine can use a random number generator to create the test account each time on the testnet. Then that account can be used to perform all of the testing tasks required.

btsfav commented 6 years ago

@wmbutler you'd still need a loaded account for executing orders/trades etc. is there a way to setup a testnet that's automatically loading an account?

wmbutler commented 6 years ago

Why not let your test load up an account?

sschiessl-bcp commented 6 years ago

With the new release cycle automated and mandatory manual tests should be conducted that every release needs to pass.

What is the current stage of this issue?

wmbutler commented 6 years ago

We need someone who is dedicated to writing front-end tests. Haven't seen anyone who has shown interest yet.

bitshares / bitshares-ui

QA Proposal: Test Scripts #1203