e2e tests

[fvictorio]

I experimented a little with using Puppeteer to create an end-to-end test for the ICO Wizard. The result is here:

https://github.com/fvictorio/ico-wizard-e2e

Just cloning it, doing npm install and npm start should work. It will run a single test that creates a simple crowdsale. By default it uses the local node, so before running it you should go to the ico-wizard repository and execute npm run ganache and (in another terminal) npm run deployRegistry.

Doing this kind of tests is hard because there is a browser extension involved. In fact, I couldn't find a single tutorial for doing e2e tests for dapps with metamask. But since we spend a lot of time manually testing that everything is working, this may be useful.

Let me know what you think.

[fvictorio]

@igorbarinov Some thoughts about this:

• You mentioned wanting to know when the transaction was confirmed. My approach right now is to refresh the metamask page N times until a new transaction appears. If this happens is because the previous transaction was confirmed. If the N retries are done (say, 60 seconds), it stops trying. Obviously this works because the ICO Wizard makes the transactions sequentially. If your test really needs to know, this approach does not work. I will think more about this, but I believe the current approach is good enough for the basic tests we could do for the wizard.
• One problem with having to interact with a browser extension is that you cannot use headless chrome for the tests. This is a problem for CI, because (as far as I know) most CI servers don't have a windowed environment. Still, if we make some set of basic tests, having the possibility to run them manually before a PR is already a big help in my opinion.
• There are some limitations to the things that can be done in the metamask tab, because it's not the same as regular web page (opened with http). For example, puppeteer lets you wait until an element appears in the DOM, which is really useful. But you can't do that in the metamask page, so I had to resort to ugly things like using conservative timeouts, or keep refreshing until some element appears. Again, I think it's better than nothing.

For completeness, I should mention that another approach is possible: this article (the only thing I found about e2e tests for dapps) injects a custom web3 instance during the tests and uses that to interact with the dapp. I'm not sure if this approach allows, for example, to simulate a user rejecting a transaction.

tl;dr: I think my approach, while somewhat limiting, is good enough for writing some basic tests and save us some time doing them manually. Integrating them with a CI with a windowed environment is something that we should investigate.

How do you think we should proceed?

[dennis00010011b]

Do we really need a headless browser for testing the ICO wizard? With headless testing e2e script will work faster but how is it important ?
e2e script contains two parts:

1. Installation MetaMask and fill out crowdsales forms. It takes 10 second with Selenium and less than 1 second with Puppeteer.
2. Around 20 transactions. It takes more than 10 minutes in both cases.

Anyway time of e2e scrip execution too long and takes around 10 minutes. As a result there is no advantage to use Puppeteer.

I see two approaches to create e2e script.

1. Using Puppeteer:

   • advantages: faster because headless browser can be used;
   • disadvantages: scope limited by Chrome browser; problem with having to interact with a browser extension;

2. Using Selenium:

   • advantage: universal tool, can be used with many browsers(MetaMask can be installed for FireFox) ;
   • disadvantages: slower; In my opinion using browser testing with Selenium is better than headless with Puppeteer.

Both approach can be integrated with Circle CI.

[dennis00010011b]

I have created end-to-end scrypt for the Token wizard .

https://www.npmjs.com/package/create-poa-crowdsale https://www.youtube.com/watch?v=fcmSzWso1_U

Scrypt performs:

• open pre-installed Chrome browser
• install MetaMask Chrome extension
• activate MetaMask with given account number and private key (only Rinkeby network now). Data in *.json format.
• going through POA Wizard with given data in *.json format.
• save result's files in pathToOutputDirectory
• package includes the file ./circleci/config.yml for CI server: https://circleci.com/

[vbaranov]

@dennis00010011b

is it possible to add URL of Token Wizard to config, for example, to test local deployment before deploying to main link?

[dennis00010011b]

@vbaranov did you mean a kind of breakpoint? Good idea. I'll do it asap

[vbaranov]

@dennis00010011b Breakpoints are good to have too. But I meant that utility uses this link https://wizard.poa.network now. But, for example, I need to test new patch for wizard locally before merging to the main link. I would like to change endpoint in the configuration file and start the utility for another link. Maybe, to add an endpoint to crowdsale.json or to separate config file. If I am not wrong, with the current version of the utility, I need to change the link in many places to connect to wizard on another link.

[dennis00010011b]

Updated. Now startURL can be added in config.json Also , installation of MetaMask can be skipped

{
  "startURL" : "https://wizard.poa.network/",
  "installMetaMask":true
}

[igorbarinov]

@dennis00010011b could we transfer the repo to poanetwork github org?

[dennis00010011b]

@igorbarinov Done https://github.com/poanetwork/e2e-test-token-wizard

[vbaranov]

@dennis00010011b link to the repo in the readme should be fixed then

[dennis00010011b]

@fvictorio @vbaranov I need your opinion about e2e scrypt. How we can use it ? How we can hook up scrypt to CI and launch it with each pull request? Which CI is better to use: Circle,Travis, anything else? Any thoughts?

[vbaranov]

@dennis00010011b We use Travis CI for Token Wizard, which includes Mocha unit tests for Solidity contracts. My suggestion is to provide binary for e2e utility: https://github.com/poanetwork/e2e-test-token-wizard/issues/6. Thus, we will be able to add launching of it to npm test script of Token wizard. And, maybe, organize e2e tests with using of Mocha too to unify results of Solidity and e2e tests.

[fvictorio]

Ideally we would have a way to automatically run the e2e tests in the deploy previews that Netlify creates, but I don't know how to do that.

The next best thing is for Travis to:

1. Create a build of the app and serve it
2. Start ganache and deploy the registry contract
3. Run the e2e tests
4. Stop the local server and ganache

Using ganache is optional, but it would mean faster executions. This is more doable, but, as Victor said, we would need to be able to add the e2e repo as a dependency, submodule or just moving the code to this repo.