add regression testing and CI

autonome commented 2 years ago

For production readiness for when Metamask flask architecture goes mainline, we want to ensure we are proactively notified when upstream MM changes to Flask architecture break the FIL snap.

set of basic automated functional tests
running daily in CI (or at a cadence recommended by MM team)
broken tests files new issue

Perhaps Glif and/or MetaMask folks already have testing frameworks to use and maybe even CI we can inject ourselves into (change the flow - their "build" breaks when FIL snap breaks).

cc @rekmarks @Schwartz10

Redoudou commented 2 years ago

@BeroBurny following our synch, PL would like ChainSafe to write down a proposal / plan to build those automated test for FilSnap. @autonome recommend us to get in touch with Lidel and Hugo Diaz as we could reuse the code from IPFS Companion automated testing.

lidel commented 2 years ago

quick brain dump, hopefully some pointers will be useful / save time:

afaik there is no real (end-to-end) test automation for WebExtensions, but you can run functional tests in WebExtension context
- Firefox XUL-based extensions had a nice tests runner with hooks that allowed for doing asserts about things like address bar, buttons, windows, and interact with all of it.
- Unfortunately, browsers (including Firefox) moved to Manifest V2 created by Google, and we no longer have these capabilities.
- ipfs-companion only has some basic unit tests for request redirect logic, there was no way to do browser chrome asserts at the time.
@hugomrdias created playwright-test runner which has --extension parameter for running tests in a real WebExtension context (PR).
- This is better than nothing, depending on the functionality you want to test, could be enough.
- Some things remain impossible to automate since there are no hooks for end-to-end testing any part of native UI such as "is chrome.browserAction added to the ui?" etc.

mpetrunic commented 2 years ago

hey @lidel @autonome, sorry it took me a while to write a response here. We discussed this with Metamask already. We are maintaining https://github.com/chainsafe/dappeteer which uses puppeteer to install metamask into chrome/firefox and controls it. We could use it to write e2e tests for snaps right now, but we are delaying that as we are discussing integration directly into snaps cli, making it easier and faster to write those tests.

Let us know if you would like us to wait a bit more to figure out if it's going to be integrated or to do it as is (in which case we should refactor if it gets integrated)?

mpetrunic commented 1 year ago

This is finally possible with dappeteer: https://github.com/ChainSafe/dappeteer#usage-with-snaps

ChainSafe / filsnap

add regression testing and CI #132