Closed jgaehring closed 2 years ago
Phantom.js is a headless browser for integration testing that @pcambra recommended: https://phantomjs.org/.
Also, here's the repo that @kirsten-mc has started for storing the demo data: https://github.com/kirsten-mc/farm_demo.
FWIW Drupal core uses something called Nightwatch: https://www.drupal.org/docs/testing/javascript-testing-using-nightwatch
I dug up the original thread in Drupal core when they decided to go with Nightwatch, in case we want to review the thought process that went into that decision: https://www.drupal.org/project/drupal/issues/2869825
Some info from the Vue site as well: https://vuejs.org/v2/guide/testing.html#End-to-End-E2E-Testing
Nightwatch is among their recommended frameworks, but they also have a nice section on what things to consider when choosing your framework.
I'm debating how much of this I want to do for the 0.7.0 milestone. I guess I had originally had in mind the kind of integration testing where I just test if a network call or database call works, but I think @mstenta had more End-to-End testing where we test a specific workflow.
I think one question we have to ask ourselves is, what do we want to have under test specifically with the Logs API? And what can we test if we just restrict ourselves to that API? Particularly if we're talking about an aspect of Field Kit Core, and not of a particular module like the My Logs module, I think we're restricted to the first kind of integration test I mentioned above, not E2E testing. That may still require some heavy automation, like spinning up a server in a docker container and simulating a browser (in particular browser API's like IDB, fetch, etc), but I don't think we necessarily need to be testing things like what happens when a particular button is pushed. That latter type of E2E testing might need to wait til 0.8.0 or even 0.9.0 when we have clearly defined what modules will part of the standard set of modules that come with Field Kit. And then those tests should probably also live with those modules (still open for debate if that should be in this repo). If we've established that we can only really do integration and not E2E tests at this point, then what are the costs/benefits of such integration tests, and what resources do we want to invest in that at this time?
I'm wrapping up my work on the Logs API this week, with the main syncing and caching API work being completed this morning, and hopefully the last work on the new farmLog
implementation being completed by tomorrow. So I'll probably circle back around to this issue of testing some time next week. @mstenta, do you have any particular thoughts on this before you start vacation?
Pasting what I said to @jgaehring in chat for posterity... :-)
I think the most valuable tests will be the end to end ones And we're not ready to set that up until the farm_demo module comes together But if you think there are unit tests that would be useful to test the functions you're developing, that's great too!
Moving this to 0.9.0 milestone, which seems like the soonest this can happen.
Closing this, because I don't think we ever came to any well-defined requirements for the kind of tests would be effective, while also achievable in the near-term. I still think there will be a lot of value in E2E testing that can run in a single container along with a server instance that also serves the field modules and provides a pre-populated testing database. But I think we're still a long ways off from that capability at the moment.
I'm happy to reopen if/when that changes, but for now, I'm removing it from milestones, while also adding a new "tests" label so we can find this thread again later.
As we begin work on syncing and caching logs in milestone 0.7.0, with the intention of doing the same for assets and areas in later milestones, it seems like a good time to finally put together a toolchain for integration tests.
This could work in conjunction with the proposal to create a demo farmOS server, particularly when it comes to creating, seeding and updating the data. The data we use for integration testing might be a good way to start the demo data we would need for that.