search

farmOS / field-kit

A modular, offline-first companion app to farmOS.
https://farmOS.org
GNU General Public License v3.0
60 stars 39 forks source link

Add integration testing #386

Closed jgaehring closed 2 years ago

jgaehring commented 4 years ago

As we begin work on syncing and caching logs in milestone 0.7.0, with the intention of doing the same for assets and areas in later milestones, it seems like a good time to finally put together a toolchain for integration tests.

This could work in conjunction with the proposal to create a demo farmOS server, particularly when it comes to creating, seeding and updating the data. The data we use for integration testing might be a good way to start the demo data we would need for that.

jgaehring commented 4 years ago

Phantom.js is a headless browser for integration testing that @pcambra recommended: https://phantomjs.org/.

Also, here's the repo that @kirsten-mc has started for storing the demo data: https://github.com/kirsten-mc/farm_demo.

mstenta commented 4 years ago

FWIW Drupal core uses something called Nightwatch: https://www.drupal.org/docs/testing/javascript-testing-using-nightwatch

mstenta commented 4 years ago

I dug up the original thread in Drupal core when they decided to go with Nightwatch, in case we want to review the thought process that went into that decision: https://www.drupal.org/project/drupal/issues/2869825

jgaehring commented 4 years ago

Some info from the Vue site as well: https://vuejs.org/v2/guide/testing.html#End-to-End-E2E-Testing

Nightwatch is among their recommended frameworks, but they also have a nice section on what things to consider when choosing your framework.

jgaehring commented 4 years ago

I'm debating how much of this I want to do for the 0.7.0 milestone. I guess I had originally had in mind the kind of integration testing where I just test if a network call or database call works, but I think @mstenta had more End-to-End testing where we test a specific workflow.

I think one question we have to ask ourselves is, what do we want to have under test specifically with the Logs API? And what can we test if we just restrict ourselves to that API? Particularly if we're talking about an aspect of Field Kit Core, and not of a particular module like the My Logs module, I think we're restricted to the first kind of integration test I mentioned above, not E2E testing. That may still require some heavy automation, like spinning up a server in a docker container and simulating a browser (in particular browser API's like IDB, fetch, etc), but I don't think we necessarily need to be testing things like what happens when a particular button is pushed. That latter type of E2E testing might need to wait til 0.8.0 or even 0.9.0 when we have clearly defined what modules will part of the standard set of modules that come with Field Kit. And then those tests should probably also live with those modules (still open for debate if that should be in this repo). If we've established that we can only really do integration and not E2E tests at this point, then what are the costs/benefits of such integration tests, and what resources do we want to invest in that at this time?

I'm wrapping up my work on the Logs API this week, with the main syncing and caching API work being completed this morning, and hopefully the last work on the new farmLog implementation being completed by tomorrow. So I'll probably circle back around to this issue of testing some time next week. @mstenta, do you have any particular thoughts on this before you start vacation?

mstenta commented 4 years ago

Pasting what I said to @jgaehring in chat for posterity... :-)

I think the most valuable tests will be the end to end ones And we're not ready to set that up until the farm_demo module comes together But if you think there are unit tests that would be useful to test the functions you're developing, that's great too!

jgaehring commented 3 years ago

Moving this to 0.9.0 milestone, which seems like the soonest this can happen.

jgaehring commented 2 years ago

Closing this, because I don't think we ever came to any well-defined requirements for the kind of tests would be effective, while also achievable in the near-term. I still think there will be a lot of value in E2E testing that can run in a single container along with a server instance that also serves the field modules and provides a pre-populated testing database. But I think we're still a long ways off from that capability at the moment.

I'm happy to reopen if/when that changes, but for now, I'm removing it from milestones, while also adding a new "tests" label so we can find this thread again later.