Ok so now that I proved it can be done in #195, we face a hard discussion/implementation road for actual testing.
A high level of things I see that need to be tested. We'll start from easiest to hardest concept:
Unit Tests - Model Methods
Basic Odoo tests for the most part. Make sure the models do the correct things, and mock out any external resources.
This will suck from a manpower perspective & I have a feeling we'll flush out a lot of hidden bugs. Typical of any model testing after the fact really.
Unit Tests - Images
Test of the Docker images that Clouder uses, without using Clouder.
Does the image build?
Are the correct files on the final image?
Does the image provide the assets it is supposed to (entrypoint, binaries, etc)?
We can use DQT and the strategy in #195 for this. We'll likely make things more efficient as we go along too, but overall it's not bad & shouldn't take too long.
Integration Tests - Clouder Controller
This is the hard one. We need a way to run a service creation through Clouder. Ideal would be a full Odoo spin-up, but I think that might take too long and I'm not sure how much benefit it would offer assuming we've already tested the stuff above. I see the intent of these tests mainly as a verification that Node A will actually talk to Node B in the way we want & have tested.
Here's a strategy I think may work:
Create a Clouder Test Host Dockerfile
This will provide a node with SSH & Docker Engine (assuming/hoping that Docker engine will run in Docker, I've never tried before)
Clouder Test Host Dockerfile will be built & run in Travis using DQT, similar to the #195
Add Clouder demo data to match SSH config
IP will need to be manually added on runtime? Or maybe some DNS magix
Clouder instance will spin-up during normal Travis test cycle & load demo data
Integration tests will run against the instance configured via the demo data and run through Travis
And the real problem
The main reason I made this RFC though is we now how ourselves a real N+1 problem here. These docker images are insanely expensive to be building in a sequence like this. By the time everything is said and done, our tests will be hours long and Travis will kill them at 1.
We can alleviate some of this with the Travis matrix, which will allow us to build different things at different times such as in DQT. We would need to isolate dependency paths though in a sense, because the matrices operate independently from each other.
Interestingly enough though, while the images are actually dependent on each other, they wouldn't fully be in the testing environment. A FROM statement inside of the Dockerfile will still be pulling from the repository, and not internal cache, unless we match the tag names. This could be a double edged sword so we will probably need to figure out the tags.
Ok so now that I proved it can be done in #195, we face a hard discussion/implementation road for actual testing.
A high level of things I see that need to be tested. We'll start from easiest to hardest concept:
Unit Tests - Model Methods
Basic Odoo tests for the most part. Make sure the models do the correct things, and mock out any external resources.
This will suck from a manpower perspective & I have a feeling we'll flush out a lot of hidden bugs. Typical of any model testing after the fact really.
Unit Tests - Images
Test of the Docker images that Clouder uses, without using Clouder.
We can use DQT and the strategy in #195 for this. We'll likely make things more efficient as we go along too, but overall it's not bad & shouldn't take too long.
Integration Tests - Clouder Controller
This is the hard one. We need a way to run a service creation through Clouder. Ideal would be a full Odoo spin-up, but I think that might take too long and I'm not sure how much benefit it would offer assuming we've already tested the stuff above. I see the intent of these tests mainly as a verification that Node A will actually talk to Node B in the way we want & have tested.
Here's a strategy I think may work:
And the real problem
The main reason I made this RFC though is we now how ourselves a real N+1 problem here. These docker images are insanely expensive to be building in a sequence like this. By the time everything is said and done, our tests will be hours long and Travis will kill them at 1.
We can alleviate some of this with the Travis matrix, which will allow us to build different things at different times such as in DQT. We would need to isolate dependency paths though in a sense, because the matrices operate independently from each other.
Interestingly enough though, while the images are actually dependent on each other, they wouldn't fully be in the testing environment. A
FROMstatement inside of the Dockerfile will still be pulling from the repository, and not internal cache, unless we match the tag names. This could be a double edged sword so we will probably need to figure out the tags.