search

juliangruber / test-npm-dependants

Run the test suites of all modules depending on a given module.
https://cto.ai/blog/ops-by-example-test-npm-dependants/
Other
21 stars 2 forks source link

[Discussion] Future Ideas #5

Open wesleytodd opened 4 years ago

wesleytodd commented 4 years ago

Awesome work @juliangruber! I have been looking at this space recently, so thought I would brain dump a bit and see if you liked any of these thoughts. Sorry for the unstructured issue, but I didn't want to create too much noise if some of this would not move forward.

1. Sandboxing

So I see that you are using this Op platform for some form of sandboxing, but for folks looking to do this in other environments, it would be nice to abstract this away so that folks could provide their own sandbox. An example of this, a docker container would be a great simple sandbox (and allow for simple testing of my next point). If there was an abstraction layer about what an execution looks like, we could use the orchestration layer in this module but run the actual un-trusted code remotely.

2. Node versions

For node, there is CITGIM, which tests node versions against many popular packages. We talked a bit about the matrix testing we really want, which is a matrix of this module version X dependant module version X this module dependency version X node version. Obviously this is huge matrix, so I will talk about a point on this next, but having a few more knobs would be awesome to test a more detailed matrix.

3. The test matrix

As I mentioned above, the matrix can get huge. One of the things we encountered at Netflix was that the larger number of dependents and matrix variants, the larger the load and longer the tests take. To improve this we can do a few things:

  1. Have a list of modules to test
  2. Only pick a random selection
  3. Add a learning algo to test the most likely to fail
  4. Have a staged approach (depending on the change and the SDLC stage, choose a different approach)

4. CI environments

I see you have #2, so this is probably on your mind, but I think this is a very important consideration. If you take a simple OSS setup on GitHub Actions, you probably want to leverage their matrix strategy. So having some story for this type of testing in an environment like that would be awesome. Again, I think this needs to be a bit abstracted so that folks with different CI systems can also use it.

5. Multi-Registry Support/Fetching Source

Looks like the package fetch-package-source does not really have hooks for alternate source fetching. It seems like just making this an option you can inject would solve for folks needing to fetch from different sources.

I wanted to get the conversation started, and I think I missed some other points we brought up in the Package Maintenance Working Group, but hopefully this helps. Also, If you are open to it I an on board with working on some of these ideas here in this repo, so let me know if you are open to PRs.

juliangruber commented 4 years ago

1. Sandboxing

So I see that you are using this Op platform for some form of sandboxing, but for folks looking to do this in other environments, it would be nice to abstract this away so that folks could provide their own sandbox. An example of this, a docker container would be a great simple sandbox (and allow for simple testing of my next point). If there was an abstraction layer about what an execution looks like, we could use the orchestration layer in this module but run the actual un-trusted code remotely.

Ops actually uses Docker for sandboxing, it was the quickest way to get started. I guess we could also implement a manual Docker option?

2. Node versions

For node, there is CITGIM, which tests node versions against many popular packages. We talked a bit about the matrix testing we really want, which is a matrix of this module version X dependant module version X this module dependency version X node version. Obviously this is huge matrix, so I will talk about a point on this next, but having a few more knobs would be awesome to test a more detailed matrix.

This is actually where a manual Docker approach would come in handy, as with Ops (currently) one option can only use one Docker image. We could however use something like nvm to switch node versions at runtime, instead of using a Docker image each?

With this size a matrix a hosted solution with a worker queue and caching seems way more practical?

4. CI environments

I see you have #2, so this is probably on your mind, but I think this is a very important consideration. If you take a simple OSS setup on GitHub Actions, you probably want to leverage their matrix strategy. So having some story for this type of testing in an environment like that would be awesome. Again, I think this needs to be a bit abstracted so that folks with different CI systems can also use it.

Yes! I would like something like .test-npm-dependantsrc, where you can define the 1st class supported dependants that have to pass.

5. Multi-Registry Support/Fetching Source

Looks like the package fetch-package-source does not really have hooks for alternate source fetching. It seems like just making this an option you can inject would solve for folks needing to fetch from different sources.

Absolutely! GitHub was used just for 80/20, this needs a more complete solution, and great idea to make this configurable, so it would also work with private setups.