Integration testing for IPFS - IPTB vs kubernetes-ipfs vs mininet

[daviddias]

We've reached a point where it is becoming really hard to assert things with unit and interop tests, there are many network scenarios in which IPFS needs to be able to perform that can't be captured by those. We need integration tests.

There has been multiple endeavors started towards this end, for example IPTB and kubernetes-ipfs. These have been essential to test go-ipfs and ipfs-cluster, however, AFAIK, they don't let us create interesting network topologies.

Here is where mininet comes in, a virtual network toolkit that enables us to create all sorts of network topologies. Bonus, @mafintosh made it very easy to use through tapenet and @JGAntunes has been playing with it for a while to run tests for his M.Sc work.

I know a lot of folks here have opinions on testing and I want to make sure all of those are captured before starting yet another p2p test framework.

I've a short list with my humble requirements, these are:

• Code tests very easily and fast (writing a test shouldn't be harder than writing a unit test)
• Specify different network topologies, play with things like latency, packet loss and NAT configurations
• Run js-ipfs and go-ipfs daemons, js-ipfs instances (in process nodes), js-ipfs browser nodes (Chrome and Firefox)

[victorb]

We've been playing around with mininet before but the setup is a bit of a hassle currently. Packages for Ubuntu seems to work fine now but optimally, we would have docker images for the setup to become easy enough (I don't know if mininet runs outside of Linux envs right now so docker images might be a hard need if we want to use it)

However, first time I discovered tapenet last week, which made writing the tests a lot simpler and nicer. Would fit perfectly to replace the tests we have in ipfs/interop, as long as we keep a structure where we can replace nodes easily (possibly via js-ipfsd-ctl).

I also know that there has been light talks in #ipfs-dev about implementing NAT in mininet, as mininet currently supports just very basic setups now. That's something that we would probably have to help upstream in mininet.

Otherwise, I think tapenet is the way to go for these network/interop/intergration tests.

[JGAntunes]

Hey everyone,

So, I've actually been using a fork of mininet, containernet, which works under the same principles of mininet but the hosts are docker containers. This for my test scenarios is a bit more useful since I'm not forced to have all the needed dependencies locally to run each host (can run multiple versions of IPFS in the same network, on different hosts, for examples). I've been working on a way to create simple test nets in a declarative way - https://github.com/JGAntunes/pulsarcast-test-harness (specifically https://github.com/JGAntunes/pulsarcast-test-harness/blob/master/networks/tiny.json) - so feel free to poke around that repo if you're interested (it's still a WIP)

On the same note, just leaving this here also - https://maxinet.github.io/ - I haven't tried it, but could be of some use if you end up using either mininet or containernet.

Let me know if there's any other way I can help :+1:

[hugomrdias]

Regarding browsers more research is needed and probably a tapenet like package to get everything working together.

my thoughts right now about it:

• build topologies between daemons, different browsers(chrome, firefox= and different browser nodes (webworker,serviceworker,etc)
• build a simple app with a series of input/submit mini forms to execute operations, if wiring everything in test code and injecting becomes to hard.
• start browsers pointing to that app, get peer ids and wire tests with a test runner
• we should be able to do integrations tests between a IPFS node in the js thread, a node in a webworker, a node in a service worker and web extension ipfs companion
• should be able to inject code (karma like approach), have access to browser context, get output back to the test runner and have interaction capability (selenium like).
• electron stuff in here too (very little experience here)

tools:

• https://github.com/airtap/airtap
• https://github.com/DevExpress/testcafe
• https://github.com/GoogleChrome/puppeteer/
• https://github.com/autonome/puppeteer-fx

might also be possible to wire other browsers into a puppeteer like api through webdriver.

Please give feedback on this and contribute with your ideas

[lanzafame]

There is also this lightweight mininet alternative: https://laraget.com/blog/ovs-mesh-script-generator