search

ipfs / benchmarks

Benchmarking for IPFS
MIT License
19 stars 8 forks source link

Architecture to add libp2p tests #198

Open Elexy opened 5 years ago

Elexy commented 5 years ago

Question that came out is they want to write benchmarks against libp2p. We can clone the repo and infrastructure for libp2p or add the login in the runner to use the libp2p or js-ipfs repo. The same dashboard could be used and filter by project. @e_lexy @matteo ^^^^^ when you get a chance thoughts on the above.

matteo [5:12 PM] @e_lexy you do the architecture for that solution? Sharing things might be good.. up to deploying a double. What’d be the cleanest way? Can you assemble a proposal on an gdoc and I’ll review?

vasco-santos commented 5 years ago

Any update on this @Elexy @mcollina ?

Elexy commented 5 years ago

Working on this today.

Elexy commented 5 years ago

@vasco-santos I've added a branch with a design diagram. https://github.com/ipfs/benchmarks/blob/add_libp2p_arch/runner/Libp2p%20Runner.png The key differentiator would be the provisioning. This takes care of setting up the right libs at the right version on the Minion.

Elexy commented 5 years ago

@vasco-santos I've made a design diagram

The main difference is the Provisioning. This is where we make sure the minion has the right libs at the right version before we run the benchmarks.

vasco-santos commented 5 years ago

Thanks for the proposal @Elexy !

So, in this approach we need to use this repo for all the tests and mark each test as being part of a project?

Elexy commented 5 years ago

Hi @vasco-santos The way we get the library in the test bench is quite specific to this setup. https://github.com/ipfs/benchmarks/blob/master/tests/getIpfs.sh The config.js also relies on paths defined by ENV vars (or defaults) for this project.

We could generalize is even more and make it a generic runner, able to run arbitrary scripts and retrieve the output, but I'd say we'd probably be building a full blow CI system.

vasco-santos commented 5 years ago

I understand that generalize this solution more, at least at this moment seems not the best approach.

I would like to have the libp2p benchmark tests in its own repo (libp2p/benchmarks). Moreover, I think it would be ok if we use this repo and infrastructure for both, and it gets the libp2p/benchmark tests from there. What do you think?

Elexy commented 5 years ago

@vasco-santos that makes sense. @litzenberger should we lead the way by splitting the js-ipfs tests off of this repo?

litzenberger commented 5 years ago

@Elexy I think the js-ipfs tests in the repo are fine, he just wants to pull another repo that will need to be synced by the runner. @vasco-santos did I read that right?

vasco-santos commented 5 years ago

Yes, we should be able to get tests from other repos, such as libp2p/benchmarks repo, and in the future probably ipld/benchmarks as well.

Elexy commented 5 years ago

Currently we are rsyncing the tests over to the minion from the controller from this repo.In the new situation I'd need to run a git clone of a specified repo before we can run the tests run the tests.

Elexy commented 5 years ago

As discussed in the latest sync call I want to present a proposal to allow the runner to run benchmark tests from a different repository. The proposal is here. /cc @vasco-santos @alanshaw @daviddias

vasco-santos commented 5 years ago

Thanks for the proposal @Elexy !