More thorough benchmarks [$200]

[Krastanov]

Bug bounty logistic details (click to expand)

To claim exclusive time to work on this bounty either post a comment here or message [skrastanov@umass.edu](mailto:skrastanov@umass.edu) with: - your name - github username - **(optional)** a brief list of previous pertinent projects you have engaged in If you want to, you can work on this project without making a claim, however claims are encouraged to give you and other contributors peace of mind. Whoever has made a claim takes precedence when solutions are considered. You can always propose your own funded project, if you would like to contribute something of value that is not yet covered by an official bounty.

The project is claimed by @coderight1 until Aug 12th 2024.

Project: "More thorough benchmarks" [$200]

We have a small benchmark suite already implemented, which is executed as part of our CI runs. It is defined in the benchmark folder and reported for each pull request. We would like to expand this benchmark suite to include many more facets of this library. E.g. basic register operations using a variety of backends; queries and tags and locks on registers and channels; time to import; time to run examples. The new benchmarks should be legible, easy to follow, and organized by topic. Most of them should be microbenchmarks testing only one concept, but a few holistic benchmarks would make sense as well.

Required skills: Generic Julia skills.

Reviewer: Stefan Krastanov

Duration: 1 month

Payout procedure:

The Funding for these bounties comes from the National Science Foundation and from the NSF Center for Quantum Networks. The payouts are managed by the NumFOCUS foundation and processed in bulk once every two months. If you live in a country in which NumFOCUS can make payments, you can participate in this bounty program.

Click here for more details about the bug bounty program.

[coderight1]

Hi @Krastanov , I have recently started learning Julia , and willing to work on this.

[Krastanov]

Please feel free to go ahead with it. Before I reserve it for you though, let's discuss what you plan to add in this comment thread. While this is a good issue for a first-time developer, it is not an easy issue for someone who has not written any user code with this library. I would suggest trying to run a few examples and creating a simulation of something interesting first. The benchmarks are supposed to have a very thorough coverage.

[coderight1]

Hi @Krastanov,

Thank you for the opportunity. I appreciate your advice and would like to proceed with the following plan for expanding the benchmark suite:

Basic Register Operations: Add benchmarks for creating, updating, and deleting registers using various backends.

Queries, Tags, and Locks: Benchmark the time for queries, adding/removing tags, and locking/unlocking operations on registers and channels.

Import Time: Measure the time it takes to import the library in different environments.

Example Run Time: Benchmark the execution time of existing examples. Create and benchmark new examples to cover additional scenarios.

To keep things organized, I plan to create different files for each facet of the library, such as basic register operations, queries and tags, locks on registers and channels, etc. This way, the benchmarks will be easier to manage and understand.

Please let me know if there are any specific areas you would like me to prioritize or any additional details I should consider.

[Krastanov]

Sounds good! A few suggestions:

• Check out the benchmarks setup of QuantumClifford.jl for reference
• Be careful with the eval and samples keywords: in some situations, especially if you are modifying/deleting things, you have to have eval=1, otherwise you get very misleading results
• Make sure the code is relatively easy to execute interactively benchmark by benchmark (i.e. factor out neatly potential setup/prepartion routines so a dev who wants to experiment knows what they need to load in their REPL)
• Have benchmarks both for large number of entities and for small number of entities (e.g. both large networks and small networks, large registers and registers with only a few slots, queries on large number of prerecorded tags and just a few tags)
• Be careful with setting things up that might require the discrete event simulator. There are many functions that just "set up an event" and the event does not run until you execute the simulator. Particularly around locks. I would suggest grouping this separately from the tag/query topic.
• For import time, it might be a bit awkward, because the benchmark will basically have to launch a separate julia instance. That also probably should have a limited number of eval and samples. Make sure that you actually wait on the newly started process (otherwise you might just launch it in the background and get a report that it is incredibly fast)
• Testing duration of examples might also be difficult (this is actually why this project is 200$ instead of 50$). No need to test all of them, especially ones that contain plots are probably a bad idea. You can see in the test_examples how we run them for tests, that would probably be a good reference (and it will make it more obvious which ones involve plotting and should be skipped). These should probably have just one eval and one sample given how incredibly slow they can be. It would be important to run these in a separate module (e.g. the way they are done in the tests). I suspect you will need something like eval(Module(), :(include("example file"))) to run this benchmark.
• Make sure that you also run each benchmark you define interactively (by changing @benchmarkable to @benchmark) just to see what would be the dev experience of someone who is running a single benchmark manually to explore why there is a slowdown in some future PR
• avoid using loops to generate benchmark instances (e.g. if you are just changing a single parameter). Sure, that would lead to more repetitive code and duplication, but it will help when in the future someone wants to play with a specific benchmark

I will make this reserved for 1month, but we can extend if necessary. Thank you for taking this on.