search

qiboteam / qibojit-benchmarks

Benchmark code for qibojit performance accessment
Apache License 2.0
2 stars 3 forks source link

Add benchmark circuits #6

Closed stavros11 closed 3 years ago

stavros11 commented 3 years ago

Fixes #3 by adding the following circuits from Table 1 of the HyQuas paper:

Note that the depth/number of gates of some of these circuits is affected by parameters other than the number of qubits, such as the graph structure in QAOA or the target shift bitstring in Hidden shift. Care should be taken so that such parameters are kept the same when we are benchmarking different backends.

stavros11 commented 3 years ago

All circuits from the paper are now implemented and tested. The Bernstein-Vazirani, Hidden shift and QAOA circuits are implemented directly in qibo, while the basis change, supremacy and quantum volume are built using Cirq or qiskit and ported to qibo via OpenQASM. Note that in some cases the number of gates in the circuits is different than what is reported in the HyQuas paper table, however this may be because of difference in parameters (eg. in the QAOA the number of gates depends on the graph which is randomly generated).

Here are some numbers for these circuits on qibo machine:

CPU - qft - swaps=True nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 3 | 0.01605 | 0.05104 | 0.00032 | 0.00023 | 0.00001 | 0.00000 4 | 0.01889 | 0.05188 | 0.00041 | 0.00036 | 0.00001 | 0.00000 5 | 0.02351 | 0.05288 | 0.00055 | 0.00049 | 0.00001 | 0.00000 6 | 0.02231 | 0.05236 | 0.00076 | 0.00065 | 0.00001 | 0.00000 7 | 0.03618 | 0.05259 | 0.00099 | 0.00082 | 0.00001 | 0.00000 8 | 0.01510 | 0.05312 | 0.00096 | 0.00106 | 0.00001 | 0.00000 9 | 0.02503 | 0.05409 | 0.00147 | 0.00127 | 0.00001 | 0.00000 10 | 0.02699 | 0.05377 | 0.00143 | 0.00149 | 0.00001 | 0.00000 11 | 0.02740 | 0.05421 | 0.00205 | 0.00198 | 0.00001 | 0.00000 12 | 0.02812 | 0.05492 | 0.00245 | 0.00210 | 0.00001 | 0.00000 13 | 0.02289 | 0.05504 | 0.00296 | 0.00248 | 0.00002 | 0.00000 14 | 0.02532 | 0.05550 | 0.00271 | 0.00287 | 0.00003 | 0.00000 15 | 0.04149 | 0.05656 | 0.00384 | 0.00363 | 0.00006 | 0.00000 16 | 0.03312 | 0.05733 | 0.00477 | 0.00392 | 0.00011 | 0.00000 17 | 0.03821 | 0.05862 | 0.00478 | 0.00484 | 0.00019 | 0.00000 18 | 0.12776 | 0.06067 | 0.00815 | 0.00630 | 0.00047 | 0.00000 19 | 0.04438 | 0.06308 | 0.01050 | 0.00866 | 0.00098 | 0.00000 20 | 0.05176 | 0.06912 | 0.01990 | 0.01315 | 0.00219 | 0.00000 21 | 0.07233 | 0.07809 | 0.03955 | 0.02233 | 0.00650 | 0.00000 22 | 0.10179 | 0.09261 | 0.07000 | 0.03867 | 0.01129 | 0.00001 23 | 0.17879 | 0.13717 | 0.14177 | 0.08662 | 0.02280 | 0.00000 24 | 0.40603 | 0.31324 | 0.33979 | 0.24138 | 0.04458 | 0.00001 25 | 0.96288 | 1.07424 | 0.85978 | 0.93733 | 0.08494 | 0.00001
CPU - variational - nlayers=1, varlayer=False nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 3 | 0.02309 | 0.04926 | 0.00024 | 0.00024 | 0.00001 | 0.00000 4 | 0.02885 | 0.05031 | 0.00040 | 0.00039 | 0.00001 | 0.00000 5 | 0.01919 | 0.04889 | 0.00047 | 0.00041 | 0.00001 | 0.00000 6 | 0.02639 | 0.04968 | 0.00059 | 0.00050 | 0.00001 | 0.00000 7 | 0.02609 | 0.05046 | 0.00064 | 0.00056 | 0.00001 | 0.00000 8 | 0.01353 | 0.04976 | 0.00077 | 0.00065 | 0.00001 | 0.00000 9 | 0.02787 | 0.04922 | 0.00093 | 0.00070 | 0.00001 | 0.00000 10 | 0.01805 | 0.05056 | 0.00073 | 0.00080 | 0.00001 | 0.00000 11 | 0.01820 | 0.05066 | 0.00080 | 0.00084 | 0.00001 | 0.00000 12 | 0.02865 | 0.05034 | 0.00109 | 0.00094 | 0.00001 | 0.00000 13 | 0.02417 | 0.05017 | 0.00125 | 0.00110 | 0.00002 | 0.00000 14 | 0.03341 | 0.05076 | 0.00140 | 0.00117 | 0.00003 | 0.00000 15 | 0.02197 | 0.05096 | 0.00130 | 0.00132 | 0.00005 | 0.00000 16 | 0.03022 | 0.05122 | 0.00185 | 0.00145 | 0.00010 | 0.00000 17 | 0.02293 | 0.05175 | 0.00204 | 0.00187 | 0.00020 | 0.00000 18 | 0.03011 | 0.05398 | 0.00335 | 0.00298 | 0.00049 | 0.00000 19 | 0.02358 | 0.05614 | 0.00520 | 0.00430 | 0.00101 | 0.00000 20 | 0.03202 | 0.05945 | 0.00895 | 0.00709 | 0.00226 | 0.00000 21 | 0.04459 | 0.06340 | 0.02332 | 0.01252 | 0.00609 | 0.00000 22 | 0.06116 | 0.08024 | 0.03993 | 0.02335 | 0.01146 | 0.00000 23 | 0.10267 | 0.10913 | 0.07499 | 0.05177 | 0.02283 | 0.00000 24 | 0.23924 | 0.22409 | 0.21636 | 0.16315 | 0.04603 | 0.00001 25 | 0.56671 | 0.55589 | 0.53530 | 0.48281 | 0.08594 | 0.00001
CPU - bernstein-vazirani - nqubits=3 nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 3 | 0.01089 | 0.05941 | 0.00042 | 0.00025 | 0.00002 | 0.00000 4 | 0.02453 | 0.05228 | 0.00039 | 0.00035 | 0.00001 | 0.00000 5 | 0.02184 | 0.05141 | 0.00038 | 0.00042 | 0.00001 | 0.00000 6 | 0.01712 | 0.05166 | 0.00057 | 0.00047 | 0.00001 | 0.00000 7 | 0.01516 | 0.05204 | 0.00086 | 0.00056 | 0.00001 | 0.00000 8 | 0.00990 | 0.05473 | 0.00078 | 0.00062 | 0.00001 | 0.00000 9 | 0.02023 | 0.05190 | 0.00089 | 0.00069 | 0.00001 | 0.00000 10 | 0.01619 | 0.05299 | 0.00071 | 0.00077 | 0.00001 | 0.00000 11 | 0.02073 | 0.05202 | 0.00103 | 0.00084 | 0.00001 | 0.00000 12 | 0.01589 | 0.05217 | 0.00090 | 0.00091 | 0.00001 | 0.00000 13 | 0.02221 | 0.05926 | 0.00121 | 0.00101 | 0.00002 | 0.00000 14 | 0.02280 | 0.05264 | 0.00137 | 0.00113 | 0.00003 | 0.00000 15 | 0.02147 | 0.05413 | 0.00157 | 0.00129 | 0.00006 | 0.00000 16 | 0.02587 | 0.05362 | 0.00161 | 0.00144 | 0.00010 | 0.00000 17 | 0.02379 | 0.05778 | 0.00208 | 0.00179 | 0.00020 | 0.00000 18 | 0.02580 | 0.05570 | 0.00307 | 0.00292 | 0.00046 | 0.00000 19 | 0.01873 | 0.06164 | 0.00528 | 0.00404 | 0.00102 | 0.00000 20 | 0.03714 | 0.06383 | 0.00802 | 0.00692 | 0.00215 | 0.00000 21 | 0.05519 | 0.06728 | 0.02136 | 0.01342 | 0.00653 | 0.00001 22 | 0.05349 | 0.08320 | 0.03744 | 0.02332 | 0.01176 | 0.00000 23 | 0.10833 | 0.09902 | 0.08117 | 0.05575 | 0.02292 | 0.00000 24 | 0.29456 | 0.26861 | 0.26737 | 0.21286 | 0.04422 | 0.00001 25 | 0.58761 | 0.58014 | 0.55619 | 0.53397 | 0.08509 | 0.00001
CPU - hidden-shift nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 3 | 0.02340 | 0.05025 | 0.00032 | 0.00037 | 0.00001 | 0.00000 4 | 0.01410 | 0.05166 | 0.00069 | 0.00058 | 0.00001 | 0.00000 5 | 0.01987 | 0.05343 | 0.00074 | 0.00069 | 0.00001 | 0.00000 6 | 0.02465 | 0.05258 | 0.00108 | 0.00083 | 0.00001 | 0.00000 7 | 0.02535 | 0.05211 | 0.00106 | 0.00093 | 0.00001 | 0.00000 8 | 0.02188 | 0.05296 | 0.00136 | 0.00108 | 0.00001 | 0.00000 9 | 0.02009 | 0.05382 | 0.00172 | 0.00109 | 0.00002 | 0.00000 10 | 0.04066 | 0.05319 | 0.00157 | 0.00134 | 0.00001 | 0.00000 11 | 0.01946 | 0.05292 | 0.00139 | 0.00133 | 0.00001 | 0.00000 12 | 0.01766 | 0.05346 | 0.00204 | 0.00163 | 0.00002 | 0.00000 13 | 0.02395 | 0.05417 | 0.00169 | 0.00183 | 0.00002 | 0.00000 14 | 0.02674 | 0.05418 | 0.00239 | 0.00217 | 0.00003 | 0.00000 15 | 0.02149 | 0.05406 | 0.00209 | 0.00215 | 0.00005 | 0.00000 16 | 0.02691 | 0.05475 | 0.00306 | 0.00254 | 0.00011 | 0.00000 17 | 0.02100 | 0.05593 | 0.00317 | 0.00328 | 0.00020 | 0.00000 18 | 0.02143 | 0.05819 | 0.00588 | 0.00565 | 0.00090 | 0.00000 19 | 0.03567 | 0.06311 | 0.00931 | 0.00833 | 0.00164 | 0.00000 20 | 0.03107 | 0.06756 | 0.01499 | 0.01203 | 0.00328 | 0.00000 21 | 0.05756 | 0.07470 | 0.03094 | 0.02012 | 0.00652 | 0.00001 22 | 0.07356 | 0.09541 | 0.05912 | 0.03822 | 0.01219 | 0.00001 23 | 0.16543 | 0.13968 | 0.14359 | 0.08185 | 0.02268 | 0.00001 24 | 0.39992 | 0.42897 | 0.38190 | 0.31793 | 0.04501 | 0.00001 25 | 1.15255 | 1.05709 | 1.08410 | 1.03383 | 0.08613 | 0.00001 26 | 2.07642 | 2.02287 | 2.01725 | 1.92217 | 0.15952 | 0.00002 27 | 3.91565 | 3.80036 | 3.80515 | 3.65596 | 0.30381 | 0.00002 28 | 8.20477 | 8.02504 | 8.40155 | 7.95015 | 0.58060 | 0.00002 29 | 19.33704 | 19.11399 | 19.58697 | 18.97671 | 1.14542 | 0.00002 30 | 34.70662 | 34.12049 | 35.15644 | 33.93655 | 2.26153 | 0.00003
CPU - qaoa - nparams=2 nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 4 | 0.00455 | 0.04964 | 0.00069 | 0.00070 | 0.00001 | 0.00000 6 | 0.00573 | 0.05123 | 0.00105 | 0.00098 | 0.00001 | 0.00000 8 | 0.00733 | 0.05472 | 0.00130 | 0.00136 | 0.00001 | 0.00000 10 | 0.00803 | 0.05197 | 0.00152 | 0.00168 | 0.00001 | 0.00000 12 | 0.01054 | 0.05136 | 0.00241 | 0.00196 | 0.00002 | 0.00000 14 | 0.02122 | 0.05292 | 0.00330 | 0.00254 | 0.00004 | 0.00000 16 | 0.01344 | 0.05557 | 0.00384 | 0.00381 | 0.00010 | 0.00000 18 | 0.02039 | 0.06058 | 0.00863 | 0.00819 | 0.00081 | 0.00000 20 | 0.03944 | 0.08351 | 0.02540 | 0.02096 | 0.00317 | 0.00000 22 | 0.08053 | 0.13025 | 0.07269 | 0.07596 | 0.01157 | 0.00000 24 | 0.54744 | 0.68262 | 0.44266 | 0.53602 | 0.04534 | 0.00001 26 | 2.89582 | 2.91815 | 2.85205 | 2.74395 | 0.15942 | 0.00001
CPU - supremacy - depth=2, seed=123 nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 3 | 0.00365 | 0.05488 | 0.00033 | 0.00029 | 0.00001 | 0.00000 4 | 0.00387 | 0.04876 | 0.00039 | 0.00039 | 0.00001 | 0.00000 5 | 0.00405 | 0.04777 | 0.00045 | 0.00053 | 0.00001 | 0.00000 6 | 0.00445 | 0.04778 | 0.00056 | 0.00056 | 0.00001 | 0.00000 7 | 0.01119 | 0.04779 | 0.00059 | 0.00062 | 0.00001 | 0.00000 8 | 0.00650 | 0.04833 | 0.00072 | 0.00071 | 0.00001 | 0.00000 9 | 0.00532 | 0.04864 | 0.00080 | 0.00077 | 0.00001 | 0.00000 10 | 0.00574 | 0.04922 | 0.00089 | 0.00089 | 0.00001 | 0.00000 11 | 0.00661 | 0.04955 | 0.00120 | 0.00096 | 0.00001 | 0.00000 12 | 0.00623 | 0.04869 | 0.00107 | 0.00102 | 0.00001 | 0.00000 13 | 0.00666 | 0.05408 | 0.00115 | 0.00112 | 0.00002 | 0.00000 14 | 0.01539 | 0.04875 | 0.00147 | 0.00131 | 0.00003 | 0.00000 15 | 0.00736 | 0.04929 | 0.00154 | 0.00145 | 0.00006 | 0.00000 16 | 0.00811 | 0.04948 | 0.00191 | 0.00171 | 0.00010 | 0.00000 17 | 0.00937 | 0.05706 | 0.00255 | 0.00213 | 0.00020 | 0.00000 18 | 0.01571 | 0.05264 | 0.00350 | 0.00331 | 0.00045 | 0.00000 19 | 0.02326 | 0.05744 | 0.00579 | 0.00480 | 0.00093 | 0.00000 20 | 0.02051 | 0.05959 | 0.01003 | 0.00777 | 0.00203 | 0.00000 21 | 0.03096 | 0.06952 | 0.02359 | 0.01582 | 0.00633 | 0.00000 22 | 0.05152 | 0.08588 | 0.04409 | 0.02998 | 0.01145 | 0.00000 23 | 0.09911 | 0.12422 | 0.08864 | 0.06760 | 0.02289 | 0.00000 24 | 0.25970 | 0.18627 | 0.24981 | 0.11970 | 0.04537 | 0.00001 25 | 0.65134 | 0.71938 | 0.67256 | 0.63149 | 0.08605 | 0.00001
CPU - basis-change - simulation_time=1, seed=123 nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 3 | 0.03669 | 0.19282 | 0.00260 | 0.00203 | 0.00001 | 0.00000 4 | 0.02611 | 0.19529 | 0.00490 | 0.00407 | 0.00001 | 0.00000 5 | 0.04101 | 0.20224 | 0.00782 | 0.00654 | 0.00001 | 0.00000 6 | 0.07564 | 0.20633 | 0.01126 | 0.00972 | 0.00001 | 0.00000 7 | 0.06950 | 0.21151 | 0.01535 | 0.01341 | 0.00001 | 0.00000 8 | 0.09499 | 0.21820 | 0.01973 | 0.01761 | 0.00001 | 0.00000 9 | 0.12427 | 0.22474 | 0.02129 | 0.02304 | 0.00001 | 0.00000 10 | 0.13950 | 0.23619 | 0.02989 | 0.02900 | 0.00001 | 0.00000 11 | 0.15998 | 0.24450 | 0.04147 | 0.03412 | 0.00001 | 0.00000 12 | 0.18231 | 0.25666 | 0.04140 | 0.04210 | 0.00002 | 0.00000 13 | 0.32632 | 0.26881 | 0.05118 | 0.05176 | 0.00002 | 0.00000 14 | 0.35376 | 0.28218 | 0.05651 | 0.05858 | 0.00005 | 0.00000 15 | 0.25522 | 0.16930 | 0.07174 | 0.07991 | 0.00007 | 0.00000 16 | 0.30005 | 0.18698 | 0.08530 | 0.08896 | 0.00011 | 0.00000 17 | 0.37337 | 0.21590 | 0.12921 | 0.11840 | 0.00021 | 0.00000 18 | 0.47100 | 0.27136 | 0.17344 | 0.17234 | 0.00047 | 0.00000 19 | 0.63373 | 0.36099 | 0.27553 | 0.24425 | 0.00104 | 0.00000 20 | 0.91893 | 0.56027 | 0.43089 | 0.42259 | 0.00217 | 0.00000 21 | 1.49453 | 0.93307 | 0.83482 | 0.64832 | 0.00633 | 0.00000 22 | 2.36890 | 1.73958 | 1.52776 | 1.19398 | 0.01259 | 0.00001 23 | 4.18439 | 3.61030 | 3.26464 | 3.17466 | 0.02390 | 0.00001 24 | 10.40961 | 5.96261 | 8.68620 | 5.57630 | 0.04576 | 0.00001 25 | 59.49890 | 60.35601 | 57.75743 | 59.75753 | 0.08632 | 0.00001
CPU - quantum-volume - depth=1, seed=123 nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 3 | 0.00365 | 0.04814 | 0.00031 | 0.00034 | 0.00001 | 0.00000 4 | 0.00491 | 0.04853 | 0.00058 | 0.00057 | 0.00001 | 0.00000 5 | 0.00482 | 0.04852 | 0.00059 | 0.00059 | 0.00001 | 0.00000 6 | 0.01050 | 0.04965 | 0.00082 | 0.00083 | 0.00001 | 0.00000 7 | 0.00587 | 0.05018 | 0.00082 | 0.00087 | 0.00001 | 0.00000 8 | 0.00705 | 0.05283 | 0.00108 | 0.00111 | 0.00001 | 0.00000 9 | 0.00708 | 0.05176 | 0.00109 | 0.00113 | 0.00001 | 0.00000 10 | 0.00820 | 0.05344 | 0.00135 | 0.00153 | 0.00001 | 0.00000 11 | 0.00840 | 0.05409 | 0.00138 | 0.00142 | 0.00001 | 0.00000 12 | 0.00953 | 0.05127 | 0.00159 | 0.00168 | 0.00001 | 0.00000 13 | 0.00945 | 0.05079 | 0.00165 | 0.00171 | 0.00002 | 0.00000 14 | 0.01076 | 0.05127 | 0.00201 | 0.00212 | 0.00003 | 0.00000 15 | 0.01571 | 0.05149 | 0.00218 | 0.00222 | 0.00005 | 0.00000 16 | 0.01751 | 0.05246 | 0.00304 | 0.00272 | 0.00011 | 0.00000 17 | 0.01462 | 0.05369 | 0.00387 | 0.00343 | 0.00020 | 0.00000 18 | 0.01782 | 0.06285 | 0.00577 | 0.00540 | 0.00045 | 0.00000 19 | 0.02179 | 0.06459 | 0.00881 | 0.00816 | 0.00091 | 0.00000 20 | 0.02914 | 0.06826 | 0.01717 | 0.01186 | 0.00202 | 0.00000 21 | 0.04451 | 0.07896 | 0.03599 | 0.02513 | 0.00578 | 0.00000 22 | 0.07989 | 0.11160 | 0.07114 | 0.05291 | 0.01158 | 0.00000 23 | 0.11617 | 0.15617 | 0.10854 | 0.10741 | 0.02274 | 0.00001 24 | 0.42205 | 0.49671 | 0.39665 | 0.39118 | 0.04536 | 0.00001 25 | 1.03154 | 1.09640 | 1.07043 | 1.01721 | 0.08623 | 0.00001
GPU - qft - swaps=True nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 3 | 0.39427 | 0.59848 | 0.00020 | 0.00042 | 0.00007 | 0.00002 4 | 0.39445 | 0.59925 | 0.00029 | 0.00070 | 0.00006 | 0.00003 5 | 0.39615 | 0.59989 | 0.00036 | 0.00102 | 0.00005 | 0.00003 6 | 0.39726 | 0.60191 | 0.00047 | 0.00143 | 0.00007 | 0.00003 7 | 0.39741 | 0.60213 | 0.00060 | 0.00189 | 0.00007 | 0.00003 8 | 0.40022 | 0.60536 | 0.00074 | 0.00244 | 0.00007 | 0.00003 9 | 0.40099 | 0.60244 | 0.00083 | 0.00307 | 0.00007 | 0.00003 10 | 0.40594 | 0.60559 | 0.00108 | 0.00381 | 0.00007 | 0.00003 11 | 0.40621 | 0.60699 | 0.00120 | 0.00468 | 0.00009 | 0.00003 12 | 0.40798 | 0.60792 | 0.00175 | 0.00555 | 0.00007 | 0.00003 13 | 0.40669 | 0.60842 | 0.00167 | 0.00645 | 0.00013 | 0.00004 14 | 0.40829 | 0.61196 | 0.00188 | 0.00746 | 0.00016 | 0.00006 15 | 0.41423 | 0.61506 | 0.00220 | 0.00860 | 0.00018 | 0.00009 16 | 0.41638 | 0.61356 | 0.00239 | 0.00968 | 0.00031 | 0.00015 17 | 0.41757 | 0.61629 | 0.00272 | 0.01071 | 0.00042 | 0.00021 18 | 0.41985 | 0.61780 | 0.00309 | 0.01220 | 0.00070 | 0.00035 19 | 0.42439 | 0.62128 | 0.00367 | 0.01452 | 0.00341 | 0.00063 20 | 0.42778 | 0.62755 | 0.00390 | 0.01765 | 0.00562 | 0.00118 21 | 0.43131 | 0.63304 | 0.00688 | 0.02476 | 0.02005 | 0.00578 22 | 0.43646 | 0.64491 | 0.00814 | 0.03596 | 0.04423 | 0.01129 23 | 0.45738 | 0.67420 | 0.01058 | 0.05853 | 0.09525 | 0.02268 24 | 0.50392 | 0.71670 | 0.01559 | 0.10522 | 0.20118 | 0.04577 25 | 0.59962 | 0.81502 | 0.02396 | 0.20244 | 0.41782 | 0.09049 26 | 0.80881 | 1.02130 | 0.04263 | 0.40903 | 0.86224 | 0.17954 27 | 1.25027 | 1.45227 | 0.07687 | 0.84398 | 1.78189 | 0.35916 28 | 2.16617 | 2.37749 | 0.14578 | 1.76903 | 3.65996 | 0.71787 29 | 4.11378 | 4.31858 | 0.28439 | 3.71741 | 7.54639 | 1.43527 30 | 8.24065 | 8.43621 | 0.54602 | 7.85551 | 15.52659 | 2.86209
GPU - variational - nlayers=1, varlayer=False nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 3 | 0.39352 | 0.60176 | 0.00023 | 0.00040 | 0.00006 | 0.00002 4 | 0.39555 | 0.59690 | 0.00029 | 0.00059 | 0.00006 | 0.00002 5 | 0.39609 | 0.59773 | 0.00033 | 0.00065 | 0.00006 | 0.00002 6 | 0.39756 | 0.59821 | 0.00068 | 0.00082 | 0.00009 | 0.00002 7 | 0.39639 | 0.60017 | 0.00040 | 0.00090 | 0.00005 | 0.00003 8 | 0.39882 | 0.59866 | 0.00048 | 0.00106 | 0.00006 | 0.00003 9 | 0.39693 | 0.59976 | 0.00051 | 0.00119 | 0.00007 | 0.00003 10 | 0.39954 | 0.59909 | 0.00084 | 0.00139 | 0.00009 | 0.00003 11 | 0.40106 | 0.59909 | 0.00103 | 0.00150 | 0.00011 | 0.00003 12 | 0.40081 | 0.59938 | 0.00067 | 0.00170 | 0.00007 | 0.00003 13 | 0.40143 | 0.59919 | 0.00073 | 0.00183 | 0.00009 | 0.00004 14 | 0.40272 | 0.59998 | 0.00080 | 0.00203 | 0.00012 | 0.00006 15 | 0.40423 | 0.60012 | 0.00082 | 0.00208 | 0.00022 | 0.00009 16 | 0.40444 | 0.60400 | 0.00133 | 0.00228 | 0.00030 | 0.00015 17 | 0.40129 | 0.60305 | 0.00085 | 0.00238 | 0.00038 | 0.00021 18 | 0.40300 | 0.60528 | 0.00101 | 0.00280 | 0.00110 | 0.00035 19 | 0.40351 | 0.60497 | 0.00120 | 0.00348 | 0.00260 | 0.00061 20 | 0.40512 | 0.60573 | 0.00141 | 0.00486 | 0.00478 | 0.00116 21 | 0.40962 | 0.60772 | 0.00343 | 0.00750 | 0.01927 | 0.00579 22 | 0.41905 | 0.61385 | 0.00460 | 0.01284 | 0.03930 | 0.01162 23 | 0.42700 | 0.62381 | 0.00694 | 0.02378 | 0.08009 | 0.02253 24 | 0.45065 | 0.64868 | 0.01126 | 0.04690 | 0.16382 | 0.04466 25 | 0.50062 | 0.69247 | 0.02002 | 0.09354 | 0.33262 | 0.09014 26 | 0.59837 | 0.79130 | 0.03707 | 0.19293 | 0.67353 | 0.17830 27 | 0.79912 | 0.98790 | 0.07167 | 0.39334 | 1.36295 | 0.36082 28 | 1.21678 | 1.41266 | 0.14011 | 0.81769 | 2.74996 | 0.72444 29 | 2.07006 | 2.25483 | 0.27752 | 1.67445 | 5.55115 | 1.45459 30 | 3.86936 | 4.04557 | 0.54129 | 3.48305 | 11.22867 | 2.86268
GPU - bernstein-vazirani - nqubits=3 nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 3 | 0.39543 | 0.59683 | 0.00023 | 0.00039 | 0.00006 | 0.00002 4 | 0.39424 | 0.59630 | 0.00027 | 0.00051 | 0.00005 | 0.00002 5 | 0.39438 | 0.59820 | 0.00032 | 0.00064 | 0.00005 | 0.00003 6 | 0.39310 | 0.59919 | 0.00037 | 0.00075 | 0.00006 | 0.00003 7 | 0.39609 | 0.59634 | 0.00043 | 0.00088 | 0.00007 | 0.00003 8 | 0.39511 | 0.60136 | 0.00046 | 0.00101 | 0.00008 | 0.00003 9 | 0.39675 | 0.59813 | 0.00052 | 0.00116 | 0.00006 | 0.00003 10 | 0.39648 | 0.59785 | 0.00070 | 0.00128 | 0.00008 | 0.00003 11 | 0.39764 | 0.59736 | 0.00063 | 0.00148 | 0.00006 | 0.00003 12 | 0.39820 | 0.60499 | 0.00093 | 0.00163 | 0.00011 | 0.00003 13 | 0.39804 | 0.59956 | 0.00091 | 0.00179 | 0.00011 | 0.00004 14 | 0.39755 | 0.60123 | 0.00076 | 0.00191 | 0.00012 | 0.00006 15 | 0.39775 | 0.59899 | 0.00084 | 0.00204 | 0.00015 | 0.00009 16 | 0.39822 | 0.60135 | 0.00082 | 0.00218 | 0.00024 | 0.00015 17 | 0.39905 | 0.60004 | 0.00086 | 0.00230 | 0.00038 | 0.00021 18 | 0.39943 | 0.60254 | 0.00134 | 0.00275 | 0.00073 | 0.00035 19 | 0.42219 | 0.60240 | 0.00114 | 0.00354 | 0.00195 | 0.00061 20 | 0.39907 | 0.60255 | 0.00129 | 0.00497 | 0.00491 | 0.00115 21 | 0.40253 | 0.61011 | 0.00343 | 0.00803 | 0.01977 | 0.00591 22 | 0.40673 | 0.61280 | 0.00451 | 0.01380 | 0.04010 | 0.01139 23 | 0.41855 | 0.62261 | 0.00712 | 0.02582 | 0.08221 | 0.02225 24 | 0.44253 | 0.64718 | 0.01114 | 0.05072 | 0.16870 | 0.04450 25 | 0.49642 | 0.69899 | 0.01957 | 0.10262 | 0.34112 | 0.09141 26 | 0.60406 | 0.80824 | 0.03793 | 0.21030 | 0.69058 | 0.17880 27 | 0.82085 | 1.02684 | 0.07179 | 0.43261 | 1.39477 | 0.35461 28 | 1.27569 | 1.48355 | 0.14038 | 0.89287 | 2.83730 | 0.72337 29 | 2.21244 | 2.43715 | 0.27738 | 1.84456 | 5.71022 | 1.42313 30 | 4.15160 | 4.39386 | 0.53885 | 3.80935 | 11.55327 | 2.85149
GPU - hidden-shift nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 3 | 0.39708 | 0.59717 | 0.00034 | 0.00056 | 0.00005 | 0.00003 4 | 0.39944 | 0.59767 | 0.00045 | 0.00087 | 0.00007 | 0.00003 5 | 0.39883 | 0.68273 | 0.00051 | 0.00096 | 0.00005 | 0.00003 6 | 0.39992 | 0.59918 | 0.00065 | 0.00123 | 0.00005 | 0.00003 7 | 0.39959 | 0.59849 | 0.00066 | 0.00133 | 0.00005 | 0.00003 8 | 0.39936 | 0.60195 | 0.00076 | 0.00154 | 0.00004 | 0.00003 9 | 0.39902 | 0.59600 | 0.00075 | 0.00168 | 0.00006 | 0.00003 10 | 0.40052 | 0.60077 | 0.00099 | 0.00201 | 0.00006 | 0.00003 11 | 0.39874 | 0.59993 | 0.00116 | 0.00218 | 0.00008 | 0.00003 12 | 0.39945 | 0.60033 | 0.00112 | 0.00254 | 0.00007 | 0.00004 13 | 0.40213 | 0.60115 | 0.00116 | 0.00263 | 0.00014 | 0.00004 14 | 0.40164 | 0.59918 | 0.00125 | 0.00298 | 0.00010 | 0.00006 15 | 0.40184 | 0.60087 | 0.00217 | 0.00310 | 0.00018 | 0.00010 16 | 0.40587 | 0.59979 | 0.00228 | 0.00333 | 0.00038 | 0.00016 17 | 0.40215 | 0.60298 | 0.00138 | 0.00345 | 0.00037 | 0.00022 18 | 0.40236 | 0.60158 | 0.00163 | 0.00404 | 0.00067 | 0.00035 19 | 0.40508 | 0.60748 | 0.00192 | 0.00568 | 0.00231 | 0.00063 20 | 0.40352 | 0.60494 | 0.00219 | 0.00812 | 0.00614 | 0.00117 21 | 0.41090 | 0.61006 | 0.00414 | 0.01264 | 0.02312 | 0.00581 22 | 0.42153 | 0.62165 | 0.00560 | 0.02338 | 0.04758 | 0.01122 23 | 0.43699 | 0.64121 | 0.00751 | 0.04108 | 0.09588 | 0.02231 24 | 0.49057 | 0.68835 | 0.01209 | 0.09097 | 0.20689 | 0.04502 25 | 0.58899 | 0.78979 | 0.02092 | 0.19111 | 0.42744 | 0.08970 26 | 0.74919 | 0.94837 | 0.03842 | 0.35221 | 0.82969 | 0.17917 27 | 1.05931 | 1.25792 | 0.07228 | 0.66427 | 1.63036 | 0.35576 28 | 1.83227 | 2.04070 | 0.14065 | 1.44971 | 3.37959 | 0.72143 29 | 3.83872 | 4.03810 | 0.27879 | 3.45962 | 7.34118 | 1.43777 30 | 6.55192 | 6.76266 | 0.53856 | 6.20507 | 13.90758 | 2.86778
GPU - qaoa - nparams=2 nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 4 | 0.39870 | 0.59888 | 0.00054 | 0.00104 | 0.00008 | 0.00003 6 | 0.39921 | 0.60090 | 0.00076 | 0.00156 | 0.00005 | 0.00003 8 | 0.40364 | 0.60198 | 0.00093 | 0.00203 | 0.00007 | 0.00003 10 | 0.40406 | 0.60269 | 0.00182 | 0.00274 | 0.00008 | 0.00003 12 | 0.40621 | 0.60688 | 0.00123 | 0.00416 | 0.00040 | 0.00004 14 | 0.41052 | 0.60515 | 0.00154 | 0.00487 | 0.00043 | 0.00006 16 | 0.40958 | 0.60475 | 0.00177 | 0.00556 | 0.00058 | 0.00016 18 | 0.40952 | 0.60909 | 0.00195 | 0.00653 | 0.00142 | 0.00036 20 | 0.42028 | 0.61893 | 0.00269 | 0.01473 | 0.01160 | 0.00117 22 | 0.44650 | 0.64520 | 0.00588 | 0.04303 | 0.06658 | 0.01132 24 | 0.56913 | 0.76410 | 0.01251 | 0.16347 | 0.27601 | 0.04519 26 | 1.08536 | 1.28255 | 0.03888 | 0.68363 | 1.15349 | 0.18097 28 | 3.29941 | 3.50769 | 0.14190 | 2.92377 | 4.82184 | 0.71890 30 | 12.77168 | 13.07408 | 0.54146 | 12.54116 | 20.15763 | 2.88949
GPU - supremacy - depth=2, seed=123 nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 3 | 0.39432 | 0.59874 | 0.00025 | 0.00048 | 0.00007 | 0.00002 4 | 0.39534 | 0.59733 | 0.00029 | 0.00059 | 0.00007 | 0.00003 5 | 0.39461 | 0.59834 | 0.00035 | 0.00071 | 0.00006 | 0.00003 6 | 0.39744 | 0.59982 | 0.00041 | 0.00085 | 0.00006 | 0.00003 7 | 0.39650 | 0.59791 | 0.00047 | 0.00098 | 0.00007 | 0.00003 8 | 0.39778 | 0.59840 | 0.00051 | 0.00108 | 0.00007 | 0.00003 9 | 0.39728 | 0.60099 | 0.00058 | 0.00124 | 0.00006 | 0.00003 10 | 0.39777 | 0.60040 | 0.00062 | 0.00141 | 0.00007 | 0.00003 11 | 0.39789 | 0.59776 | 0.00092 | 0.00165 | 0.00009 | 0.00003 12 | 0.39982 | 0.60014 | 0.00128 | 0.00183 | 0.00010 | 0.00004 13 | 0.39848 | 0.59920 | 0.00078 | 0.00194 | 0.00009 | 0.00004 14 | 0.40180 | 0.60289 | 0.00104 | 0.00208 | 0.00012 | 0.00006 15 | 0.40169 | 0.60277 | 0.00119 | 0.00227 | 0.00018 | 0.00009 16 | 0.39932 | 0.60070 | 0.00097 | 0.00239 | 0.00026 | 0.00015 17 | 0.40123 | 0.60026 | 0.00103 | 0.00252 | 0.00039 | 0.00021 18 | 0.40005 | 0.60127 | 0.00110 | 0.00301 | 0.00069 | 0.00035 19 | 0.40390 | 0.60192 | 0.00133 | 0.00402 | 0.00205 | 0.00062 20 | 0.40253 | 0.60495 | 0.00166 | 0.00558 | 0.00502 | 0.00116 21 | 0.40562 | 0.60948 | 0.00307 | 0.00894 | 0.02047 | 0.00582 22 | 0.41414 | 0.61430 | 0.00480 | 0.01597 | 0.04195 | 0.01122 23 | 0.42785 | 0.62686 | 0.00697 | 0.02965 | 0.08554 | 0.02237 24 | 0.45484 | 0.65614 | 0.01140 | 0.05839 | 0.17640 | 0.04577 25 | 0.51430 | 0.71481 | 0.01997 | 0.11840 | 0.35674 | 0.08991 26 | 0.63641 | 0.84100 | 0.03751 | 0.24489 | 0.72744 | 0.18124 27 | 0.89154 | 1.09585 | 0.07012 | 0.50069 | 1.46852 | 0.36282 28 | 1.41767 | 1.62144 | 0.14058 | 1.03181 | 2.97650 | 0.72449 29 | 2.50606 | 2.71355 | 0.27705 | 2.13452 | 6.02655 | 1.44101 30 | 4.78650 | 4.99729 | 0.54057 | 4.43876 | 12.16207 | 2.86434
GPU - basis-change - simulation_time=1, seed=123 nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 3 | 0.42136 | 0.63432 | 0.00154 | 0.00384 | 0.00006 | 0.00003 4 | 0.44388 | 0.64046 | 0.00295 | 0.00724 | 0.00005 | 0.00003 5 | 0.44693 | 0.65425 | 0.00447 | 0.01183 | 0.00007 | 0.00003 6 | 0.46446 | 0.66077 | 0.00675 | 0.01774 | 0.00005 | 0.00003 7 | 0.48751 | 0.66930 | 0.00906 | 0.02428 | 0.00005 | 0.00003 8 | 0.51553 | 0.68645 | 0.03146 | 0.03318 | 0.00009 | 0.00003 9 | 0.54946 | 0.70011 | 0.01465 | 0.04326 | 0.00006 | 0.00003 10 | 0.57080 | 0.71809 | 0.01913 | 0.05439 | 0.00007 | 0.00003 11 | 0.61466 | 0.74425 | 0.02236 | 0.06939 | 0.00008 | 0.00003 12 | 0.63972 | 0.76778 | 0.02895 | 0.08358 | 0.00007 | 0.00004 13 | 0.70162 | 0.79652 | 0.03206 | 0.10104 | 0.00010 | 0.00004 14 | 0.73239 | 0.82575 | 0.04071 | 0.15815 | 0.00013 | 0.00006 15 | 0.81361 | 0.85703 | 0.04453 | 0.13417 | 0.00017 | 0.00009 16 | 0.85283 | 0.89417 | 0.05302 | 0.15435 | 0.00028 | 0.00015 17 | 0.87415 | 0.93853 | 0.05550 | 0.17840 | 0.00045 | 0.00022 18 | 0.99488 | 1.11393 | 0.06979 | 0.21703 | 0.00084 | 0.00036 19 | 1.14716 | 1.21486 | 0.08634 | 0.29639 | 0.02716 | 0.00064 20 | 1.24067 | 1.38241 | 0.18828 | 0.44365 | 0.05172 | 0.00118 21 | 1.30973 | 1.55735 | 0.40242 | 0.72043 | 0.10713 | 0.00577 22 | 2.01705 | 2.17301 | 0.87639 | 1.31214 | 0.20883 | 0.01174 23 | 3.21857 | 3.43047 | 1.92488 | 2.54804 | 0.41225 | 0.02318 24 | 5.96566 | 6.11803 | 4.23166 | 5.20896 | 0.81946 | 0.04505 25 | 11.98415 | 11.90756 | 9.28966 | 10.96520 | 1.63501 | 0.09173 26 | 24.68222 | 24.30075 | 20.32023 | 23.33609 | 3.26422 | 0.18033 27 | 51.66280 | 50.96698 | 44.29223 | 49.96208 | 6.52189 | 0.35205 28 | 108.90314 | 108.06207 | 96.13157 | 107.02601 | 13.04496 | 0.71656 29 | 232.39766 | 230.31452 | 208.02524 | 229.23329 | 26.09329 | 1.43844 30 | 494.94456 | 491.45061 | 448.46891 | 490.28990 | 52.16359 | 2.89787
GPU - quantum-volume - depth=1, seed=123 nqubits | dry run (qibotf) | dry run (qibojit) | simulation (qibotf) | simulation (qibojit) | transfer (qibotf) | transfer (qibojit) -- | -- | -- | -- | -- | -- | -- 3 | 0.39581 | 0.59824 | 0.00027 | 0.00054 | 0.00005 | 0.00002 4 | 0.39759 | 0.60260 | 0.00046 | 0.00102 | 0.00005 | 0.00003 5 | 0.39701 | 0.60086 | 0.00049 | 0.00101 | 0.00006 | 0.00003 6 | 0.40060 | 0.60192 | 0.00089 | 0.00152 | 0.00009 | 0.00003 7 | 0.40071 | 0.60095 | 0.00062 | 0.00150 | 0.00006 | 0.00003 8 | 0.40138 | 0.60074 | 0.00081 | 0.00200 | 0.00008 | 0.00003 9 | 0.40297 | 0.60340 | 0.00098 | 0.00202 | 0.00007 | 0.00003 10 | 0.40330 | 0.60316 | 0.00093 | 0.00255 | 0.00006 | 0.00003 11 | 0.40090 | 0.60332 | 0.00100 | 0.00279 | 0.00006 | 0.00003 12 | 0.40373 | 0.60424 | 0.00113 | 0.00328 | 0.00007 | 0.00003 13 | 0.40603 | 0.60393 | 0.00120 | 0.00323 | 0.00009 | 0.00004 14 | 0.41030 | 0.61117 | 0.00134 | 0.00375 | 0.00011 | 0.00006 15 | 0.40559 | 0.60650 | 0.00128 | 0.00379 | 0.00017 | 0.00009 16 | 0.41114 | 0.60802 | 0.00146 | 0.00443 | 0.00035 | 0.00015 17 | 0.41326 | 0.60608 | 0.00155 | 0.00448 | 0.00039 | 0.00028 18 | 0.40834 | 0.60976 | 0.00178 | 0.00550 | 0.00068 | 0.00044 19 | 0.41195 | 0.61014 | 0.00191 | 0.00666 | 0.00227 | 0.00063 20 | 0.41138 | 0.61894 | 0.00243 | 0.00975 | 0.00648 | 0.00118 21 | 0.41901 | 0.62495 | 0.00424 | 0.01461 | 0.02356 | 0.00582 22 | 0.42691 | 0.63139 | 0.00566 | 0.02664 | 0.04962 | 0.01133 23 | 0.44633 | 0.65438 | 0.00773 | 0.04777 | 0.10100 | 0.02255 24 | 0.49657 | 0.70162 | 0.01198 | 0.09662 | 0.21145 | 0.04636 25 | 0.59168 | 0.79556 | 0.02071 | 0.18980 | 0.42626 | 0.09033 26 | 0.80050 | 1.00358 | 0.03801 | 0.39954 | 0.87644 | 0.18104 27 | 1.20193 | 1.40265 | 0.07228 | 0.80137 | 1.76744 | 0.36210 28 | 2.11122 | 2.31147 | 0.14064 | 1.71585 | 3.65570 | 0.72685 29 | 3.76538 | 3.97242 | 0.27627 | 3.38538 | 7.26726 | 1.45134 30 | 7.69035 | 7.88589 | 0.53751 | 7.31865 | 15.03615 | 2.86192

In these cases it may be useful to compare with some other libraries too, for example Cirq or qiskit for CPU or Qulacs for GPU in order to verify whether Qibo is indeed slower as quoted in the HyQuas paper.

scarrazza commented 3 years ago

@stavros11 thank you so much for all these circuits and benchmarks. It is quite reassuring to see that on CPU both simulators match, and on GPU qibotf is dominated by transfer, as expected. Do you have the json for these runs? If so could you please upload to the data folder?

stavros11 commented 3 years ago

@stavros11 thank you so much for all these circuits and benchmarks. It is quite reassuring to see that on CPU both simulators match, and on GPU qibotf is dominated by transfer, as expected.

Thanks for checking. Indeed the numbers look consistent between qibojit/qibotf but as I wrote above I will try to compare with some external libraries to to verify that the overal qibo performance remains competitive in circuits other than QFT.

Do you have the json for these runs? If so could you please upload to the data folder?

I added both the log files in data/ and the bash script I used to run these benchmarks in scripts/. I also added README files in both folders which describe all available files.

Note that for the Hidden shift and QAOA circuits the number of gates depends on the hidden bitstring / MaxCut graph respectively, so I generated some random bitstrings and graphs which I placed in a new graph/ folder to make sure that exactly the same circuits are used for all backends.

scarrazza commented 3 years ago

@stavros11 shall we enable github actions?

stavros11 commented 3 years ago

@stavros11 shall we enable github actions?

Thank you for the review. Yes, that would be great. Will you do this?