Chunksize manual benchmarking notebook for equilibrium spectra added

[sagarchotalia]

Added a notebook for benchmarking of chunksize vs. non-chunksize computations, plotting the results.

Changes Made

• Computed and compared an NO equilibrium spectrum, with ~53k lines.
• Calculated and showed residual, plot_diff
• Computed the same spectra multiple times for statistical fluctuations, comparing it in the end using the no-chunksize spectrum as a baseline.

Future Objectives

• Add the comparison with chunksize = "auto" feature, once it is implemented (see the Chunksize PR)
• Add non-equilibrium spectra calculations in the same benchmark.

[erwanp]

Saw your notebook, great! https://github.com/sagarchotalia/radis-benchmark/blob/master/manual_benchmarks/chunksize_benchmark.ipynb

Can you somehow output the number of chunks ? I see there are only ~50,000 lines in the NO example, which is a small number, so it's possible that beyond chunksize=1e5 we're having all lines in a single chunk.

[sagarchotalia]

Yes, so I included the "N" calculation parameter, here are the results:

• For the case of the molecule NO, the temporary chunk dataframe dg has a size of 1 until chunksize = 1e6, since the N parameter gets larger than df1.
• Starting at chunksize = 1e6, the sizes of the chunks in memory are:
  • 8, 83, 828, and 7572, corresponding to 1e6, 1e7, 1e8 and 1e9 respectively. Starting 1e10, the value of N becomes 1, hence the entire dataframe is considered at once.