Binary size

[michael-a-hansen]

I've found that the binary generated by pyjac through cython is quite large for bigger mechanisms (pushing towards 1000s of species). See data below, from 11 to 654 species. The plot shows quadratic scaling with species count and somewhere between linear and quadratic with reaction count, and that pyjac's binary will cross 1 GB somewhere around ~1000 species or ~10000 reactions.

Are there any plans to work on reducing binary size, notably for really, really big mechanisms above 1000 species? My bet is that the only way through this could be fine-tuning the auto-generated code more or reducing what gets auto-generated (a 'middle ground' that uses loops and conditionals for some types of calculations while auto-generating unrolled code for others).

Mechanism	No. species	No. reactions	Binary size (MB)
Hydrogen, Burke et al.	11	27	0.2
Biodiesel, Luo et al.	115	659	10
Heptane, Mehl et al.	654	4846	271

[michael-a-hansen]

FYI, it definitely looks like the dependency is stronger on reaction count than species count, which makes sense. Below are two similar mechanisms, with nearly identical species counts, but a factor of 1.7 between the reaction counts, which happens to match the factor of 1.7 between the binary sizes.

Mechanism	No. species	No. reactions	Binary size (MB)
Biodiesel, Luo et al. low-T	115	659	10
Biodiesel, Luo et al. high-T	118	1148	17

[michael-a-hansen]

I compiled a 1034-species mechanism (PRF, Curran et al. from the Livermore website, with 7558 reactions) and the binary is 656 MB. Fun fact: the generated Jacobian was 30.6 million lines of code, which were split into 807 files for compilation!

I wonder how compiler-dependent this might be. These binaries are from gcc on my mac (latest OS, clang, etc.).

[skyreflectedinmirrors]

@michael-a-hansen you are absolutely correct that the size of the generated Jacobian library is massive -- to the point where we had to include code that would split the Jacobian evaluation into separate files so the compilers wouldn't break during compilation (gcc 4.4.7 used to segfault for big enough mechanisms).

The new version of pyJac will:

a) be sparse, and won't have to update (nearly) every single species for every reaction b) won't be hard coded (and therefore will be much smaller)

As you've noticed the generated Jacobians get huge for large mechanisms as a result of these two reasons. Is the size of the binary an issue other than the sheer ridiculousness of it?

Sidenote, I'm just about to begin validation testing so I should be able to give you a ballpark binary size for similar mechanisms in a day or two

[skyreflectedinmirrors]

Also, you might look into the -Os flag which tells GCC to try and minimize the size of the developed object, it would need to be added to libgen.py in a from source build

[michael-a-hansen]

I eagerly await the new version! Let me know if I can help out in testing.

What do you mean by "won't be hard coded"?

[skyreflectedinmirrors]

@michael-a-hansen it will look much more like a loop / conditional based code (with options to unroll). This direction was chosen to avoid the huge filesizes, as well as to enable opencl vectorization

[michael-a-hansen]

Terrific! I'm very interested to see how various degrees of unrolling impact performance and memory across mechanism size. Thanks for the good work.