martinvandriel
commented
3 years ago Hi Aakash,
the database transform is essentially transposing a giant matrix and involves no computations, everything is IO limited and it is difficult to speed up. However, you can also use the raw database as output by axisem directly, at the cost of slower access for the seismogram extraction from file. This may be in large parts compensated by the built in buffer in instaseis, that avoids file access all together if the source location has been sampled before and this may well be the case in your application (compare figure 16 in [1], source inversion is not IO bound in that test). Depending on you machine, you can increase cache hit likelihood further by increasing the buffer size using the 'buffer_size_in_mb' option which defaults to only 100MB (https://instaseis.net/instaseis.html#open-db-function).
An experimental alternative additional tweak may be to write the database directly in a different chunking. There are two possible ways to achieve that:
- the CHUNK_TIME_TRACES and NETCDF_DUMP_BUFFER options as explained in the inparam_advanced. However, this is not well tested, so please verify the results independently.
- using serial netcdf instead of parallel by setting USE_PAR_NETCDF = false in make_axisem.macros. While this is a bit of an historic implementation of non-blocking IO using pthreads that we later replaced with the parallel version, it also leads to different chunking that is beneficial for IO along the time axis.
Hope this helps, Martin
[1] Driel, Martin van, Lion Krischer, Simon Christian Stähler, Kasra Hosseini, and Tarje Nissen-Meyer. 2015. “Instaseis: Instant Global Seismograms Based on a Broadband Waveform Database.” Solid Earth 6 (2): 701–17. https://doi.org/10.5194/se-6-701-2015.
aakash10gupta
I am trying to transform the axisem-generated databases for faster access for iterative inversion processes, as suggested by the developers. My axisem jobs are on our cluster, and I will need the database on the cluster for rapid access of millions of different sources (hypocenter + moment tensor). I have run both global-scale domains as well as localized (in depth and laterally) domains. See table below showing details of some test run times. For me, the transformation of the database is the bottleneck for our science applications.
The issue I face is that the transformation of the generated database takes many days, whereas the job to generate the database takes hours (depending on # cores). I believe the transformation of the database is a 1 core job and hence slow (and will depend on the machine). Please let me know if you have advice. Partly I also wanted to share these tests with others.