search

imcgreer / simqso

Module for generating simulated quasar spectra
BSD 3-Clause "New" or "Revised" License
12 stars 11 forks source link

add parallelization #7

Open imcgreer opened 7 years ago

imcgreer commented 7 years ago

simulating 96,000 quasars in a single process:

               stage     time  elapsed     frac
     Initialize Grid    0.029    0.029    0.000
     Generate Forest    0.013    0.042    0.000
Build Quasar Spectra  338.536  338.578    0.999
            PhotoMap    0.303  338.881    0.001
              Finish    0.000  338.881    0.000

using 7 processes (on 8-core machine):

               stage     time  elapsed     frac
     Initialize Grid    0.030    0.030    0.000
     Generate Forest    0.013    0.043    0.000
Build Quasar Spectra  194.856  194.899    0.998
            PhotoMap    0.300  195.200    0.002
              Finish    0.000  195.200    0.000

after removing samplers from variables to reduce pickling overhead:

               stage     time  elapsed     frac
     Initialize Grid    0.029    0.029    0.000
     Generate Forest    0.015    0.044    0.000
Build Quasar Spectra  125.248  125.293    0.997
            PhotoMap    0.299  125.591    0.002
              Finish    0.000  125.591    0.000
imcgreer commented 7 years ago

This is working in version 1.1, but suffers from bad memory leakage.