get_spectrum loading on laptop vs remote computer

[gkhullar]

Hi,

I am using python-fsps with prospector to fit SFH models to optical and IR galaxy spectra, and am running into some runtime issues. I am using the same code on my laptop vs a remote unix computer (accessing via ssh) and fsps outputs are generated 5 times slower on the remote computer than on my laptop.

To demonstrate this, I ran the following code with python-fsps (versions of FSPS and python-fsps match on both the laptop and remote computer):

import fsps
import time

tstart = time.time()
sp = fsps.StellarPopulation(compute_vega_mags=False, zcontinuous=1,
                                sfh=0, logzsol=0.0, dust_type=2, dust2=0.2)
time2 = time.time()
print("time to load fsps.stellarpop is",time2 - tstart)

wave, spec = sp.get_spectrum(tage=13.7)
time3 = time.time()
print("time to load spectrum from sp object is",time3 - time2)

print("TOTAL TIME is",time3 - tstart)

The output looks like the following:

a) For Laptop (MacOS High Sierra 10.13.6; Processor: 2.7 GHz Intel Core i7, Memory: 16 GB 2133 MHz LPDDR3): Python 3.7.1, IPython 6.5.0, python_fsps v0.3.0

time to load fsps.stellarpop is 13s time to load spectrum from sp object is 58s TOTAL TIME is 71s

b) For remote computer (comparable : 64 processors w 2 GHz each) Python 3.6.8, IPython 7.4.0, python_fsps v0.3.0

time to load fsps.stellarpop is 67s time to load spectrum from sp object is 314s TOTAL TIME is 381s

I suspected there may be something going on with FSPS or versions mismatch, but could not find a solution here. The root cause here may be similar to #108. I am also assuming that ~60s on the laptop is not a problem, which may not be true.

[bd-j]

The laptop times you report are similar to what I get on my laptop for MIST isochrones and fsps compiled with gcc 9.1.0. I don't know why the remote computer should be so much slower. Note the number of cores does not matter, since the underlying fortran being called during setup and initial SSP generation is not multithreaded.

Also it is important to note that subsequent calls to get_spectrum() should be much faster. In the first call the SSPs for the relevant metallicities are built in a time-consuming process, but they are cached for future use. As long as you aren't changing any parameter listed in sp.params.ssp_params the calls to get_spectrum should go to a few milliseconds (without nebular emission). Though if you change the metallicity a lot then new SSPs will need to be built and cached (the old ones will remain in the cache) so the first call for a very different metallicity will also take ~60s.

e.g. in ipython

import numpy as np
 %timeit  wave, spec = sp.get_spectrum(tage=np.random.uniform(0.1, 13.7))    

gives 2 ms call times on my laptop.

[gkhullar]

Thanks! I believe that since I am generating CSPs in prospector, call times for my remote computer are still slower by a factor of 5-6 (when fitting with emcee). I suspect the core problem here would then have to do with the underlying fortran code.

If this is an FSPS or prospector issue, is it better to post the issue in the respective repos? I am unfortunately not super familiar with fortran, and am happy to provide any outputs/screenshots to solve this, if possible.

[bd-j]

Sorry, I can't devote time to diagnosing slowdowns on a particular computer environment or architecture. I would suggest checking the fsps makefiles and compiler version used on the remote machine, and, if possible, matching python environments between the laptop and remote machine. You might also try timing some other codes on the two machines.

[gkhullar]

Thanks, this is helpful. I will try these suggestions!

[bd-j]

assuming resolved and closing.