Forced fitting slow on nimbus

[ddobie]

Earlier today (2022-04-05 00:50 UTC) ran identical code on both nimbus and ada, running a simple query with and without forced fitting.

from vasttools.query import Query
from astropy.coordinates import SkyCoord, Angle
import astropy.units as u
from vasttools.utils import get_logger

from timeit import default_timer as timer

if __name__ == '__main__':
    t0 = timer()
    example_query = Query(
        coords=SkyCoord(['17:23:23.18'], ['-28:37:57.17'], unit=(u.hourangle, u.deg)),
        crossmatch_radius=5.,
        epochs='1',
    )
    t1 = timer()
    example_query.find_fields()
    t2 = timer()
    example_query.find_sources()
    t3 = timer()

    print(t1-t0)
    print(t2-t1)
    print(t3-t2)

    t0 = timer()
    example_query = Query(
        coords=SkyCoord(['17:23:23.18'], ['-28:37:57.17'], unit=(u.hourangle, u.deg)),
        crossmatch_radius=5.,
        epochs='1',
        forced_fits=True
    )
    t1 = timer()
    example_query.find_fields()
    t2 = timer()
    example_query.find_sources()
    t3 = timer()

    print(t1-t0)
    print(t2-t1)
    print(t3-t2)

On ada I get fairly reasonable numbers, with find_sources taking 6s without forced fitting and 12s with it. On nimbus (in a notebook) I get 6s without forced fitting and 50s with it.

Last night I ran an actual query for the same source and all epochs with forced fitting that took 10+ minutes. I wasn't timing the code, but even without forced fitting it was still taking a few minutes. I ran top from terminal and it didn't look like there were any other jobs running.

It's not clear to me what the issue could be, nor even how to begin debugging it.

[ddobie]

This is an issue with our IOPS being limited on nimbus. We have been upgraded to the highest IOPS that Pawsey will allow, so things will continue to be slower than ideal, but should be tolerable.