astronomy-commons / axs

Astronomy eXtensions for Spark: Fast, Scalable, Analytics of Billion+ row catalogs
https://axs.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
23 stars 12 forks source link

issue with AXA optimization for catalogue cross-matches. #23

Open AltieriBruno opened 4 years ago

AltieriBruno commented 4 years ago

This part of the code (below) for the optimisation of the AXS crossmatches will not work for regions close to the equatorial poles, as for the window around ra the r (cross correlation radios) should be divided by cos(dec), otherwise the preselected region will be smaller than r along the right ascension axis as the distance along the right ascension is delta(ra)*cos(dec).

val join = if (useSMJOptim) df1.join(df2, df1("zone") === df2("zone") and (df1("ra") between(df2("ra") - r, df2("ra") + r))) else df1.join(df2, df1("zone") === df2("zone") and (df1("ra") between(df2("ra") - r, df2("ra") + r)) and (df1("dec") between(df2("dec") - r, df2("dec") + r)))

The current implementation leads to a loss of cross-correlated sources at high latitudes if optimisation in enabled.

stargaser commented 4 years ago

We see a drop in source density when using AXS to crossmatch AllWISE and Gaia DR2 in Galactic coordinates.

Here is sample code we ran on the epyc system, and the 2-D histogram.

allwise = axs_catalog.load("allwise_1am_dup")
gaia = axs_catalog.load("gaia_dr2_1am_dup")
allwise_gaia = (gaia.crossmatch(allwise).selectExpr(
           "ra", "dec", "source_id", "pmra", "pmdec", "parallax",
           "phot_g_mean_mag as G", "w1mpro", "w2mpro", "ecl_lon",
           "ecl_lat", "l", "b"))
image_histogram = allwise_gaia.histogram2d(
             allwise_gaia['l'], allwise_gaia['b'],
             360, 180, min1=0, max1=360, min2=-90, max2=90)
x,y,img = image_histogram
extent = (0, 360, -90, 90)
matplotlib.rcParams['figure.dpi'] = 150
plt.imshow(img.T, origin='lower', extent=extent,
      interpolation=None, vmax=0.2*np.max(img))
plt.xlabel("Galactic longitude [deg]")
plt.ylabel("Galactic latitude [deg]")
plt.title('Gaia DR2 xmatched with AllWISE')
plt.savefig('gaia_allwise_galactic.png');

gaia_allwise_galactic