search

astropy / astroquery

Functions and classes to access online data resources. Maintainers: @keflavich and @bsipocz and @ceb8
http://astroquery.readthedocs.org/en/latest/
BSD 3-Clause "New" or "Revised" License
706 stars 399 forks source link

XMatch.query missing cat1 when usign two previously uploaded tables #3115

Closed masgura closed 1 month ago

masgura commented 1 month ago

Hi, I'm trying to match two previously uploaded catalogue this way:

radec_mid = SkyCoord(a_mid, d_mid, unit=(u.deg, u.deg))
columns = ['Gmag', 'BP-RP', 'RAJ2000', 'DEJ2000', 'ra', 'dec', 'pmra', 'mpdec', 'source_id']
custom_visizer = Vizier(columns=columns)
custom_visizer.ROW_LIMIT = -1

astropy.utils.data.clear_download_cache(hashorurl=None, pkgname='astropy')

stars_1 = custom_visizer.query_region(radec_mid, radius=self.telescope.fov, catalog='I/355/gaiadr3',
                                    **kwargs)
stars_2 = custom_visizer.query_region(radec_mid, radius=self.telescope.fov, catalog='I/350/gaiaedr3',
                                      **kwargs)

stars = XMatch.query(cat1=stars_1[0], cat2=stars_2[0], max_distance=1 * u.arcsec, colRA1='RAJ2000',
                     colDec1='DEJ2000', colRA2='RAJ2000', colDec2='RAJ2000')

I checked that the two tables are correctly uploaded, but the XMatch.query function gives me this error:

Message: Missing parameter "cat1".
                 Trace:
cds.xmatch.exception.CmdParseException: Missing parameter "cat1".
        at cds.xmatch.servlet.SyncXMatch.exec(SyncXMatch.java:656)
        at cds.xmatch.servlet.SyncXMatch.doPost(SyncXMatch.java:476)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:498)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
        at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2486)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:844)

But, then I do it this way:

stars = XMatch.query(cat1=stars_1[0], cat2='vizier:I/350/gaiaedr3, max_distance=1 * u.arcsec, 
                    colRA1='RAJ2000', colDec1='DEJ2000' )

Everything works fine

ManonMarchand commented 1 month ago

I can reproduce. But why don't you directly crossmatch the online versions of the catalogs? Is there something that does not allow you to use the second version?

Notes (MRE + payload)

MRE: ```python from astropy.coordinates import SkyCoord from astroquery.vizier import Vizier from astroquery.xmatch import XMatch import astropy.units as u a_mid = [0, 1, 2] d_mid = [0, 1, 2] radec_mid = SkyCoord(a_mid, d_mid, unit=(u.deg, u.deg)) columns = ['Gmag', 'BP-RP', 'RAJ2000', 'DEJ2000', 'ra', 'dec', 'pmra', 'pmdec', 'source_id'] custom_visizer = Vizier(columns=columns) stars_1 = custom_visizer.query_region(radec_mid, radius=1 * u.arcmin, catalog='I/355/gaiadr3') stars_2 = custom_visizer.query_region(radec_mid, radius=1 * u.arcmin, catalog='I/350/gaiaedr3') stars = XMatch.query(cat1=stars_1[0], cat2=stars_2[0], max_distance=1 * u.arcsec, colRA1='RAJ2000',colDec1='DEJ2000', colRA2='RAJ2000', colDec2='DEJ2000', get_query_payload=True) ``` payload ```json {'request': 'xmatch', 'distMaxArcsec': 1.0, 'RESPONSEFORMAT': 'votable', 'colRA1': 'RAJ2000', 'colDec1': 'DEJ2000', 'colRA2': 'RAJ2000', 'colDec2': 'RAJ2000', 'area': 'allsky'}, {'files': {'cat2': ('cat1.csv', '_q,Gmag,BP-RP,RAJ2000,DEJ2000,RA_ICRS,DE_ICRS,pmRA,Source\n1,19.143797,2.243973,0.00946094931,-0.00079864601,0.00943691398,-0.00089684879,-5.408,2546034966433885568\n1,15.303568,1.211627,359.98957129186,0.01231070377,359.98956482804,0.01221930842,-1.454,2738188646457069696\n2,17.732948,1.480492,0.99561091719,0.98872594776,0.99561768899,0.98866391680,1.523,2738366075850046848\n2,18.970488,0.706100,1.00424618364,0.99990676310,1.00424775988,0.99986963242,0.355,2738366110209789440\n2,21.569563,1.293295,0.99109555451,1.00262549104,0.99109555451,1.00262549104,,2738369030787551616\n2,19.180887,0.893375,0.99873735124,1.01248301258,0.99873885341,1.01243455415,0.338,2738369065147293312\n2,19.353508,1.927811,0.99153877353,1.01194957128,0.99153566149,1.01189465689,-0.700,2738369133866769664\n3,19.884580,2.667747,2.01246088493,2.00572405014,2.01246672429,2.00574515931,1.313,2738898823593708032\n3,17.465452,1.530697,2.00085284723,2.01526896679,2.00087905030,2.01524154494,5.892,2738910574624232320\n3,21.072521,0.654532,1.99263015486,2.01139934726,1.99263015486,2.01139934726,,2738910608984947200\n')} ```

masgura commented 1 month ago

Since I need only some columns, I noted that uploading the catalogue by column names is way faster than using the whole catalogue (7 seconds compared to 1/2 minutes), so I was trying to check if uploading both + matching them is faster than uploading the first and matching it with the online one (which takes about 25-30 seconds). Matching the two online versions is waay slower

ManonMarchand commented 1 month ago

Were you able to try again? It should be fixed in the development version, see here to see how to install it: Building from source

masgura commented 3 weeks ago

Sorry for the late replay, everything works fine now, thanks!