BUG: HEASARC query_region to parse image responses

[jkrick]

This is similar to the original issue in #1637 but that issue has been closed, so not sure exactly. I am trying to run the following query and get the below error. I think the problem is that there is no match to the query, but there should be a real response for that, and not an error. I don't even know why it is expecting HDUs. Pip says I am running astroquery 0.4.5

from astroquery.ipac.ned import Ned
from astropy.coordinates import SkyCoord
from astroquery.heasarc import Heasarc
from astropy import units as u

CLAGN = Ned.query_refcode('2018ApJ...862..109Y')

coords_list = [
    SkyCoord(ra, dec, frame='icrs', unit='deg')
    for ra, dec in zip(CLAGN['RA'], CLAGN['DEC'])
]

c = 1 
heasarc = Heasarc()
mission = 'fermilpsc'
radius = 0.1*u.degree
results = heasarc.query_region(coords_list[c], mission = mission, radius = radius)#, 

but I get the following error:

ValueError Traceback (most recent call last) /opt/conda/lib/python3.8/site-packages/astroquery/heasarc/core.py in _parse_result(self, response, verbose) 241 data = BytesIO(response.content) --> 242 table = Table.read(data, hdu=1) 243 return table

/opt/conda/lib/python3.8/site-packages/astropy/table/connect.py in call(self, *args, *kwargs) 61 ---> 62 out = self.registry.read(cls, args, **kwargs) 63

/opt/conda/lib/python3.8/site-packages/astropy/io/registry/core.py in read(self, cls, format, cache, *args, *kwargs) 198 reader = self.get_reader(format, cls) --> 199 data = reader(args, **kwargs) 200

/opt/conda/lib/python3.8/site-packages/astropy/io/fits/connect.py in read_table_fits(input, hdu, astropy_native, memmap, character_as_bytes) 213 try: --> 214 return read_table_fits(hdulist, hdu=hdu, 215 astropy_native=astropy_native)

/opt/conda/lib/python3.8/site-packages/astropy/io/fits/connect.py in read_table_fits(input, hdu, astropy_native, memmap, character_as_bytes) 201 else: --> 202 raise ValueError("No table found") 203

ValueError: No table found

During handling of the above exception, another exception occurred:

TypeError Traceback (most recent call last) /opt/conda/lib/python3.8/site-packages/astroquery/heasarc/core.py in _parse_result(self, response, verbose) 245 try: --> 246 return self._fallback(response.text) 247 except Exception as e:

/opt/conda/lib/python3.8/site-packages/astroquery/heasarc/core.py in _fallback(self, text) 199 data = StringIO(text) --> 200 header = fits.getheader(data, 1) # Get header for column info 201 colstart = [y for x, y in header.items() if "TBCOL" in x]

/opt/conda/lib/python3.8/site-packages/astropy/io/fits/convenience.py in getheader(filename, *args, *kwargs) 108 mode, closed = _get_file_mode(filename) --> 109 hdulist, extidx = _getext(filename, mode, args, **kwargs) 110 try:

/opt/conda/lib/python3.8/site-packages/astropy/io/fits/convenience.py in _getext(filename, mode, ext, extname, extver, *args, kwargs) 1094 -> 1095 hdulist = fitsopen(filename, mode=mode, kwargs) 1096

/opt/conda/lib/python3.8/site-packages/astropy/io/fits/hdu/hdulist.py in fitsopen(name, mode, memmap, save_backup, cache, lazy_load_hdus, ignore_missing_simple, kwargs) 175 --> 176 return HDUList.fromfile(name, mode, memmap, save_backup, cache, 177 lazy_load_hdus, ignore_missing_simple, kwargs)

/opt/conda/lib/python3.8/site-packages/astropy/io/fits/hdu/hdulist.py in fromfile(cls, fileobj, mode, memmap, save_backup, cache, lazy_load_hdus, ignore_missing_simple, **kwargs) 410 --> 411 return cls._readfrom(fileobj=fileobj, mode=mode, memmap=memmap, 412 save_backup=save_backup, cache=cache,

/opt/conda/lib/python3.8/site-packages/astropy/io/fits/hdu/hdulist.py in _readfrom(cls, fileobj, data, mode, memmap, cache, lazy_load_hdus, ignore_missing_simple, *kwargs) 1100 # Check the SIMPLE card is there but not written correctly -> 1101 match_sig_relaxed = re.match(rb"SIMPLE\s=\s*[T|F]", simple) 1102

/opt/conda/lib/python3.8/re.py in match(pattern, string, flags) 190 a Match object, or None if no match was found.""" --> 191 return _compile(pattern, flags).match(string) 192

TypeError: cannot use a bytes pattern on a string-like object

During handling of the above exception, another exception occurred:

AttributeError Traceback (most recent call last)

in 5 mission = 'fermilpsc' 6 radius = 0.1*u.degree ----> 7 results = heasarc.query_region(coords_list[c], mission = mission, radius = 0.1*u.degree)#, 8 # fields = 'RA, DEC, ERROR_RADIUS, TRIGGER_TIME, TRIGGER_TYPE, RELIABILITY, LOCALIZATION_SOURCE') 9 #from astropy import units as u /opt/conda/lib/python3.8/site-packages/astroquery/utils/class_or_instance.py in f(*args, **kwds) 23 def f(*args, **kwds): 24 if obj is not None: ---> 25 return self.fn(obj, *args, **kwds) 26 else: 27 return self.fn(cls, *args, **kwds) /opt/conda/lib/python3.8/site-packages/astroquery/utils/process_asyncs.py in newmethod(self, *args, **kwargs) 27 if kwargs.get('get_query_payload') or kwargs.get('field_help'): 28 return response ---> 29 result = self._parse_result(response, verbose=verbose) 30 self.table = result 31 return result /opt/conda/lib/python3.8/site-packages/astroquery/heasarc/core.py in _parse_result(self, response, verbose) 246 return self._fallback(response.text) 247 except Exception as e: --> 248 return self._old_w3query_fallback(response.content) 249 250 def _args_to_payload(self, **kwargs): /opt/conda/lib/python3.8/site-packages/astroquery/heasarc/core.py in _old_w3query_fallback(self, content) 179 f = fits.open(BytesIO(content)) 180 --> 181 for c in f[1].columns: 182 if c.disp is not None: 183 c.format = c.disp AttributeError: 'ImageHDU' object has no attribute 'columns'

[caseyjlaw]

I'm also getting that error for the mission='nvss'. I'm running astroquery 0.4.6 and astropy 5.1.

[keflavich]

This looks like the HEASARC returns are of the wrong type. I'll dig a bit.

[keflavich]

OK, the problem is straightforward but a bit weird: when HEASARC gets no results, it returns a FITS file with a blank ImageHDU instead of a blank BinTableHDU:

In [27]: fits.open('/home/adam/Downloads/browse_results.fits')
Out[27]: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7fc051f78b50>, <astropy.io.fits.hdu.image.ImageHDU object at 0x7fbfc8f9b370>]

In [28]: fits.open('/home/adam/Downloads/browse_results.fits')[0]
Out[28]: <astropy.io.fits.hdu.image.PrimaryHDU at 0x7fc051b9c850>

In [29]: fits.open('/home/adam/Downloads/browse_results.fits')[0].data

In [30]: fits.open('/home/adam/Downloads/browse_results.fits')[0].header
Out[30]:
SIMPLE  =                    T / Browse (FitsDisplay V0.2)
BITPIX  =                    8
NAXIS   =                    0 / Dummy HDU
EXTEND  =                    T / Extensions allowed

In [31]: fits.open('/home/adam/Downloads/browse_results.fits')[1].header
Out[31]:
XTENSION= 'IMAGE   '           / Tables not returning data
BITPIX  =                    8
NAXIS   =                    0
PCOUNT  =                    0
GCOUNT  =                    1

COMMENT    heasarc_fermigbrst : No matching rows
COMMENT    heasarc_fermigtrig : No matching rows
COMMENT    heasarc_fermille : No matching rows
COMMENT    heasarc_fermilasp : No matching rows
COMMENT    heasarc_fermilpsc : No matching rows
COMMENT    heasarc_fermi2favs : No matching rows
COMMENT    heasarc_fermiltrns : No matching rows
COMMENT    heasarc_fermifhl : No matching rows
COMMENT    heasarc_fermilac : No matching rows
COMMENT    heasarc_fermilatra : No matching rows
COMMENT    heasarc_fermilgrb : No matching rows
COMMENT    heasarc_fermilbsl : No matching rows
COMMENT    heasarc_fermi3fhl : No matching rows
COMMENT    heasarc_fer2fusrid : No matching rows
COMMENT    heasarc_fermil2psr : No matching rows
COMMENT    heasarc_wisehspcat : No matching rows
COMMENT    heasarc_fermilhesc : No matching rows
COMMENT    heasarc_at20g1fgl : No matching rows
COMMENT    heasarc_fermi3fgl : No matching rows
COMMENT    heasarc_fermigdays : Error in query (cone search not supported?)
COMMENT    heasarc_fermilweek : Error in query (cone search not supported?)
COMMENT    heasarc_fermigsol : Error in query (cone search not supported?)

This is a pretty easy fix. For now, you could work around this by:

try:
    heasarc.query_region(...)
except AttributeError:
    print("No result found")

for example

[bsipocz]

OK, I'm cross linking https://github.com/astropy/astroquery/issues/1838, we should put some efforts towards fixing this for all the modules.

[keflavich]

I'm working on this specific one - I was going to have it return an empty Table with some metadata in the header about the failure. What are we unifying to, though? (continue this conversation in #1838)

[keflavich]

I am a little confused about why HEASARC seems to have so many different ways to say "I didn't find anything."

[keflavich]

By doing a text-only search, I discovered that it's because HEASARC is trying to summarize that there are many different tables being searched all with null results:

-----------------------------------------------------------
No matches for:
     Fermi GBM Burst Catalog (fermigbrst)
     Fermi GBM Trigger Catalog (fermigtrig)
     Fermi LAT Low-Energy Events Catalog (fermille)
     Fermi LAT Monitored Source List (fermilasp)
     Fermi LAT 12-Year Point Source Catalog (4FGL-DR3) (fermilpsc)
     Fermi LAT Second Catalog of Gamma-Ray Pulsars (2PC) (fermil2psr)
     Second Catalog of Hard Fermi-LAT Sources (2FHL) (fermifhl)
     Third Catalog of Hard Fermi-LAT Sources (3FHL) (fermi3fhl)
     Fermi LAT Long-Term Transient Source Catalog (fermiltrns)
     Fermi LAT Second Gamma-Ray Burst Catalog (fermilgrb)
     Fermi LAT Bright Source List (fermilbsl)
     Fermi 2FGL Unassociated Gamma-Ray Sources Possible Radio Identifications (fer2fusrid)
     2WHSP Catalog of High Synchrotron Peaked Blazars and Candidates (wisehspcat)
     Fermi LAT Sources Refined Associations Catalog (fermilatra)
     Fermi LAT Second AGN Catalog (fermilac)
     Fermi All-Sky Variability Analysis Second Catalog of Flaring Gamma-Ray Sources (fermi2favs)
     Fermi LAT High-Energy Source Catalog (1FHL) (fermilhesc)
     AT20G/Fermi 1FGL Source Catalog (at20g1fgl)
     Fermi LAT 4-Year Point Source Catalog (fermi3fgl)

------------------------------------------------------------
Search errors
     Table does not support coordinate searches: Fermi GBM Daily Data (fermigdays)
     Table does not support coordinate searches: Fermi LAT Weekly Data (fermilweek)
     Table does not support coordinate searches: Fermi GBM Solar Flare Catalog (fermigsol)

This is going to be conveyed in the returned metadata in #2624.

[bsipocz]

OK, the problem is straightforward but a bit weird: when HEASARC gets no results, it returns a FITS file with a blank ImageHDU instead of a blank BinTableHDU:

I would suggest someone upstream this weirdness/problem to HEASARC though their official channels (we don't have a direct contact with astroquery), otherwise we have a workaround now. Thanks all!