Problem solving SExtractor-generated list of stars

[kirxkirx]

I'm getting the following error when trying to run solve-field on SExtractor-generated source list:

$ solve-field --objs 1000 --depth 10,20,30-50 --overwrite --no-plots --x-column X_IMAGE --y-column Y_IMAGE --sort-column FLUX_APER --width 3352 --height 2532 --scale-units arcminwidth --scale-low 315.0 --scale-high 560.0 out908.xyls
Reading input file 1 of 1: "out908.xyls"...
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/astropy/io/fits/hdu/hdulist.py", line 1157, in _try_while_unread_hdus
    return func(*args, **kwargs)
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/astrometry/bin/removelines", line 15, in <module>
    sys.exit(main())
  File "/usr/local/astrometry/lib/python/astrometry/util/removelines.py", line 77, in main
    return removelines(infile, outfile, xcol=opt.xcol, ycol=opt.ycol,
  File "/usr/local/astrometry/lib/python/astrometry/util/removelines.py", line 40, in removelines
    T = fits_table(infile, lower=False)
  File "/usr/local/astrometry/lib/python/astrometry/util/fits.py", line 675, in fits_table
    data = pf[hdunum].data
  File "/usr/lib/python3.8/site-packages/astropy/io/fits/hdu/hdulist.py", line 322, in __getitem__
    return self._try_while_unread_hdus(super().__getitem__,
  File "/usr/lib/python3.8/site-packages/astropy/io/fits/hdu/hdulist.py", line 1159, in _try_while_unread_hdus
    if self._read_next_hdu():
  File "/usr/lib/python3.8/site-packages/astropy/io/fits/hdu/hdulist.py", line 1200, in _read_next_hdu
    hdu = _BaseHDU.readfrom(fileobj, **kwargs)
  File "/usr/lib/python3.8/site-packages/astropy/io/fits/hdu/base.py", line 332, in readfrom
    hdu = cls._readfrom_internal(fileobj, checksum=checksum,
  File "/usr/lib/python3.8/site-packages/astropy/io/fits/hdu/base.py", line 475, in _readfrom_internal
    hdu = cls(data=DELAYED, header=header, **new_kwargs)
  File "/usr/lib/python3.8/site-packages/astropy/io/fits/hdu/table.py", line 855, in __init__
    from astropy.table import Table
  File "/usr/lib/python3.8/site-packages/astropy/table/__init__.py", line 4, in <module>
    from .column import Column, MaskedColumn, StringTruncateWarning, ColumnInfo
  File "/usr/lib/python3.8/site-packages/astropy/table/column.py", line 12, in <module>
    from astropy.units import Unit, Quantity
  File "/usr/lib/python3.8/site-packages/astropy/units/__init__.py", line 17, in <module>
    from .quantity import *
  File "/usr/lib/python3.8/site-packages/astropy/units/quantity.py", line 32, in <module>
    from .quantity_helper import (converters_and_unit, can_have_arbitrary_unit,
  File "/usr/lib/python3.8/site-packages/astropy/units/quantity_helper/__init__.py", line 10, in <module>
    from . import helpers, function_helpers
  File "/usr/lib/python3.8/site-packages/astropy/units/quantity_helper/function_helpers.py", line 116, in <module>
    np.asscalar, np.alen,
  File "/home/kirx/.local/lib/python3.8/site-packages/numpy/__init__.py", line 311, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'asscalar'
augment-xylist.c:591:backtick Failed to run command: /usr/local/astrometry/bin/removelines -X X_IMAGE -Y Y_IMAGE out908.xyls /tmp/tmp.removelines.tkvnS3
 ioutils.c:568:run_command_get_outputs Command failed: return value 1
$ solve-field --version
0.93
$ echo "import astropy; print(astropy.__version__); print(astropy.__path__)" | python
5.0
['/usr/lib/python3.8/site-packages/astropy']
$ echo "import numpy; print(numpy.__version__); print(numpy.__path__)" | python
1.23.5
['/home/kirx/.local/lib/python3.8/site-packages/numpy']
$

The FITS-table with the source list and a BASH script that runs the above commands may be found in this archive: astrometry_net_test.zip

The problem with my scripts using astrometry.net solver appeared after a system upgrade which (I think) pulled new versions of both astropy and numpy. I was hoping that recompiling astrometry.net using the latest version from github may solve the issue but it persists. Would appreciate any suggestions on how to fix this.

I know that I should probably stop using system package manager to install python-related stuff and fully switch to pip or conda instead.

[kirxkirx]

Just realized that above I have a local copy of numpy. Checked that the problem persists with both system-wide installation of numpy 1.23.5 and the local (user) installation upgraded to numpy 1.24.1.

[dstndstn]

Thanks for the bug report! This looks like an astropy/numpy interaction to me. Could you please try opening the file with astropy and see if you still get this bug -- that is, remove Astrometry.net from the equation? Eg something like

python -c "from astropy.table import Table; Table.read('out908.xyls', format='fits')"

?

Another thing you could try would be to install fitsio to read FITS files (pip install fitsio), instead of astropy.

[kirxkirx]

Thanks for the quick reply and the test suggestion!

Yes, it turns out from astropy.table import Table fails:

$ python
Python 3.8.16 (default, Dec 15 2022, 15:50:21) 
[GCC 11.2.1 20211127] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from astropy.table import Table
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/astropy/table/__init__.py", line 4, in <module>
    from .column import Column, MaskedColumn, StringTruncateWarning, ColumnInfo
  File "/usr/lib/python3.8/site-packages/astropy/table/column.py", line 12, in <module>
    from astropy.units import Unit, Quantity
  File "/usr/lib/python3.8/site-packages/astropy/units/__init__.py", line 17, in <module>
    from .quantity import *
  File "/usr/lib/python3.8/site-packages/astropy/units/quantity.py", line 32, in <module>
    from .quantity_helper import (converters_and_unit, can_have_arbitrary_unit,
  File "/usr/lib/python3.8/site-packages/astropy/units/quantity_helper/__init__.py", line 10, in <module>
    from . import helpers, function_helpers
  File "/usr/lib/python3.8/site-packages/astropy/units/quantity_helper/function_helpers.py", line 116, in <module>
    np.asscalar, np.alen,
  File "/usr/lib/python3.8/site-packages/numpy/__init__.py", line 311, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'asscalar'
>>> 

while other parts of astropy (like reading FITS images) seems to work fine. So this is an astropy issue not directly related to astrometry.net. I'm sorry for the confusion.

[dstndstn]

No problem! Thanks for filing a detailed bug report, makes it much easier to see what's going on!