Allocation overflow

[kevinss5]

EDIT: Apologizes, I misread the code. The area of code I pointed to before I think is fine. Sorry about that! Though, there is a memory issue somewhere in the C code, I think.

99% of the time solving has no issues, but occasionally there is a malloc error (overflow maybe). This seems to be related to the more crowded fields.

Length = 1686 rows #len of star sources
INFO:root:loaded 48 index files
5200 scale: 5
INFO:root:solve 1: start
...
INFO:root:solve 1: slice=[900, 925[ (37 / 67), index="5200/index-5205-13.fits" (1 / 1)
INFO:root:solve 1: slice=[925, 950[ (38 / 67), index="5200/index-5205-13.fits" (1 / 1)
INFO:root:solve 1: slice=[950, 975[ (39 / 67), index="5200/index-5205-13.fits" (1 / 1)
INFO:root:solve 1: slice=[975, 1000[ (40 / 67), index="5200/index-5205-13.fits" (1 / 1)
INFO:root:solve 1: slice=[1000, 1025[ (41 / 67), index="5200/index-5205-13.fits" (1 / 1)
INFO:root:solve 1: slice=[1025, 1050[ (42 / 67), index="5200/index-5205-13.fits" (1 / 1)
malloc(): invalid size (unsorted)
Aborted (core dumped)

sometimes the error is different:

python3: malloc.c:4302: _int_malloc: Assertion `(unsigned long) (size) >= (unsigned long) (nb)' failed.

or

malloc(): unaligned tcache chunk detected

I am able to get around this by providing a smaller sample of star positions at different scales.

Thank you for creating this python module! It is very useful.

[aMarcireau]

Hi @kevinss5,

Could you send me an example of a Python snippet that causes the issue?

[kevinss5]

Here is the code snippet:

arcsec_per_px_A = 1.509446918918919
arcsec_per_px_B = 1.8448795675675678
r =       1.1153135567567567
ra_deg =  277.4297583333333
dec_deg = 35.87993888888889
scale =   5
solver = astrometry.Solver(
                        astrometry.series_5200.index_files(
                            cache_directory=astrometry_dir,
                            scales=set([scale]),))

solution = solver.solve(
                        stars=stars,
                        size_hint=astrometry.SizeHint(
                            lower_arcsec_per_pixel=arcsec_per_px_A,
                            upper_arcsec_per_pixel=arcsec_per_px_B),
                        position_hint=astrometry.PositionHint(
                            ra_deg=ra_deg,
                            dec_deg=dec_deg,
                            radius_deg=r),
                        solution_parameters=astrometry.SolutionParameters(
                            logodds_callback=lambda logodds_list: (
                                astrometry.Action.STOP
                                if logodds_list[0] >= 100.0
                                else astrometry.Action.CONTINUE)))     

If the length of the 2D array of stars is greater than a certain value, for example 1686, then the malloc error occurs, if the solving condition hasn't been met yet.

I didn't attached the values of the 2D array of star px positions. I can if you want, though, randomly generated values would likely suffice.

[aMarcireau]

@kevinss5, I think I found the issue.

Astrometry.net (which this repository wraps) has a hard-coded "throttle" for the maximum number of stars in a field (https://github.com/dstndstn/astrometry.net/blob/3095565955e3ff26507cf1f7233b1e973a88c643/solver/solver.c#L791). numxy is not the number of stars in the batch but the upper bound of the indexes in the current batch (even with 25 objects per batch, numxy goes beyond 1000 for very large fields).

numxy is used in malloc a few lines below (https://github.com/dstndstn/astrometry.net/blob/3095565955e3ff26507cf1f7233b1e973a88c643/solver/solver.c#L875) and field[A] is used to index the allocated array. This is not usually an issue since field[A] is smaller than startobj, which is itself smaller than numxy. numxy gets capped to 1000 but startobj does not, which is most likely the cause of the error you encountered.

I am going to open an issue in Astrometry.net. In the mean time, you can work around the issue by manually restricting the field to 1000 before calling this library.