When matching fits files: "ValueError: Big-endian buffer not supported on little-endian compiler"

[hjmcc]

Dear all,

thanks for providing this well documented package, it seems just what I need to align some images I have for a project with students.

• I tried the demo notebook, all seems to work fine without any trouble.

• I am running the most recent version of astroalign (2.2).

• However I tried to align two sets of two images, two images of 47 tuc, two images of the helix nebulae. For 47 tuc where there are lots of stars it should surely work!

• Instead I get this message about 'Big-endian buffer not supported on little-endian compiler'.

As far as I can tell, the shapes of the input arrays from fitsio are fine. I suspect this is hiding something else. I get the same error on macos as on a linux system. Surely I am making some simple mistake here?

Many thanks! Henry

image_file = os.path.join(data_dir,'calibrated-T30-fetedelascience-NGC104-20201119-230656-R-BIN1-W-300-001.fit')
source=fits.open(image_file)[0].data

target_file = os.path.join(data_dir,'calibrated-T30-fetedelascience-NGC104-20201119-231248-R-BIN1-W-300-002.fit')
target=fits.open(target_file)[0].data

registered_image, footprint = aa.register(target, source)

ValueError                                Traceback (most recent call last)
<ipython-input-9-aea43f655c9d> in <module>
----> 1 registered_image, footprint = aa.register(target, source)

~/miniconda3/envs/py37-astropy/lib/python3.7/site-packages/astroalign.py in register(source, target, fill_value, propagate_mask, max_control_points, detection_sigma, min_area)
    453     )
    454     aligned_image, footprint = apply_transform(
--> 455         t, source, target, fill_value, propagate_mask
    456     )
    457     return aligned_image, footprint

~/miniconda3/envs/py37-astropy/lib/python3.7/site-packages/astroalign.py in apply_transform(transform, source, target, fill_value, propagate_mask)
    384         cval=_np.median(source_data),
    385         clip=True,
--> 386         preserve_range=True,
    387     )
    388     footprint = warp(

~/miniconda3/envs/py37-astropy/lib/python3.7/site-packages/skimage/transform/_warps.py in warp(image, inverse_map, map_args, output_shape, order, mode, cval, clip, preserve_range)
    877                 warped = _warp_fast[ctype](image, matrix,
    878                                            output_shape=output_shape,
--> 879                                            order=order, mode=mode, cval=cval)
    880             elif image.ndim == 3:
    881                 dims = []

skimage/transform/_warps_cy.pyx in skimage.transform._warps_cy._warp_fast()

ValueError: Big-endian buffer not supported on little-endian compiler

[martinberoiz]

Hi @hjmcc,

The problem seems to be in the wheels installation for scikit-image. Or maybe the data on the fits file is saved with a different endian-ness than the local machines.

I found this to swap endian-ness.

Apparently, if you do:

target = target.newbyteorder()
source = source.newbyteorder()

before registering, it should correct the problem.

I certainly never saw this problem before, so please let me know if this fixes it! :)

[hjmcc]

hi @martinberoiz,

Thank you for this super-rapid response! Well I tried your byte-swapping, and code does not crash immediately as before, but runs for about 30 seconds before annoncing:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-70-e249e1970256> in <module>
----> 1 registered_image, footprint = aa.register(target, source, detection_sigma=10)

~/miniconda3/envs/py37-astropy/lib/python3.7/site-packages/astroalign.py in register(source, target, fill_value, propagate_mask, max_control_points, detection_sigma, min_area)
    450         max_control_points=max_control_points,
    451         detection_sigma=detection_sigma,
--> 452         min_area=min_area,
    453     )
    454     aligned_image, footprint = apply_transform(

~/miniconda3/envs/py37-astropy/lib/python3.7/site-packages/astroalign.py in find_transform(source, target, max_control_points, detection_sigma, min_area)
    281         )
    282 
--> 283     source_invariants, source_asterisms = _generate_invariants(source_controlp)
    284     source_invariant_tree = KDTree(source_invariants)
    285 

~/miniconda3/envs/py37-astropy/lib/python3.7/site-packages/astroalign.py in _generate_invariants(sources)
    137     inv = []
    138     triang_vrtx = []
--> 139     coordtree = KDTree(sources)
    140     # The number of nearest neighbors to request (to work with few sources)
    141     knn = min(len(sources), NUM_NEAREST_NEIGHBORS)

~/miniconda3/envs/py37-astropy/lib/python3.7/site-packages/scipy/spatial/kdtree.py in __init__(self, data, leafsize)
    249         self.mins = np.amin(self.data,axis=0)
    250 
--> 251         self.tree = self.__build(np.arange(self.n), self.maxes, self.mins)
    252 
    253     class node(object):

~/miniconda3/envs/py37-astropy/lib/python3.7/site-packages/scipy/spatial/kdtree.py in __build(self, idx, maxes, mins)
    312                 # _still_ zero? all must have the same value
    313                 if not np.all(data == data[0]):
--> 314                     raise ValueError("Troublesome data array: %s" % data)
    315                 split = data[0]
    316                 less_idx = np.arange(len(data)-1)

ValueError: Troublesome data array: [1053.   nan   nan   nan   nan   nan   nan   nan   nan   nan 1669.   nan
   nan 1973.   nan   nan   nan   nan 2751.   nan   nan   nan   nan   nan
   nan   nan   nan 2342.   nan   nan   nan   nan   nan   nan   nan   nan
   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan
   nan 2097.]

It's strange, your demo notebook works perfectly. My images are nothing special, these two are images of the helix nebulae taken by one of the itelescope.net telescopes (t30). I tried passing a higher threshold to SEP, but it still crashes.

Is there something wrong with my conda installation? Or perhaps I should just try with two images with only stars and galaxies.

Thanks!

[hjmcc]

hi @martinberoiz I did some more experiements on my side with some other data I had. It worked fine in that case. Looking more closely, I noticed that those data were dtype=uint16 whereas the data I had been using were dtype=float32.

So now if I do

registered_image, footprint = aa.register(source.astype('uint16'), target.astype('uint16'))

It works ! So my conclusion is that something somwhere is expecting dtype=uint16 .

I guess we can close the issue.

[martinberoiz]

Hi,

To be honest, that seems strange to me. Images should work with int's or floats. Let's close this issue for now and if it reappears, feel free to open a new issue.

BTW if you don't mind sharing a few of the problem images for me to try out, you can send them to my gmail account.

Thanks!

[prajwel]

@martinberoiz I am also facing this issue while aligning images.

import numpy as np
import astroalign as aa

from astropy.io import fits

coo_ref = np.genfromtxt('AS1A02_028T03_9000000788uvtFIIPC00F1A_l2imb_coo.dat', delimiter=',')
coo_to_match = np.genfromtxt('AS1A02_028T03_9000000788uvtNIIPC00F2A_l2imb_coo.dat', delimiter=',')
transf, (source_list, target_list) = aa.find_transform(coo_ref, coo_to_match)

data_ref = fits.open('AS1A02_028T03_9000000788uvtFIIPC00F1A_l2imb.fits.gz')
data_to_match = fits.open('AS1A02_028T03_9000000788uvtNIIPC00F2A_l2imb.fits.gz')

registered_image = aa.apply_transform(transf, data_ref[0].data, data_to_match[0].data)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-22-d36cdadc22fb> in <module>
----> 1 aa.apply_transform(transf, data_ref[0].data, data_to_match[0].data)

~/.local/lib/python3.8/site-packages/astroalign.py in apply_transform(transform, source, target, fill_value, propagate_mask)
    372         target_data = target
    373 
--> 374     aligned_image = warp(
    375         source_data,
    376         inverse_map=transform.inverse,

~/.local/lib/python3.8/site-packages/skimage/transform/_warps.py in warp(image, inverse_map, map_args, output_shape, order, mode, cval, clip, preserve_range)
    875             ctype = 'float32_t' if image.dtype == np.float32 else 'float64_t'
    876             if image.ndim == 2:
--> 877                 warped = _warp_fast[ctype](image, matrix,
    878                                            output_shape=output_shape,
    879                                            order=order, mode=mode, cval=cval)

skimage/transform/_warps_cy.pyx in skimage.transform._warps_cy._warp_fast()

ValueError: Big-endian buffer not supported on little-endian compiler

[martinberoiz]

Hi @prajwel,

Did you try byte-swapping before aligning?

data_ref = data_ref.newbyteorder()
data_to_match = data_to_match.newbyteorder()

registered_image = aa.apply_transform(transf, data_ref[0].data, data_to_match[0].data)

I'm still investigating this, seems like it's an issue with scikit-image.

[prajwel]

I tried it but received the following error.

data_ref = data_ref.newbyteorder()
data_to_match = data_to_match.newbyteorder()

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-27c60e5640d6> in <module>
----> 1 data_ref = data_ref.newbyteorder()
      2 data_to_match = data_to_match.newbyteorder()

AttributeError: 'HDUList' object has no attribute 'newbyteorder'

But the following works,

data_ref[0].data = data_ref[0].data.newbyteorder()
data_to_match[0].data = data_to_match[0].data.newbyteorder()
registered_image, footprint = aa.apply_transform(transf, data_to_match[0].data, data_ref[0].data)

Additionally, the following works too,

data_ref[0].data = data_ref[0].data + 0.0001
data_to_match[0].data = data_to_match[0].data + 0.0001
registered_image, footprint = aa.apply_transform(transf, data_to_match[0].data, data_ref[0].data)

Note: These are UV data and has many pixels with zero values. This messed up the registered image with many Nan values in the previous successful attempt with newbyteorder. That is why I tried adding the constant.

[martinberoiz]

Oh, glad to hear it worked in the end.