numpy TypeError on coordinate transform (TypeError: For this input type lists must contain either int or Ellipsis)

[bengranett]

A numpy TypeError is raised when doing a coordinate transformation on an array. This appeared with numpy v 1.14.0. The following code gives the error, copied below. python version is 2.7 astropy version is 2.0.3

import numpy as np
import astropy
from astropy import coordinates, units

print "numpy v", np.__version__
print "astropy v", astropy.__version__

c = coordinates.SkyCoord(ra=[10, 11, 12, 13]*units.degree, dec=[41, -5, 42, 0]*units.degree)
print c.transform_to('galactic')

numpy v 1.14.0
astropy v 2.0.3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ben/.virtualenvs/test/lib/python2.7/site-packages/astropy/coordinates/sky_coordinate.py", line 482, in transform_to
    new_coord = trans(self.frame, generic_frame)
  File "/Users/ben/.virtualenvs/test/lib/python2.7/site-packages/astropy/coordinates/transformations.py", line 1314, in __call__
    curr_coord = t(curr_coord, curr_toframe)
  File "/Users/ben/.virtualenvs/test/lib/python2.7/site-packages/astropy/coordinates/transformations.py", line 1232, in __call__
    newrep = self._apply_transform(fromcoord, M, None)
  File "/Users/ben/.virtualenvs/test/lib/python2.7/site-packages/astropy/coordinates/transformations.py", line 1005, in _apply_transform
    rep = rep.transform(matrix)
  File "/Users/ben/.virtualenvs/test/lib/python2.7/site-packages/astropy/coordinates/representation.py", line 1115, in transform
    newxyz = np.einsum('...ij,j...->i...', matrix, oldxyz.value)
  File "/Users/ben/.virtualenvs/test/lib/python2.7/site-packages/numpy/core/einsumfunc.py", line 1087, in einsum
    einsum_call=True)
  File "/Users/ben/.virtualenvs/test/lib/python2.7/site-packages/numpy/core/einsumfunc.py", line 688, in einsum_path
    input_subscripts, output_subscript, operands = _parse_einsum_input(operands)
  File "/Users/ben/.virtualenvs/test/lib/python2.7/site-packages/numpy/core/einsumfunc.py", line 432, in _parse_einsum_input
    raise TypeError("For this input type lists must contain "
TypeError: For this input type lists must contain either int or Ellipsis

[pllim]

FWIW, I don't see any error using Python 3 and Astropy dev. Can you update Python and/or Astropy?

>>> import numpy as np
>>> import astropy
>>> from astropy import coordinates, units
>>> print("numpy v", np.__version__)
numpy v 1.14.0
>>> print("astropy v", astropy.__version__)
astropy v 3.1.dev21142
>>> c = coordinates.SkyCoord(ra=[10, 11, 12, 13]*units.degree, dec=[41, -5, 42, 0]*units.degree)
>>> print(c.transform_to('galactic'))
<SkyCoord (Galactic): (l, b) in deg
    [(120.60763058, -21.82012272), (118.02401029, -67.80084115),
     (122.24838304, -20.86718474), (123.24009635, -62.87141125)]>

[mhvk]

I cannot directly reproduce the error on python3 (astropy 2.0.3, numpy-dev), but can reproduce a later part:

c.cartesian.transform(np.eye(3)[np.newaxis])
# ValueError: ("Size of label '%s' for operand %d does not match previous terms.", 'N', 1)

Or more directly proving this is an upstream problem:

np.einsum('...ij,j...->i...', np.eye(3)[np.newaxis], np.array([[1., 1.], [2., 2.], [3., 3.]]))

It is almost certainly related to https://github.com/numpy/numpy/issues/10343, since I can get rid of it by passing on optimize=False.

I labelled it upstream-fix required since the underlying problem is numpy, though if need be we probably could have a work-around.

[pllim]

Huh... Incidentally there is a numpy-dev failure at https://travis-ci.org/astropy/astropy/jobs/326547832 that also mentioned "einsum". Is that related?

[mzechmeister]

I have this problem when I try get to run this http://docs.astropy.org/en/stable/time/#barycentric-and-heliocentric-light-travel-time-corrections.

I traced it back to unicode_literals used in astropy/coordinates/builtin_frames/utils.py. Because

from __future__ import unicode_literals
import numpy as np
np.einsum('...i,...i', [1,2,3], [2,3,4])

gives the error. Without unicode_literals it runs. I figured out that immediately in the call np.einsum '...i,...i' is converted to u'...i,...i'. Maybe related to this https://lists.gt.net/python/dev/1292881.

[saimn]

It's a Numpy 1.14 issue, since it worked with the previous versions (I just upgraded to Numpy 1.14 in my conda py2 env and now I get the error). It should work as before with unicode_literals since the string contains only ascii characters.

EDIT: see https://github.com/numpy/numpy/issues/10369

[jobovy]

I'm getting this error using the astropy implementation of (ra,dec) -> (l,b) coordinate transformations

https://travis-ci.org/jobovy/galpy/jobs/328214749

Only my Python2 tests fail, Python3 all pass.

[mhvk]

@jobovy - This indeed looks like the same bug, at least if you are also using numpy 1.14.

[jobovy]

This is a travis build, using whatever numpy conda install numpy currently gives (can't go back and check!), but I assume this is numpy 1.14.

[cdeil]

In Gammapy CI tests also started to fail, I think it's the same issue. See error on Windows with Python 2.7, Numpy 1.14.0 and Astropy 2.0.3 : https://ci.appveyor.com/project/cdeil/gammapy/build/1.0.2980/job/8nquacyxlj4lyyt9#L2392

Why didn't this issue come up before the Numpy 1.14 release in Numpy dev builds? Is it because there was no config triggering it? Worth adding one?

Do you think there will be an upstream fix deployed within the next week or so? If no, I guess we should adapt our tests and add xfail or skip markers for Numpy 1.14?

[cdeil]

I've investigated another error we now got in Gammapy CI with Python 2.7, Numpy 1.14.0 and Astropy 2.0.3: https://travis-ci.org/gammapy/gammapy/jobs/328225843#L2254 Minimal example:

>>> from astropy.coordinates import SkyCoord
>>> a = SkyCoord(0, 0, unit='deg')
>>> b = SkyCoord([0], 0, unit='deg', frame='galactic')
>>> a.separation(b)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/coordinates/sky_coordinate.py", line 749, in separation
    raise TypeError('Can only get separation to another SkyCoord '
TypeError: Can only get separation to another SkyCoord or a coordinate frame with data

Actually this originates from the same issue in Numpy einsum triggered by SkyCoord.transform_to discussed above. It just looks different because the NumpyTypeError is just caught and hidden by this code in SkyCoord.separation: https://github.com/astropy/astropy/blob/4ea556e14340760bd58e53ea2cbccde7ce5567c3/astropy/coordinates/sky_coordinate.py#L757

            try:
                other = other.transform_to(self, merge_attributes=False)
            except TypeError:
                raise TypeError('Can only get separation to another SkyCoord '
                                'or a coordinate frame with data')

I thought I'd mention here, in case other users encounter that error, or if someone (@eteq?) can think of a better way to refactor SkyCoord.separation to use some other mechanism to check if the other coordinate frame has data instead of catching TypeError.

[mhvk]

numpy 1.14.1 is supposed to be released by end of January.

[mhvk]

p.s. The reason we didn't catch it is that our numpy-dev test is done with python3, and there the same path is not taken.

[bsipocz]

This is really a 2.0.4 issue as it only affects python2 (but in practice that release will come out very close to 3.0.0 anyway).

[mhvk]

@bsipocz - not quite: on python3, I have a problem with

c.cartesian.transform(np.eye(3)[np.newaxis])

The fact that we don't use this internally doesn't mean it cannot get done...

[eteq]

The core question is: are we satisfied just saying astropy is "not compatible" with 1.14.0 and requires 1.14.1? It's pretty clear this is a straight-up bug in numpy (c.f. numpy/numpy#10369 / numpy/numpy#10369 numpy/numpy#10371), so saying we aren't going to address it is fairly reasonable, IMHO.

The alternative would probably require adjusting the einsum call to do whatever set of transposes/etc are necessary to replicate the einsum. That would have both performance and readability implications that I am not too comfortable with.

So with that in mind I think we should not make a change, and close this issue once 1.14.1 is out (and we are sure it's working). Anyone disagree?

If we do go that route, though, we may want to hold off on the astropy 3.0 release until 1.14.1 is out... @mhvk, do you have any idea whether "end of Jan" is likely to be acheived or if we should have a backup in case not?

[eteq]

Scratch that - I realize the fix is pretty trivial (should have read https://github.com/astropy/astropy/issues/7051#issuecomment-356113821 more carefully) - set the optimize keyword explicitly to False only for numpy 1.14.0... See #7105 . So I take back all of https://github.com/astropy/astropy/issues/7051#issuecomment-359225657 and instead would say just merging #7105 is the best solution...

[bsipocz]

@eteq - I'm in favour of having the fix for now, as the numpy milestoning is a little bit confusing as the fix upstream is milestoned to 1.15. We can always remove the workaround if the upstream fix comes out before astropy 3.0

[rkpradhan]

Can you help me? @jobovy

import astropy.units as u import astropy.coordinates as coord icrs = coord.ICRS(ra=258.58356362u.deg, dec=14.55255619u.deg, ... radial_velocity=-16.1*u.km/u.s) Traceback (most recent call last): File "", line 2, in File "/usr/lib/python2.7/dist-packages/astropy/coordinates/baseframe.py", line 574, in init list(kwargs))) TypeError: Coordinate frame got unexpected keywords: ['radial_velocity']

[mhvk]

@pythonastroM31 - this is not related to this bug -- are you sure you are using astropy version >= 2.0? (the radial_velocity was only introduced in 2.0). If so, please open a new issue and give more details about versions, etc.

[rkpradhan]

@mhvk -- first of all, thank you so much. Now I have upgraded the astropy-2.0.3; again the same issue raised when I run the same code in the terminal. I am looking for conversion of phase-space coordinates from the galacto-centric frame of reference, and for calculating the values of radial, angular and azimuthal velocity from given ra, dec, d_gc etc. of CSV file data.

[adrn]

@pythonastroM31 Your question is not related to this issue. Please raise your question in a new issue, or in the astropy slack channel

[rkpradhan]

@adrn Okay, my problem is related to typeError in running the program. I just run in the terminal; code like that...

import astropy.units as u import astropy.coordinates as coord icrs = coord.ICRS(ra=258.58356362u.deg, dec=14.55255619u.deg, radial_velocity=-16.1*u.km/u.s) Traceback (most recent call last): File "", line 2, in File "/usr/lib/python2.7/dist-packages/astropy/coordinates/baseframe.py", line 574, in init list(kwargs))) TypeError: Coordinate frame got unexpected keywords: ['radial_velocity']

Can you help me with this problem? How can we resolve this?

[mhvk]

@pythonastroM31 - please open a new issue - it is not helpful to us to have comments about unrelated problems in this thread.