Closed timhowgego closed 1 year ago
Thanks for posting this issue @timhowgego. Can you help narrow down the problem by figuring out which file, column, and values in your particular GTFS file are failing to parse? Partridge doesn't convert GTFS ID columns to integers. You can see the default "converters" for each file in config.py.
I've been encountering the same problem as well since ~2ish days ago (worked flawlessly before this), the problem seems to be with the route IDs in my GTFS feed. Maybe it is one of your dependencies?
I built a bare bones empty GTFS file - basically common headers only, no data - and then proceeded to add dummy data.
With one data line in trips.txt (all IDs as strings and other files header-only), the error triggers at the first column, which was route_id. Upon setting the route_id data to an integer, the error shifts to the next column, service_id. Set service_id to an integer and the error shifts to the third column, trip_id.
However, with all IDs as integers, the error moves to trip_short_name. So it looks like any variable that isn't an integer anywhere in the data will trigger this error. I'd mistakenly concluded this was ID issue because ID fields tend to occur first in GTFS rows. Apologies.
Dependencies may be at issue. Today I specifically uninstalled and reinstalled pandas and numpy (via pip) to fix an unrelated issue. My version reports 1.2.1 for pandas (11 days old) and 1.20.0 for numpy (released yesterday).
Maybe it is one of your dependencies?
I think you are right.
the error triggers at the first column
~Possibly related to the index_col
kwarg to pandas.read_csv
?~ nope, see below
It appears a bug was introduced in numpy when deprecating type aliases. The bug caused np.unicode
to be aliased to np.compat.long
which is an alias for int
. This was quick to track down because of a typo in the deprecation message (unciode
). Details: https://github.com/numpy/numpy/pull/18230/files#diff-09a2c2e86cca91dac0c336e4dad47197ba3cd837ff6c85abf8525cc3b879d3e6R213-R216
Here's a reproduction of the issue in pandas: https://gist.github.com/invisiblefunnel/a896921083f8cf03843e72da8dba783f.
I will file a bug with numpy if someone else can confirm my assessment. Meanwhile, is anyone interested in updating partridge to use str
instead of np.unicode
?
I will file a bug with numpy if someone else can confirm my assessment.
Looks like it! long
was incorrectly substituted in this change, before it was using str
as the substitution. There's also a bug with aliasing np.long
introduced in the same section of code, if you want to mention it.
Meanwhile, is anyone interested in updating partridge to use str instead of np.unicode?
I can take a crack at updating it to str
to unblock myself :) Seems like it is just a few simple changes.
There's also a bug with aliasing np.long introduced in the same section of code, if you want to mention it.
What's the bug with np.long
?
Actually on second look it should be fine. Cheers.
Numpy bug report https://github.com/numpy/numpy/issues/18287
@BlackSpade741 Obviously not my baby, but on casual reading your changes look fine to me. My only suggestion would be to also switch other depreciated types to a native Python type, simply to eliminate redundant (yet evidently breakable) dependencies:
@timhowgego Good point! I'll take a look later :)
Seems like number types with specific precisions (specifically np.float64
and np.int64
that we are using in test_readers.py
) are not being deprecated. Curious if you folks think it is better to keep these or move to the more general int
.
Better to scope your changes to deprecated type aliases 👍 .
Updated the PR to include np.object
as well!
Description
As is, it appears that any non-numeric ID field will crash Partridge because Pandas is expecting IDs to be integers. Unfortunately the GTFS spec states that a valid ID "is a sequence of any UTF-8 characters", and in practice the use of non-numeric IDs is widespread.
What I Did