Closed philshem closed 4 years ago
Thank you, I have actually originally started with gpxpy
and utm
(I'm already using utm
to convert back to latlon to plot in ipyleaflet
), but it was comparatively much slower (speaking about 60k+ points, which represents over 900km of tracks) when loading all raw GPX files each time:
def read_segments():
for path in sorted(Path(".").glob("gpx/*.gpx")):
gpx = gpxpy.parse(path.read_text())
for track in gpx.tracks:
for segment in track.segments:
if len(segment.points) < 5:
continue
yield [
(point.time_difference(segment.points[0]),
utm.from_latlon(point.latitude, point.longitude, 32, "U")[:2])
for point in segment.points
]
segments = list(read_segments())
UTM system is perfect for local geometry and distances, but maybe with GeoPandas the expensive conversion could be skipped.
The performance problem is also the conversion of ALL points, which could be solved by some simplifying methods of gpxpy.
Or some sort of database/cache for all converted points. And we're back at CSVs, which offer the user a stable and reviewable step in data processing.
Another option is to run gpsbabel in the Jupyter notebook with Unix commands (!)
https://intro.syzygy.ca/unix-tricks/#unix-and-magic-in-a-notebook
(Would have to run with ‘wget’ or ‘git clone’ to get local ‘gpsbabel’ binary.)
utm
can do NumPy
arrays, it is just too shy about it. That's why I added it to its docs: https://github.com/Turbo87/utm/pull/50
Converting 1M points in a loop takes 90 seconds, a 1M numpy array needs 0.26s. Now it's an option again.
gpxpy
and utm
used to load GPX files in https://github.com/eumiro/pygohome/commit/1a30a69d8f0089625e8ef05976fc6b2b947df067
Thanks @philshem
based on this tweet: https://twitter.com/eumiro/status/1257008378800484352?s=20
> Also I want to find a fast Pythonic way to read the GPX files to avoid (fast but still an extra step) gpsbabel.
I've had success with gpxpy (and I think now the slow datetime recognition is improved).
Basic usage:
To convert lat/long to UTM grid, you can use this package: https://github.com/Turbo87/utm
my opinion: rather than convert to a table/csv, I'd just convert the gpx to an array of dicts (one for each gps point), using the above tools. Then use pandas.from_dict().