AttributeError: 'list' object has no attribute '__array_interface__' during update

pyrog commented 1 year ago

Context:

database: planet
data: https://download.geofabrik.de/europe/france/rhone-alpes.html

Commands:

wmt-makedb -d planet db update
wmt-makedb -d planet hiking update

work several time but… hiking update finally crash.

I tried to reload the database but same issue.

PS: I also test updates with the small dataset andorra several times and it work fine.

db update log:

osgende-import ['osgende-import', '-d', 'planet', '-r', 'https://planet.openstreetmap.org/replication/minute/', '-S', '51200']
/srv/waymarkedtrails/wmtenv/bin/osgende-import:244: RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
  conn.execute("DROP INDEX IF EXISTS pk_%s_change" % n)

hiking update log:

23-08-19 15:57:48 Updating changed_objects...
23-08-19 15:57:48 Updating filtered_relations...
23-08-19 15:57:48 Updating way_relations...
23-08-19 15:57:51 Updating segments...
23-08-19 15:57:51 Using 1 parallel threads.
23-08-19 15:57:51 Collecting changed and new ways
23-08-19 15:57:51 Collecting points effected by update
23-08-19 15:57:51 Segments with bad intersections...
23-08-19 15:57:51 Processing segments
Exception in thread Thread-1 (worker_loop):
Traceback (most recent call last):
  File "shapely/speedups/_speedups.pyx", line 90, in shapely.speedups._speedups.geos_linestring_from_py
AttributeError: 'list' object has no attribute '__array_interface__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/srv/waymarkedtrails/wmtenv/lib/python3.11/site-packages/osgende/common/threads.py", line 53, in worker_loop
    process_func(req)
  File "/srv/waymarkedtrails/wmtenv/lib/python3.11/site-packages/osgende/lines/segments.py", line 368, in _process_next
    self._write_segment(properties, w)
  File "/srv/waymarkedtrails/wmtenv/lib/python3.11/site-packages/osgende/lines/segments.py", line 373, in _write_segment
    'geom' : from_shape(LineString(segment.geom), srid=self.srid)}
                        ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/shapely/geometry/linestring.py", line 47, in __init__
    ret = geos_linestring_from_py(coordinates)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "shapely/speedups/_speedups.pyx", line 156, in shapely.speedups._speedups.geos_linestring_from_py
ValueError: LineStrings must have at least 2 coordinate tuples
23-08-19 15:57:53 Internal error. Thread died. Killing other threads.
Traceback (most recent call last):
  File "/srv/waymarkedtrails/wmtenv/lib/python3.11/site-packages/osgende/common/threads.py", line 71, in add_task
    self.queue.put(data, True, 2)
  File "/usr/lib/python3.11/queue.py", line 148, in put
    raise Full
queue.Full

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/waymarkedtrails/wmtenv/bin/wmt-makedb", line 248, in <module>
    exit(action())
         ^^^^^^^^
  File "/srv/waymarkedtrails/wmtenv/bin/wmt-makedb", line 141, in update
    self.mapdb.update()
  File "/srv/waymarkedtrails/wmtenv/lib/python3.11/site-packages/osgende/mapdb.py", line 145, in update
    tab.update(self.engine)
  File "/srv/waymarkedtrails/wmtenv/lib/python3.11/site-packages/osgende/lines/segments.py", line 207, in update
    wayproc.process_cached_ways()
  File "/srv/waymarkedtrails/wmtenv/lib/python3.11/site-packages/osgende/lines/segments.py", line 296, in process_cached_ways
    self.process_ways(prev_prop, wayset)
  File "/srv/waymarkedtrails/wmtenv/lib/python3.11/site-packages/osgende/lines/segments.py", line 281, in process_ways
    self.workers.add_task((properties, ways))
  File "/srv/waymarkedtrails/wmtenv/lib/python3.11/site-packages/osgende/common/threads.py", line 143, in add_task
    self.worker.add_task(data)
  File "/srv/waymarkedtrails/wmtenv/lib/python3.11/site-packages/osgende/common/threads.py", line 74, in add_task
    self.check_worker_state()
  File "/srv/waymarkedtrails/wmtenv/lib/python3.11/site-packages/osgende/common/threads.py", line 85, in check_worker_state
    raise WorkerError("Internal error. Thread died.")
osgende.common.threads.WorkerError: Internal error. Thread died.

lonvia commented 1 year ago

This also has to do with extracts. You might want to use the daily diffs from geofabrik, which are referentially more clean.

pyrog commented 1 year ago

Thanks Sarah. That's why we founded ways all over the world even if we download a small part of France 😆

pyrog commented 1 year ago

From memory, it was a "public" file from Geofabrik : rhone-alpes-latest.osm.pbf

Does this one is suitable ? https://osm-internal.download.geofabrik.de/europe/france/rhone-alpes-latest-internal.osm.pbf

lonvia commented 1 year ago

http://download.geofabrik.de/europe/france/rhone-alpes-updates is the replication source for that file. Geofabrik doesn't keep diffs very long. You may have to redo the import with a current file and then start the replication right away.

pyrog commented 1 year ago

Thank you, I understand now.

Could you give some advices for the automatic update ?

do you use a cron in production for wmt-makedb db update, wmt-makedb hiking update… ?
at which interval ?

For our test instance with a "small" dataset from GeoFabrik:

Does the update must be one day ?
Should we use wmt-makedb -r https://download.geofabrik.de/europe/france/ if I understand the following extract from https://download.geofabrik.de/technical.html ?

.osc.gz files, or "diff updates" Whenever we produce a new exctract for a region, we also compute the difference between the new extract and the previous one, and make that available for download so that users can continuously update their own regional extract instead of having to download the full file. The file names for update files follow the convention used by Osmosis for the "read-replication-interval" task, so automatic updates are possible with Osmosis, osmupdate, and pyosmium-up-to-date.

lonvia commented 1 year ago

The production servers do continuous updates from planet-wide minutelies pretty much as per README.

If you use the replication from Geofabrik, set up a systemd job to run around 6am with:

wmt-makedb -r  http://download.geofabrik.de/europe/france/rhone-alpes-updates update db
wmt-makedb update hiking

Geofabrik only publishes daily updates, so this is the maximum you get.

waymarkedtrails / waymarkedtrails-backend

AttributeError: 'list' object has no attribute '__array_interface__' during update #12