search

AequilibraE / aequilibrae

aequilibrae - Python package for transportation modeling
https://www.aequilibrae.com
Other
166 stars 38 forks source link

Need a method for "massive" new links creation #548

Open Art-Ev opened 3 months ago

Art-Ev commented 3 months ago

Hi,

Currently, creating a project from existing files or GIS layer is done via adding links "one by one". This result in pretty long running time because .spatialite file is opened, edited, consolidated and closed once per link we want to add.

Maybe we can create a function taking a pandas dataframe as input and putting the new links all at once in the spatialite file ? (or by group of several thousand of links ?) Or maybe set the opening and closing of the spatialite file "manual" instead ?

When building project from OSM, how is the spatialite file updated @pedrocamargo ?

pedrocamargo commented 3 months ago

@Art-Ev , this is indeed a time-consuming process. Currently, the OSM importer does this with an executemany call to sqlite3

https://github.com/AequilibraE/aequilibrae/blob/a7339be9a8d8366a0a8b08cfdbafddeeec4cce54/aequilibrae/project/network/osm/osm_builder.py#L145

Creating all nodes before doing the links makes it a LOT faster, which is what we do with OSM (see line above).

Whenever I do things on a "large scale", I remove all triggers, add links and nodes and then add the triggers back. That is a hack that I can't recommend, but it could've made OK by running a set of Spatialite queries AFTER all links and nodes are added to enforce consistency.