search

NREL / mappymatch

Pure-python package for map matching
http://mappymatch.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
61 stars 20 forks source link

Support Multiprocessing #189

Open nreinicke opened 1 month ago

nreinicke commented 1 month ago

As discovered in #187, pickling the road map rtree is currently failing which disables us from leveraging multiprocessing for a batch of traces. Until the pickling issue is resolved we could explore workarounds to allow traces to be processed in parallel. Road maps could be very large and so a solution that allows the road network data to be shared would be ideal.

nreinicke commented 1 month ago

@jhoshiko - I seem to recall that you've run into a similar issue in the past. Do you have any notes from trying to map match large road networks in parallel that might be useful in this context?

jhoshiko commented 1 month ago

@nreinicke In the past, I would initialize a copy of the road network on each worker, scaling the number of workers based on available memory and the size of the road network. I haven't personally experimented with matching to a road network in shared memory, but I agree that it would be ideal. I think multiprocessing/joblib might have options for instantiating objects into shared memory, but I haven't personally tested anything.