Remodel data storage - Githubissues

Project-OSRM / osrm-backend

Open Source Routing Machine - C++ backend

http://map.project-osrm.org

BSD 2-Clause "Simplified" License

6.36k stars 3.37k forks source link

Remodel data storage #1907

Open TheMarex opened 8 years ago

TheMarex commented 8 years ago

Current we support two data backends hidden behind BaseDatafacade. Both codes uses different code paths for loading data into memory, ideally it should be the same.

The whole data backend should be re-written:

[x] Move all relevant code to libosrm-storage
[x] Implement only one data facade
[ ] Make shared memory named (e.g. by profile name)
[x] There should only be one datafacade (abstraction at different level)
[ ] Make loading new data as transparent to the server component as possible

@danpat @daniel-j-h extend if you have any more points

TheMarex commented 8 years ago

Will also fix #1894

frodrigo commented 8 years ago

This will allow multiple shared memory on same host ?

daniel-j-h commented 8 years ago

Yes, via named shared memory; e.g. for multiple profiles.

danpat commented 8 years ago

We need to implement a notification/wait strategy for data reloading as well. There is currently some confusion about which process is responsible for cleaning up no-longer-used shared memory.

The workflow should be:

osrm-datastore loads a new block of shared memory.
Processes using the old block are notified of a new block and begin the switch.
osrm-datastore waits until all users are switched, then removes the old block.

Currently, the readers remove the old block themselves, and osrm-datastore returns as soon as it's done loading. The "notification" is implemented by osrm_impl/osrm-routed checking for new data when a query comes in; this should be changed to an async notification method, and readers should not be responsible for cleaning up old shared memory once they've swapped - this would be difficult to synchronize when there are many readers, and is currently broken :-/

emiltin commented 8 years ago

It would be great to be able to run multiple instances of osrm-routed with different profiles, ie osrm-datastore should be able to keep multiple data source in memory at once.

daniel-j-h commented 7 years ago

@danpat you refactored the data facades - care to have a look here and either update what still needs to be done or close this and split off a different issue with remaining task.

danpat commented 7 years ago

https://github.com/Project-OSRM/osrm-backend/pull/3165 consolidated the logic behind the two datafacades - we now only have a single file-loading codepath.

The only thing we haven't done is make the shared memory regions named.