Show how far behind the import lag is.

hotosm / old-export-tool

Version 1 of the Export Tool is suspended - please see V3 https://github.com/hotosm/osm-export-tool

16 stars 18 forks source link

Show how far behind the import lag is. #74

Open AndrewBuck opened 10 years ago

AndrewBuck commented 10 years ago

It would be nice to show the import lag for the export database to know how current the data in the export will be. Sometimes during a crisis response you make some change to the map database and want to push that out to aid organizations quickly so you may want to re-run an export, if the lag is large though your changes might not actually be on the export server yet.

dodobas commented 10 years ago

Tasks:

[ ] create a DB model to store update script execution information
- [ ] start_time, end_time, maybe something else?
[ ] update backend/update.sh to insert new record in the database
[ ] add a simple information box on the main service page showing a few stats like, average running time in the last 2 hours, max running time in the last 24h, min running time in the last 24h

AndrewBuck commented 10 years ago

I think you misunderstood what I meant, the scripts already show how long they take to run and when they were run. What I am talking about is how current is the data in the database the extracts are being pulled from. I.E. if I add an object like a building, how long do I have to wait to run the export before that newly added building will show up in the output files.

dodobas commented 10 years ago

there is no database, regions are raw OSM files where are constantly updated, basically, for every file in the folder apply latest 'changes', at the moment it takes 10-15min to update the files

harry-wood commented 10 years ago

Yeah so I think Andrew is asking for some basic system status display exposed to end users, to reassure them that the updater process is running correctly.

We could show a table of when each region file was last updated, and maybe which region file is currently being processed. Also I was thinking we could show things like which export run is currently being processed, and how long the queue is. This lets people know that something is happening, and maybe also helps people understand what kind of jobs cause a hold-up, what time of day are the busy periods etc.

Could be done as munin graphs like at http://munin.openstreetmap.org But would that be duplicating the nagios system? Don't know how that works.

AndrewBuck commented 10 years ago

Yes, what harry suggested is what I was thinking. I know the overpass API does this when you request data it has a header at the top telling you how current their database is, so you know if it starts to fall behind. It is usually only a minute or two which is negligible, but if I want to generate something to send to Red Cross field teams or something like that, I want to know that the data inside it contains what I want it to contain, and often that is something I just added right before wanting to run the export tool. Right now the only way to find out is run it blindly and then download and check myself to see if it is in there.