flightaware / speedtables

Speed tables is a high-performance memory-resident database. The speed table compiler reads a table definition and generates a set of C access routines to create, manipulate and search tables containing millions of rows. Currently oriented towards Tcl.
https://flightaware.github.io/speedtables/
BSD 3-Clause "New" or "Revised" License
66 stars 15 forks source link

stapi::refresh_ctable doesn't rewrite the TSV cachefile #50

Open mutability opened 8 years ago

mutability commented 8 years ago

The TSV cachefile gets written once during ::stapi::open_cached, but then it never appears to get updated, even if you're calling ::stapi::refresh_ctable.

So for long-running processes that periodically refresh, the cache rarely gets used.

resuna commented 8 years ago

With a long-running process, the in-memory speedtable is used. ISTR this was created to avoid having short-running processes beating on the speedtable server.

How would you envision long-running processes using the cache?

mutability commented 8 years ago

On restart of the process, to reduce the recovery time? Loading from the cache seems to be substantially faster than loading by running the query.

resuna commented 8 years ago

If the in-core table is completely up to date (re #51 ) then this would be worthwhile doing. If you know the in-core table is valid because you're performing a complete reload or because you know there are no deletions, then updating the table would be worthwhile.

This could be done automatically in reload_ctable or manually (with an update_cache call) after refresh_ctable.

resuna commented 8 years ago

Should probably have set this to a non-zero value, originally.

  # Default "stale cache" timeout, zero for no timeout.
  variable default_timeout 0
resuna commented 8 years ago

Have you tried setting the -timeout value (seconds) which is supposed to refresh the tsv file after $timeout seconds when you open it?

resuna commented 8 years ago

Have a look at https://github.com/flightaware/speedtables/commit/bc8513cb00984b05a512fd7fc64abb9963fcc350