search

flightaware / speedtables

Speed tables is a high-performance memory-resident database. The speed table compiler reads a table definition and generates a set of C access routines to create, manipulate and search tables containing millions of rows. Currently oriented towards Tcl.
https://flightaware.github.io/speedtables/
BSD 3-Clause "New" or "Revised" License
66 stars 15 forks source link

stapi::refresh_ctable does not clear the initial table if no time column is provided #51

Open mutability opened 8 years ago

mutability commented 8 years ago

If no time ("changed") column is available then refresh_ctable will reread the whole table from the DB.

However it does not appear to clear the current table contents before doing so, so deleted rows will live forever.

resuna commented 8 years ago

This would appear to also be a problem if there is a time column available as well, since partial reads won't provide any information about deleted rows either. So either way deleted rows live forever.

I'm not sure this is generally resolveable at this level. What are your thoughts?

mutability commented 8 years ago

For schemas with no change-timestamp and where rows are never deleted, the current situation works.

For schemas with no change-timestamp and where rows are deleted, the current situation does not work but could be made to work by clearing the table before re-read. This is what this specific issue is about and is probably the most common simple setup. If we support refresh-with-no-change-timestamp-column at all, we should probably support this case.

For schemas with a change-timestamp where rows are never deleted, the current situation works. A subset of this is cases where rows are never deleted at the DB level but they are flagged as inactive which is probably the most general case.

For schemas with a change-timestamp where rows are deleted, I can't see how incremental updates can be made to work; you basically have to refresh the whole table to notice deletions; to get incremental updates you need to avoid deleting rows.

resuna commented 8 years ago

I'm mostly thinking of the semantics, it should be explicitly obvious that a table is or is not fully in sync with the underlying table. Either a flag on the table to make refresh_ctable clear the table, or (maybe better) a flag on refresh_ctable to force a reload.

resuna commented 8 years ago

I've added a routine "reload_ctable" that bypasses the time check and resets the ctable before reloading it. Semantically it's otherwise identical to refresh_ctable. I kind of want to brainstorm this before refactoring it to always reset the ctable if the time can't be determined. I'd rather make the rule "always call reload_ctable if rows may be deleted" rather than having code that changes behavior if a time column is added or deleted in the definition of the stapi table.

resuna commented 8 years ago

This was merged with stapi-fixes branch. There are separate functions refresh_ctable and reload_ctable.