Handling `created`, `updated`, and `revision` with exports and imports

[jdanish]

My pitch is:

• On all exports, include Created, Updated, and Revision regardless of what the template says. The template only controls if they are displayed (this is a special case for provenance which is different from the other types of attributes).
• On import, if any of those are blank / missing, set Created to the current date and Revision to 0
• Instead of revision -1, can we use Created and Updated? When a node is first created, set its Created date but leave Updated blank. When it is saved, set its updated to match the created and revision can stay at 0. That way we know it was last updated the same time it was created. This should be more robust in case anyone uses updated for a filter? And if you want to know it was never actually changed, you can do that using the revision field (if it is 0, it has never been changed).
• Therefore, we assume if something has been created but not yet updated, it was not saved.
• The exception is on import where we assume that if there is no updated, and revision is 0, -1 or blank set it to 0, and match updated to the created. Otherwise, match it to the current date because something is off and it was stored on import (e.g., if for some reason there is a created, no updated, and yet revision 3, which shouldn't really happen).

If that revision to the schema doesn't make sense, using revision count for the save function is fine by me, but I think this might do similar work and keep some things up to date / easier to interpret?

@benloh @kalanicraig does this work?

[benloh]

Heh, sorry I didn't realize setting the assignee would send this to you right away...there are some interacting issues that I'm still working out. I'll update later tonight or tomorrow.

[jdanish]

Ok - I thought that was code for “tell me what you want” 😂

[benloh]

We need to work out a couple of disconnects related to created, updated, and revision.

In our transition from the 1.0 UI to the current 1.5 UI, the Created, Updated, and Revision values were not fully implemented. As a result there might be missing values or "Invalid Dates" on some project and export files.

There was a disconnect with the way we were handling these fields. The created, updated, and revision fields are all generated and updated automatically by loki with every database operation. These fields are saved as meta fields (e.g. node.meta.created, and edge.meta.created). In our UI overhaul, we had flattened the fields and removed the use of the meta subproperties. This conflicted with the way the loki database was handling those updates, resulting in mixed up times and revisions. This should now be resovled.

1. revision Revision behavior is now slightly different.
   • When a new node/edge is created, the revision is set to 0.
   • If you click "Cancel" before saving the new node/edge, the new node/edge will be removed.
   • If you click "Save" the revision will be incremented to 1.
   • Any imported node/edge that does not have a revision value defined will default to 0, and then increment to 1 upon saving the import to the database.

Note that -1 is no longer used.

See also #125.

2. created and updated

When exporting data, we export created and updated fields as human-readable text. e.g. 1/25/2024, 4:22:06 PM This can be useful for browsing the exported CSV file.

The internal data format used in the db is actually an integer representing the number of milliseconds elapsed since the epoch, which is defined as the midnight at the beginning of January 1, 1970, UTC. The importer/exporter automatically does this conversion.

The overall import and export behavior has been refined.

Exports

• [X] Include created in exports if template's hidden value is false
• [X] Include updated in exports if template's hidden value is false
• [X] Include revision in exports if template's hidden value is false
• [X] created and updated will export UTC milliseconds to localeString, e.g. 1/25/2024, 4:22:06 PM

Imports

• [X] If revision is defined, use the imported value to override the db value
• [X] If revision is not defined, default to 0
• [X] created and updated will convert localeString 1/25/2024, 4:22:06 PM to internal UTC milliseconds
• [X] If created is missing, the import time will be used for created AND updated time (because writing to the db triggers setting updated, and if created is missing but updated is not, updated will still be overwritten with the import created time).
• [X] If updated is missing, the import time will be used.

Filters

• [ ] Add 'created', 'updated', and 'revision' to filters?

Caveats

• Project files and exports created between June 2023 and January 2024 might (v1.5.0 - v1.5.1) might not have accurate Created, Updated, and Revision values. The project files should read and function properly, but the dates and revision numbers may not be accurate (you might see "Invalid date" on some records).

We'll probably want to bump the version number to 1.6.0 to reflect the change in data format.

This should be more robust in case anyone uses updated for a filter?

@jdanish We do not currently support filtering based on the meta date data. Do we need to add support for filtering based on created and updated fields? revision too? Adding filters for dates is a little complicated -- we'd probably have to add a date picker.

[jdanish]

OK, great. I think let's stop there. I'm tempted to add 3 issues that we revisit later: 1) Adding date fields (would be generally nice, and also needed for 3, but not urgent) 2) Making provenance elements visible in the node / edge tables (nice but not urgent) 3) filtering by updated / created (nice, and no clue who would use, so definitely not urgent)

One of the things we'll be discussing with the teachers who just used Net.Create as well as our advisory board is what design priorities they felt would help students most next, etc. With those discussions coming, and then the comment fields being developed among other things, I think we are far enough on provenance to leave it be for the moment and see how things come together. Does that work? Otherwise, this fix seems good, but I'll test locally a bit more before officially reviewing and I'd say let's tag then because it impacts import / export.

Flagging for @kalanicraig to sign off

[benloh]

Yeah, punting on date fields and filtering on created/updated sounds good. I'll add that as an issue.

Adding provenance to tables is easy, but I think we hadn't because I had expected us to make changes to it. But this is a 5 minute change, so I'll go ahead and add it.

[jdanish]

Is it possible to put provenance all in one cell / column for space purposes on display?

[benloh]

Is it possible to put provenance all in one cell / column for space purposes on display?

I'm not sure I understand. You want it on one line?

This is what it currently looks like:

I'd rather not introduce horizontal scrolling because it becomes really hard to manage.

[jdanish]

No, that’s fine. I was thinking of adding update and revisions into those cells but that’s too much. Go with that thanks!

[benloh]

If we have time we probably need to do an overhaul of the tables anyway to better handle column widths.

[jdanish]

Yeah that’s part of why I suggested it for the comment list - having better column setup and adjustment would be nice!