Feature: Provenance

[benloh]

This introduces a revamped "Provenance" tab.

To implement this, we've introduced another major revision to the template format, and are adding a version designator to gracefully handle migrations in the future.

Breaking Changes

NOTE: the template format has changed, but the app should still gracefully load older projects. You may need to manually update the project's template to add new Provenance functionality. See Working with older projects below.

• Add Template Versioning
  • Templates now have a version designation.
  • The version should match the github Release version. Hopefully this will be less confusing than having separate version numbers. The one caveat is that it's harder to track interim development/alpha versions prior to a release.
  • When opening projects, the template version (or lack of a template version) will allow the system to decide how to migrate or ignore data fields. e.g. previously built-in fields like comments will continue to be displayed and will not break in table views.
• Nodes and Edges recently added semi-built-in comments and provenance fields to the default template as placeholders. These have now been removed in favor of the "real" implementation. d713ca1a6f85fe832f0114ef810042054fb0b903

Significant Features

• The Provenance tab has been reformatted to show two sections: "PROVENANCE" and "HISTORY".
• Any fields that are designated isProvenance will be added to the "PROVENANCE" tab.
• Added a createdBy and updatedBy built-in fields 6fc8dee6eea91c9fd96a323745fabf91f4c74d37

Minor Features

• The old semi-built-in fieldsProvenance and Comments have been removed from Node and Edge tables. Any new "isProvenance" field will be displayed in the tables.

See also #37 and Provenance Ideas

---

To Test

The easiest way to test this is to create a new project. (_default.template.toml has been updated, so any new project should use the new template schema).

1. ./nc.js --dataset=newtest
2. Create a token and log in. (ncMakeTokens('new','a','newtest',2))
3. Create a node
4. Savet eh node
5. Click on "Provenance" -- you should see "Info Source" and "HISTORY" sections with your login data as the created by and updated by author.

Working with older projects

Older projects (e.g. any project created before May 23, 2024), will still open.

• Any fields that have not been designated as isProvenance (in the template definition) will be displayed in the main "ATTRIBUTES" tab. e.g. if you recently created a project, it probaby has a provenance and comments field defined. These will show up in "ATTRIBUTES".
• The new Provenance tab should work, e.g.:
  • "HISTORY" wil show created by and updated by.
  • Any created/updated author that is missing will be displayed as "(not recorded)"
  • Since there aren't any fields designated as isProvenance, there won't be any fields available.

If you want to add provenance fields to an older project you have two options:

1. Create a new field, e.g. "Info Source", or
2. Add isProvenance to an existing field to have it appear in the "PROVENANCE" tab. e.g. you can convert the old provenance field into an active provenance field by adding the isProvenance flag.

To Do

• [x] updated field is not updating
• [x] custom edge provenance fields are not accepting input
• [x] Update template migration to support semantic versioning strings (e.g. 10.1 is NOT less than 2.0)? -- won't do now. See #217
• [X] Date fields do not support historical dates -- won't do now. See #207
• [X] Support filter by provenance dates -- won't do now. See #130

[jdanish]

I created a new network and node. The created by field seems correct but updated is not changing?

[jdanish]

So far, looks good. I assume this is on a list somewhere, but the main functionality missing is being able to filter by / view the dates of change / update. Otherwise, I'll keep banging but creating a new network and importing then tweaking an old one both appear to work AOK.

[jdanish]

Just noted that the imported template is not changing the template version. Is that meant to be hand-edited or it should update?

[jdanish]

Spoke too soon. In a network created after the code update (so version 2 template), added an edge and the user for update and add are not being filled, even though they filled for the node, and when I click on the provenance field I cannot type anything.

[benloh]

I created a new network and node. The created by field seems correct but updated is not changing?

Will look into this.

So far, looks good. I assume this is on a list somewhere, but the main functionality missing is being able to filter by / view the dates of change / update.

I think we probably want to implement historical dates before we spend time working on the filters because that will completely change how we handle dates.

Just noted that the imported template is not changing the template version. Is that meant to be hand-edited or it should update?

I guess I didn't anticipate this use model. So I'm assuming you had started with the old version template and then hand-editted the template with new fields?

Currently we're not migrating the data from a pre-2.0 template to a 2.0 template. I was assuming that you would want to retain the old functionality. So my goal with the migration was to just prevent things from breaking.

I suppose if we have a large corpus of projects that need migration we can add more explicit migration tools.

Currently the version needs to be hand updated. I didn't want that to be edited in the template editor.

Spoke too soon. In a network created after the code update (so version 2 template), added an edge and the user for update and add are not being filled, even though they filled for the node, and when I click on the provenance field I cannot type anything.

I'll look into this too. I expect there are a few more issues like this since I haven't done a full QA yet.

[jdanish]

Cool. I assumed you'd want to do the date format first but wanted to make sure nothing fell through the cracks.

Re: migration. No, there are only a few and I can update them by hand. I think this makes sense. In the future we'll want it to be automatic as we begin to expand our user base, but I'd rather do this by hand and have you do other more important things.

And all good re: issues - that's what QA is for!

[dsriseah]

Some weird behavior I noticed:

Testing New Project

On Autosave with one node, I noticed the server going haywire (launched from a terminal, node 18.18.2, arch i386)

ServerDB - AUTOSAVING! 1 NODES / 0 EDGES / 0 COMMENTS / 0 READBY <3
ServerDB - AUTOSAVING! 1 NODES / 0 EDGES / 0 COMMENTS / 0 READBY <3

--- rebuilding _ur_addons ---
    building core...
    building addons...
    triggering recompile...

[15 repetitions of the above]

13:17:35 - info: Installing packages with npm...

--- rebuilding _ur_addons ---
    building core...
    building addons...
    triggering recompile...

[endless repetitions]

I also noticed a recompile happening when trying to enter a date on provenance and the app stopped responding

A weird error after reloading, doing date entry, where it's defaulting to zero and the input rejection doesn't work because I can't delete the leading zero

[benloh]

Dates are broken at the moment. We're completely redoing date fields.

[dsriseah]

[moved to issue #222]

Speculation as to the cause of recompile loops

There is a watcher process that brunch-config.js defines:

If you are writing data into the _ur or _ur_addons directory, you will trigger a recompile. Data should be written to the NetCreate runtime directory.

TO DEBUG

If you notice it happening again, you can console.log the event and path parameters of the on('all' on line 134 of brunch-config.js to see what is getting written.