TOML Template Editor

[benloh]

Branch: dev-bl/toml

New Features:

• New TOML template file.
• New JSON-EDITOR for templates
• Auto-conversion of old JSON-format templates to new TOML-format.
• You can:
  • Edit Node Types
  • Edit Edge Types
  • Edit Current Template
  • Download Current Template
  • Create a New Template
  • Import an existing Template File
• New "color" setting for Edge Types

---

To Test

We've added a few node libraries, so you'll need to run npm ci.

1. git fetch; checkout dev-bl/toml
2. npm ci
3. npm run dev to load your last used database, or ./nc.js --dataset=projectname to load a specific database.

Because the system now uses a new template file, you should see this message in your terminal:

ServerDB - LOADING JSON TEMPLATE ./runtime/<projectname>.template
ServerDB - Converting JSON to TOML...

Your dataset should load, converting your old JSON template to a new TOML templamte format. You might find that your colors, node types, and edge types have changed. We'll fix all of this later.

First, let's just test the new template features:

4. Click on the "More..." tab.
5. Click on the new "Edit Template" tab.

You have 6 template tools you can use. Try each one. Refer to the documentation below starting at "WIKI DOCUMENTATION" (this will be copied over to the wiki).

Test Admin Access

1. Open a browser on another computer and navigate to your server, e.g. http://192.168.1.1:3000.
2. Make sure NetCreate loads.
3. You should now be logged in as a regular nonAdmin user.
4. Click on the "More..." tab
5. Notice there are now only three tabs available: "Help", "Vocabulary", and "Import/Export". The "Edit Template" tab is not visible.
6. Add ?admin=true to the url, e.g. http://192.168.1.1:3000?admin=true
7. Notice the "Edit Template" tab is now available. Only administrators can edit templates, e.g. anyone using http://localhost, http://127.0.0.1, or using ?admin=true.

Other things to test:

• Try clicking on a Node to make sure the any field changes are reflected.
• Try editing an Edge to make sure field changes are reflected.
• Make sure Node Table displays all the fields and especially the field names properly -- Try changing the template displayName and make sure the tables reflect the new display name.
• Make sure Edge Tables also display fields properly

CAVEATS / Known Problems

• Editing the template while a Node or Edge is being edited will result in js errors. Fixed as of 1/26/22.
• Import/Export is currently broken. Don't bother to test it. It requires a fair amount of remapping to properly use the template exportLabel values. We will tackle this later.
• We are not doing any data conversions with template changes. This has different implications depending on the specific data object:
  • node/edge id -- The json-editor will only allow you to edit the displayLabel and exportLabel. Other parameters are hidden.
  • node/edge displayLabel -- This should properly affect the displays in the Node Editor, Edge Editor, Node Table, and Edge Table.
  • nodetype/edgetype options -- Changing the type label and color will affect the graph display as soon as you save the template. Changing an existing label will cause some issues: a) any node/edge that uses the removed label type will still retain the original label, but that label will not be displayed in the Node Editor menu -- it will still be displayed in the Node Table, b) you need to manually select each node/edge and change its type to the new label. This is probably something we want to discuss/revisit -- do we need to add migration/conversion tools so that you can combine two into one for instance?
  • defining a new field -- While the json-editor will not allow you to define a new field, you can do so when manually editing the TOML file. However, that field will be ignored by the system and in fact may cause crashes.
  • removing an existing field -- While the json-editor will not allow you to delete a field, you can do so when manually editing the TOML file. However, removing existing fields may in fact cause crashes.
  • changing a field type -- The field type (e.g. string, number, boolean, select) should be done with care. Since existing datasets are not converted, you may end up with errors. Also the select type would expect options to be defined.
  • changing help -- Changes to help should be immediately reflected in the system.

---

---

---

WIKI DOCUMENTATION

New Template Format

As of version 1.4 (this version), the new template format now uses TOML, not JSON. The TOML format is a much more human-friendly file format for settings.

The new template files have an extension of *.template.toml.

TOML files can be edited with any text editor. Visual Studio Code has an extension "Even Better TOML" that will provide syntax highlighting and linting for TOML files.

The old *.template JSON templates are no longer necessary, but you probably want to keep them as reference for converting old projects.

The old template format was much in need of a refresh, having gone through many evolutions and expanded its purpose over time. So in addition to the file format change, we also streamlined some of the configuration fields. This will require you to do some manual conversion of old template files.

---

Editing Templates

IMPORTANT:

• Only administrators can edit templates, e.g. anyone using http://localhost, http://127.0.0.1, or using ?admin=true.
• Editing a template modifies the *.template.tomlfiles that are saved on the server.
• When a template is being edited, Node and Edge editors are disabled.
• If you're in the middle of editing a Node or Edge, the Template editors will be disabled.

To edit a template:

1. Start NetCreate on the server
2. Log onto NetCreate as an admin
3. Select the "More..." tab
4. Select the "Edit Template" tab

You can edit the current template in a variety of ways.

• Edit Node Type
• Edit Edge Type
• Edit Current Template
• Download Current Template
• New Template
• Import Template File

"Edit Node Types" / "Edit Edge Types"

Click on the "Edit Node Types" button to edit the node types defined in the current template. Click on the "Edit Edge Types" button to edit the edge types defined in the current template.

"Edit Node Types" and "Edit Edge Types" allows you to quickly edit just the node type or edge type instead of having to edit the whole current template (that's what "Edit Current Template" does). The basic functionality is the same as "Edit Current Template".

You can set:

• label of the node/edge type -- This is what is displayed in the Node Editor.
• color of the node/edge type
• Add new option
• Remove an existing option
• Change the order in which the types appear.

When you are happy with the changes, click "Save". The template will be updated, the graph redrawn, and the changes written to the current template file on the server.

What happens when you change the label? The label field determines what is displayed in the popup menu for Type in the Node Editor or Edge Editor. If you change an existing label:

• The updated label will appear in the Node Editor / Edge Editor type menu in place of the old label.
• Any existing node / edge that is using the label is not converted. So if you open any existing node / edge that uses the old label in a Node Editor or Edge Editor, it will open with a blank type (since the old selection is no longer available for populating the menu). The node / edge will still the original label information (which you can see if you look at the label in the Node or Edge Table). You need to manually select and edit the Node and reselect the new label type to change the type.

New "color" setting for Edge Types

• Colors for edge types was partially defined and implemented previously. With the template change, it was easy to add the feature. So now you can define a different line color for each edge type. Keep in mind the transparency settings will affect how the edge colors appear.

"Edit Current Template"

Clicking "Edit Current Template" will open the current template in the json-editor. You can then change any value in the template, and hit "Save" to save the changes.

IMPORTANT:

• EXPERTS ONLY!!! Do not edit the whole template if you do not know what you're doing!!! You can inadvertently break the template and dataset with the wrong change.
• The form provides some context and help for each field.

Missing Properties: Use "Object Properties"

The json-editor only displays properties that have been defined in the *.template.toml file. In some cases, if for instance you've imported/converted a JSON file or created a new TOML template file from scratch, you may find that you had inadvertently ommitted a property that you would like to change. These properties will not be available in the editor for modification.

In these cases, you CAN actually add the property by clicking on the "Object Properties" button next to the header section. For instance, if you find yourself wanting to set the searchColor but it was not included in the current template, and is therefore not visible, you would:

1. Edit Current Template
2. Click "Object Properties" next to the main "NetCreate Tempalte" heading
3. Scroll down and check the "searchColor" property
4. The parameter should immediately appear in the editor (you might have to scroll down to see it)

This should only allow you to add properties that have been previously defined in the schema.

"Download Current Template"

Clicking "Download Current Template" will allow you to download the currently loaded template to a file on your local hard disk via your browser. The file will have the same name as the template file on the server.

The downloaded file is in TOML format.

You can use this to save backups or as a starting point for defining new templates.

"New Template"

Clicking "New Template" will replace the currently loaded template with a new blank template cloned from the _default.template.toml file in build/app/assets/templates/_default.template.toml. If you save the file it will be saved with the name of the current dataset. E.g. if you had tacitus.json open, and clicked "New Template" then saved the changes, you would now have a tacitus.template.toml file.

The main use model here is if you are creating a new dataset, try some changes, and decide you don't like them and want to start fresh with a new template, you can do so without quitting and restarting the app and deleting the old template file.

"Import Template"

Click "Choose File" and then select a *.template.toml file to replace the current template file with a new TOML template file.

After you import the template, the editor will be shown. Review it to make sure it's what you want. Make any changes. Then click "Save" to save the template to the server and reload the dataset with the new template.

NOTE: If you don't click "Save", the imported template will not be applied to the current dataset. You can just click "Cancel" to leave your original template as-is.

This is probably the easiest way to create a new dataset and clone the template from another dataset without restarting the server.

"Save" / "Cancel" Buttons

All of the template editors will show a "Cancel" and "Save" buttons once you've started editing.

• "Cancel" will exit the node type editing without saving the changes.
• "Save" will save the template changes, apply them to the current dataset, and write the changes to the server.

Saves will overwrite the exisitng <projectname>.template.toml file. The previous version will be saved with a timestamp, so it should be relatively easy to revert if you need to. But we'll need to keep an eye on accumulating versions and perhaps figure out a solution for this.

---

SCHEMA

JSON-Editor requires a JSON schema definition in order to properly render the template for editing. The schema is defined in build/app/view/netcreate/template-logic.js. The schema is intimately tied to the internal data structures, so we do not recommend changing the schema.

Any template data that are loaded rely on the schema for validation and formatting.

This is a description of the schema.

General Template Settings

These are general settings that apply to the whole template and are saved at the root level of the data structure.

• name -- The name of the template. This is displayed in the template editor.
• description -- A short description of the template. This is displayed in the template editor.
• citation
  • text -- Text to display when clicking "Cite Node" or "Cite Edge", e.g. "H213 Prokopios Network"
  • hidden -- When true, hides the "Cite Node" and "Cite Edge" buttons.
• requireLogin -- Users are required to log in before they can view the graph.
• hideDeleteNodeButton -- When true, the "Delete Node" button is removed for admins. Normally only admins can delete nodes. This will disallow deleting for everyone if you need to prevent deletions.
• duplicateWarning -- Warning message to display if user is trying to create a node that already exists
• nodeIsLockedMessage -- Warning message to display if user is trying to edit a node that someone is already editing
• edgeIsLockedMessage-- Warning message to display if user is trying to edit a edge that someone is already editing
• nodeDefaultTransparency -- Default transparency for nodes. You usually want this set to 1.0 otherwise, edges will be displayed behind a translucent node.
• edgeDefaultTransparency -- Default transparency for edges.
• searchColor -- Outline color of nodes selected via search.
• sourceColor -- Outline color of node highlighted during auto complete.

Fields

The data fields for each node and edge are defined in nodeDefs and edgeDefs.

Each Node Definition and Edge Definition field has required subfields.

• type -- Javascript object type, e.g. string, number, boolean, select
• displayLabel -- Label that is displayed in the Node / Edge Editor and in the Node / Edge table headings.
• exportLabel -- Label to use for exported csv file field names, e.g. you can have a displayLabel of Id, but export it as ID
• help -- Help text to display on the Node / Edge Editor form (?) button hover.
• includeInGraphToolTip -- nodes only -- When true, this data in this field will be displayed when the user hovers over the node on the graph.
• hidden -- When true (checked), this field will not be displayed in the Node / Edge Editor or Node / Edge table.

The select type requires options

Options

Only for fields with type = 'select' you also need to define the options that are availabel. Each option consists of:

• label -- the text label that is displayed in the menu and saved with the node/edge type
• color -- hex color string, e.g. #FF0000 is red.

Built-in Fields

In addition to the custom subfields, there are a number of built-in subfields. These are in the template primarily so the author can specify the exportLabel for these fields. These fields are created automatically if they are missing. Their values are automatically set by the system.

• id
• source -- edge only
• target -- edge only
• created
• updated

Migrating from JSON to TOML

If you are trying to work with a circa v1.3 data set that uses the old JSON template format, here are a few tips:

1. Use Auto-Conversion

If you have a matching <projectname>.template JSON file, the app will automatically try to load the old template format and convert it to the new TOML format, saving the file as <projectname>.template.toml.

• The app expects the <projectname>.template file to be in the same folder as the <projectname>.loki file, usually in build/runtime
• The old *.template file will be left on the server so you might want to remove it once you've converted and confirmed that the new template works.
• This isn't VERY well tested, so there may be some older template variants that will fail the conversion. Please make sure you review the converted file to make sure it makes sense.
• After conversion, you can use the "Edit Current Template" tool to review and modify the template using the built-in json-editor.

2. Manual Migration

You can of course decide to update the template files by hand. I would recommend starting with the _default.template.toml file and copying over changes from the old template rather than starting from scratch. Here's a summary of the changes in the file format.

• File format changed from JSON to TOML
• Extension renamed from *.template to *.template.toml
• General template and app settings have been moved to the top level so that nodeDefs (formerly nodePrompts) and edgeDefs (formerly edgePrompts) now only contain field definitions and not other settings:
  • duplicateWarning has been moved out of nodePrompts.label to the top level
  • sourceNodeIsLockedmessage has been renamed nodeIsLockedMessage and moved to the top level
  • edgeIsLockedMessage has been moved out of edgePrompts and moved to the top level
  • node defaultTransparency has been renamed nodeDefaultTransparency and moved out of nodePrompts.defaultTransparency to the top level
  • edge defaultTransparency has been renamed edgeDefaultTransparency and moved out of edgePrompts.defaultTransparency to the top level
  • citationPrompts has been renamed citation.
  • citationPrompts.citation has been renamed citation.text
• nodePrompts are now nodeDefs
• There are a number of new nodeDef fields:
  • id is now included in the template, mostly to provide a mapping for exportLabel, but also for parity across fields. It is a built-in field and managed by the system.
  • updated is another built-in field managed by the system. The parameters are used for graph tooltips and exportLabels.
  • created is another built-in field managed by the system. The parameters are used for graph tooltips and exportLabels.
• For each regular nodeDef field:
  • type is now required and defaults to string
  • label is now displayLabel to distinguish it from export label.
  • exportLabel is a new field for setting the field name/header for exported csv files. e.g. use this to name the header ID rather than the Id displayed in the Node and Edge Tables.
  • help is the same
  • includeInGraphTooltip defaults to true if not defined
  • hidden is the same
• edgePrompts are now edgeDefs
• There is a new edgeDef field:
  • id is now included in the template, mostly to provide a mapping for exportLabel, but also for parity across fields. It is a built-in field and managed by the system.
• Each regular edgeDef field now also inlcudes (same as nodeDef fields):
  • type is now required and defaults to string
  • label is now displayLabel to distinguish it from export label.
  • exportLabel is a new field for setting the field name/header for exported csv files. e.g. use this to name the header ID rather than the Id displayed in the Node and Edge Tables.
  • help is the same
  • hidden is the same
  • NOTE that edgeDef fields do not use the includeInGraphTooltip field.
• For both nodeDef and edgeDef fields set to type = "select"(e.g. node type and edge type), the options definitions no longer require an id field. This was never really used and removing it simplifies the system.

[benloh]

@kalanicraig @jdanish I think the template editing is in a stable enough state for you to give it a whirl. It's definitely still rough around the edges. Please make sure you read the CAVEATS section. But let me know if this is what you had in mind.

In the mean time, I will turn towards figuring out how to handle changes to the options (e.g. combining or spiltting) and fix the edge cases.

[netcreateorg]

Awesome, thanks. Confirming we have seen this and have begun tinkering. Looks awesome.

[benloh]

@kalanicraig @jdanish Template editing is now disabled when editing a node or edge. And node or edge editing is now disabled when editing a template. Please pull the latest to test.

To Test Template/Node Edit Disabling

1. Pull latest
2. Start app
3. Log in as adminstrator
4. Click on "More..." and select "Edit Template"
5. You should see the Template edit buttons
6. Click on a Node in the graph.
7. Note that the "Edit Node" button on the selected node is enabled
8. Click on the "Edit Node" button.
9. Note the Template buttons are now removed and a message is displayed.
10. Click on an Edge
11. Click on "Edit Edge"
12. Cancel the Node edit. Note the Template buttons are still hidden.
13. Cancel the Edge edit. Note the Template buttons are now visible.
14. Click on "Edit Node Types"
15. Note that editing nodes and edges is now disabled:
    • "Add New Node" is disabled
    • "Edit Node" is disabled
    • "Edit Edge" is disabled for any open Edge.
    • "Add New Edge" is disabled
16. Click on "Save" or "Cancel" the Template edit.
17. Note the node and edge edit buttons should now be re-enabled.

I believe this covers most of the situations. Please let me know if I missed one.

[benloh]

@kalanicraig @jdanish Just FYI there's a known bug in the JSON-to-TOML migration where "type" fields with options (e.g. node type, or edge type) are inadvertently converted to "strings", so the popup menu does not display in the Filters. This is fixed in da520dc1177c40b56995fc49745ce9be9f15a767 in the dev-bl/flatten branch

[kalanicraig]

General Display Issues

• Node/edge table listings and node labels in graph are greyed out

Template Guide/Form issues

• Is "template name" a no-space name sort of thing? Needs guidance/example akin to Template Description
• How does delete behavior work? Form suggests that "hideDeleteNodeButton" will hide for all non-admins and hide for regular users but it seems like it just hides/shows for anyone with an access token. The dropdown should probably be "No Delete/Admin Delete/All Users Delete" or similar.

Template Default issues

• On template edit of any kind: if there are no edge colors selected, edges defaults to black. All edges should default to gray.
• New template does not auto-create an "unselected" node and edge type. Is this an expected behavior

  ?

[benloh]

Node/edge table listings and node labels in graph are greyed out

This is probably a result of default transparency migration problem that Joshua and I were discussing here: https://github.com/netcreateorg/netcreate-2018/pull/198#issuecomment-1024474072

If you change the nodeDefaultTransparency to 1.0 I think this problem should be fixed for now.

But popping up a level, we do need to revisit the use of transparency. I'll create a new issue for that: #199

Is "template name" a no-space name sort of thing? Needs guidance/example akin to Template Description

This can be anything and use spaces. It is only what is displayed on-screen.
The file name is currently determined by the dataset file that you load. I'm guessing that the two of you will want to tweak some of the defaults and descriptions. Currently these are all set in template-logic.js. I'll pull out the schema so it can be easier to edit in isolation.

How does delete behavior work? Form suggests that "hideDeleteNodeButton" will hide for all non-admins and hide for regular users but it seems like it just hides/shows for anyone with an access token. The dropdown should probably be "No Delete/Admin Delete/All Users Delete" or similar.

This should be a checkbox. A lot of little details like this to be set in the schema.

The current delete behavior should be this:

• If you're logged in on http://localhost you can delete
• If you're logged in on http://127.0.0.1 you can delete
• If you're logged in on http:/<url>?admin=true you can delete

So unless you were accessing the server on another computer AND did not have ?admin=true, the delete button should not have shown. If that's not the case, then something is broken.

Are you saying that you would like to enable deletion for people other than admins?

On template edit of any kind: if there are no edge colors selected, edges defaults to black. All edges should default to gray.

This might need to get resolved with #199. Currently edges default to black, but the default transparency of 0.5 lightens the line. But all of this needs to get coordinated with the transparency settings in #199. I'll try to change the default.

New template does not auto-create an "unselected" node and edge type. Is this an expected behavior

I'm fixing this right now. See https://github.com/netcreateorg/netcreate-2018/pull/198#issuecomment-1024362990

[benloh]

Issue: Node/Edge vs Template Edit disabling does not work over the network.

[benloh]

Node/Template disabling fixes are in #200. OK to merge.