Blank support in value types

[graft]

Magma value types may be present (value) or missing (null). In many instances it is also useful to mark values as "blank", that is, known to be empty. This presently exists for the file type, which reports a path of ::blank if the file is so marked.

It would be nice if each value in magma, in addition to being "null", could also be marked "blank". (Following on this, components in Timur should clearly distinguish between 'blank/known empty' and 'null/missing' when viewing or editing). However this might not always be possible. Here is some suggestions for each type:

Links:

• parent, child, table, link, collection - Link types are the trickiest, since they are maintained by a foreign key. One option would be that each model maintains a single "blank record", and empty links point to the corresponding blank record - but this only works for outgoing links (i.e., foreign keys, which may point to the blank record), not for collections. My inclination would be not to support blanking here.

Files:

• file - already supports blanking via the path ::blank
• image - ditto

Weirdos:

• match - It doesn't really make sense for a match to be blank since it is mostly used for validation.
• identifier - Similarly identifiers should not be blank.
• matrix - Matrix rows also don't really need to be blank

Values:

• string - Obviously the empty string would mean "intentionally blank", as opposed to null.
• float - we could use NaN for blank as suggested in #188
• integer - One of the most common types, but the hardest one to blank, since postgres doesn't store anything other than integers (i.e., an integer cannot be NaN, etc.)
• boolean - false vs. null already suffices as blank vs. missing.
• date_time - The date 0001-01-01 can probably be safely regarded as the blank date.

[graft]

While discussing the potential blanking of links, it occurred to me that we may save a "blank" value in the database entry for "child", "table" and "collection" links.

Currently there IS no such database entry because it is not required, but we might add one for "blank" records. The danger here is that it might get inconsistent: if there is a "blank" entry in the db column, but also an existing linked record (i.e., the other record has a foreign key into this one). One way to deal is we might (1) try to be consistent (i.e., by zeroing out the 'blank' entry whenever links are made) (2) only consult the db column for blank status if there are no linked records.

[corps]

I don't think this is actually the right solution, and it's worth unwinding why we are using the concept of blank.

If we roll out blank, it has huge implications for all clients. imagine if a float can be nul, blank, or a float value. What is the json encoding of this? Do clients need to make 4 checks? Add this value if it's an actual value. Default it to 0 if it's blank. Throw an error if it's null? Do something else if we get an object but that object isn't the object that encodes blank?

Broadly, we are increasing the complexity of all client interactions with data by multiplying the decisions that must be made by the product of occupants in the algebra.

Value = Scalar | Null | Blank is 50% more complex than Value = Scalar | Null and EVERY program interfacing with magma has to address that.

I think the reason this has worked 'ok' for files is that in truth, not many client programs directly interact with files, and if they do so they tend to just want to parse out a path or download. The distinction between blank and nil isn't important.

Is there an easy to memorize logic for handling null vs blank vs scalar for adding together an array of numbers? Or traversing a graph? What on earth does a null parent vs a blank parent really mean for logic parsing those results?

[corps]

Alternative: what if 'incomplete' is a record level concept? Like a record can be marked as 'not all fields finalized' or 'not passed QC'?

Clients have an easier time filtering for records that should have occupied or null values.

[corps]

Or maybe 'blankness' is orthogonal to a value's entry? So for instance, a blank value is really a null value, but in a separate map, the 'qc' space of wether fields are ready or not, and not conflated with 'values' themselves?

Hmmm.