Insufficient rules on custom information to allow data merging

[Ichoran]

We don't have a precise specification for custom data. This is a problem if we want to be able to merge and split time series--what do you do with custom data?

I propose we include the following specification for custom fields in the data section:

• If the data is arrayed, each value for an associated custom key must either be an array of the length of the number of timepoints, or a single value that is assumed to apply to every timepoint.
• If the data is not arrayed, there are no restrictions on values
• When data is split by time, the custom values that are arrays are split at the same indices
• When data is merged, arrays are concatenated, constant values are collapsed if they are the same, and are duplicated to every timepoint if they differ. If keys are present for some timepoints and not others, the missing timepoints will be filled in by JSON null.

This way the custom JSON data behaves the same way as the time series numeric data. (In particular, like the origin data where you can set a single origin for an arrayed time series.)

[Ichoran]

I have tried to write something along these lines in the documentation in #146 but it's not yet implemented by readers.

[MichaelCurrie]

I really like this idea. For now the Python parser just drops any custom fields as soon as the file is read, leaving it to other more specialized readers to handle the custom fields.

Being able to merge them in a way that makes sense would make the readers more useful for labs, and would mean they wouldn't have to specialize the readers at all, they could just deal with the custom fields they are interested in once the object is in memory.

So now all that's left is implementing it!

[Ichoran]

This is fully implemented in Scala (save for bugs) in #152