New parameter type: dataframe like data format

[jkiviluo]

[Updated based on comment and discussion with @soininen --> separating two issues]

Title says a lot, but something where we can have a tight and fast dataframe like data. It should have the possibility to contain array (1-d) or matrix (n-d) data in efficient manner, but should also be able to contain the index headers (including time series). Moving in the array/matrix needs to be fast - basically fixed data element size (int, float, double, etc.). Indexable. Arrow would enable this, there is a separate but related issue: https://github.com/spine-tools/Spine-Toolbox/issues/2506. We could also implement Arrow just for the dataframe like data, but we would be then already halfway to retire JSON blobs all together.

[soininen]

Are you proposing of adding another binary format besides the JSON one? I am not convinced we need that. I would rather see we did something like this:

• add new parameter types for tables (data frames) and matrices
  • in any case we need these
  • we already have 1-d array
  • we can fix element sizes for existing data types
• add new parameter type for linking external data
• test if changing the JSON representation to some other (binary) presentation would be beneficial performance-wise
• implement modules in spinedb_api that produce xarrays or Pandas dataframes
  • we should have embraced the data types used by the community from the very beginning
  • parameter_value should still be around for backwards compatibility

[jkiviluo]

Forgot that we had this: https://github.com/spine-tools/Spine-Toolbox/issues/1992