How to handle required fields (e.g. in merged datasets)

[m-mohr]

Originally posted by @m-mohr in https://github.com/fiboa/specification/issues/13#issuecomment-2051843988

Another issue that we need to discuss: What happens when files are merged that have different extensions implemented with required fields? The current "required" implementation assumes non-nullable fields, which in case of a merge fails.

@andyjenkinson wrote:

To be honest I'd been treating multiple collections as out of scope for now because it's much more complex than the examples considered to date. After all that's what our system actually does - we merge many separate collections (we call them "sources") and serve them out through the API in multiple ways:

1. As sets of "boundary references", each of which is a single geometry + metadata "opinion" about a boundary.
2. As a deduplicated and merged set l, each feature being a single normalised geometry containing multiple metadata objects (one per unique reference) for all of the stuff like determinationMethod, dates, IDs etc to live.

There are many issues with the current Fiboa data model for representing the latter "multiple collections in a single file" case, since the values of most of the fields can different between collections.

Not sure what this has to do with this topic though so probably best to talk it through separately

@cholmes wrote:

for the 'required' assumption it does seem like ideally people could choose when they merge if they want the new collection to require everything. But yeah, it does get tricky, as you ideally want the schema check on the parts of the merged file that do include the extension fields.

I wrote:

Either we declare extensions on a row-level, or we could not require fields in extensions or we add an option to the CLI to not check for required fields in extensions. Or you can only merge if all requirements are met and all other fields are omitted from the resulting dataset.

[m-mohr]

Related issue with more info in the comments: https://github.com/fiboa/schema/issues/3#issuecomment-2321346585

How to handle required fields (e.g. in merged datasets)

• Provide stac_extensions per collection
• All columns in Parquet nullable, don’t check nullable in datatypes, only check it in data validation
• Validate per collection (or row)

Generally: How to combine/merge datasets

• We don‘t handle „geometry matching“, simple „append“ operation except for
• Dataset creator can choose where to place properties that have the same value
• Per-dataset collection extensions
• Collection id -> UID if not present -> deconflict/error if fiboa_extensions is not the same
• CLI automatically moves properties around
• Same values => collection properties
• Merging datasets => move collection properties to columnar data
• CLI should allow the following options for a merge with regards to the columns:
  • Union (A, B + B, C => A, B, C)
  • Intersection (A, B + B, C => B) -> warn user that it may be invalid and he needs to select a valid subset
  • Selection (user selects B and C: A, B + B, C => B, C)

Examples:

# merged
{
  "fiboa_version": "0.3.0",
  "fiboa_extensions": {
    "abc": [
      "https://example.com/provider/schema.yaml"
    ],
    "xyz": [
      "https://example.com/provider/schema.yaml",
      "https://example.com/xyz_crop_extension/schema.yaml"
    ]
  },
  "fiboa_collection": {
    "determination_method": "surveyed"
  }
}

# single
{
  "fiboa_version": "0.3.0",
  "fiboa_extensions": [
    "https://example.com/xyz_crop_extension/schema.yaml",
    "https://example.com/provider/schema.yaml"
  ],
  "fiboa_collection": {
    "collection": "xyz",
    "provider:name": "XYZ",
    "determination_method": "surveyed"
  }
}
{
  "fiboa_version": "0.3.0",
  "fiboa_extensions": [
    "https://example.com/provider/schema.yaml"
  ],
  "fiboa_collection": {
    "collection": "abc",
    "provider:name": "ABC",
    "determination_method": "surveyed",
  }
}