Flexible plugin system for the STAC metadata editor

[danielfdsilva]

I've been researching and thinking of ways to make the STAC metadata editor more flexible and extensible.

[!NOTE] The provided code examples are just an example and not necessarily complete.

From discussions on EOEPCA/data-access#57 and other meetings I started with the following assumptions:

• Different collections may have different metadata fields and depending on the extensions there may be entire sections that are not present.
• The editor should be able to handle these differences and be able to show the correct fields for each collection.
• Another instance of this metadata editor should be able to define a different structure.
• Even though some extensions define a schema we're not going to use it generate the editor since it makes customization harder (like we'd like to have for the render extension)

I've been thinking of this as a set of plugins that can be loaded to generate the editor. Each plugin would be responsible for a section of the editor and would be able to define the fields that should be shown. This allows for a more modular approach to the editor. Each instance could use a different set of plugins to define the editor structure. These could be custom plugins or plugins from the community.

Drawing inspiration from the JSON schema spec, each plugin would define a schema to create the editor. For each field type there would be a corresponding default widget to render it, but plugins could define their own widgets (in the form of a React component) to be used by the fields.

Example for a plugin:

export class PluginMeta extends Plugin {
  editSchema() {
    return {
      type: 'root',
      properties: {
        title: {
          type: 'string'
        },
        description: {
          type: 'string'
        },
        provider: {
          type: 'string',
          'ui:widget': 'select',
          enum: [
            ['ESA', 'esa'],
            ['NASA', 'nasa'],
            ['JAXA', 'jaxa'],
            ['CNSA', 'cnsa']
          ]
        }
      }
    };
  }

  enterData(data) {
    // Transform the input data to the format expected by the editor.
    // A transformation may be needed in cases where an object key is being edited.
  }

  exitData(data) {
    // Transform the data from the editor to the format expected by STAC.
  }
}

Something more complicated would include an async init function to get values needed for the plugin.

export class PluginRender extends Plugin {
  async init(data) {
    this.colorMaps = [];

    try {
      const response = await fetch(
        'https://dev.openveda.cloud/api/raster/colorMaps'
      );
      const result = await response.json();
      this.colorMaps = result.colorMaps;
    } catch (error) {
      // oops!
    }
  }

  // ... more code
}

All this would be set up in a configuration file that would define the plugins to be used by the editor. The plugins could even be loaded dynamically depending on some condition.

export const config = {
  // Collection level plugins.
  collectionPlugins: [
    new PluginMeta()
  ],

  // Item level plugins.
  itemPlugins: [
    new PluginMeta(),
    new PluginExtension(),
    (data) => data.stac_extensions?.some((e) => e.includes('/render/'))
      ? new PluginRender()
      : null
    ],

  // Custom widgets. 
  'widgets': {
    keyname: KeynameWidget,
    select: SelectWidget
  }
};

The use of custom widgets allows us to hook into the render of a field at almost any level. If, for the render extension, we'd like to have a custom map to preview the changes we could do something like this:

export class PluginRender extends Plugin {
  editSchema() {
    return {
      type: 'root',
      properties: {
        renders: {
          type: 'array',
          items: {
            type: 'object',
            'ui:widget': 'renderExtensionItemWidget',
            properties: {
              key: {
                label: 'Key name',
                'ui:widget': 'keyname',
                type: 'string'
              },
              assets: {
                label: 'Assets',
                type: 'array',
                items: {
                  type: 'string'
                }
              },
              nodata: {
                label: 'No Data',
                type: 'string'
              },
              colormap_name: {
                label: 'Colormap Name',
                type: 'string',
              }
              // ... more fields
            }
          }
        }
      }
    };
  }
}

Then the Widget:

export function RenderExtensionItemWidget(props) {
  const [tilejson, setTilejson] = useState();
  return (
    <Box>
      <Box position='relative' aspectRatio='4/2'>
        <RMap
          mapboxAccessToken={process.env.MAPBOX_TOKEN}
          mapStyle='mapbox://styles/mapbox/satellite-v9'
        >
          {tilejson && (
            <Source tiles={tilejson.tiles} type='raster'>
              <Layer id='data' type='raster' />
            </Source>
          )}
        </RMap>
      </Box>
      <Flex gap={4} direction='column' mt={4}>
        <ObjectField pointer={props.pointer} field={props.field} />
      </Flex>
    </Box>
  );
}

ObjectField is the default widget for objects so basically we're adding a wrapper around it to show the map.

Lastly, the configuration file would be used to generate the editor:

export const config = {
  collectionPlugins: [
    new PluginMeta(),
    new PluginRender()
  ],

  'ui:widget': {
    renderExtensionItemWidget: RenderExtensionItemWidget
  }
};

[oliverroick]

In the proposed approach, the plugins drive what fields will be available in the forms and looks like there could be cases were the editor doesn’t support a STAC extension listed for the collection/item and will therefore be ignored. There’s probably no way around it, the same thing could happen stac-fields, but we should be aware of that.

Vice-versa, there could be cases where plugins add fields to the collection or item document that aren’t in the STAC spec or one of the extensions specified. This isn’t a huge problem for the functionality of the editor but it could create meta data that can’t be read by any STAC client, because they’re not aware of those fields.

Is there a way to verify the list of plugins with the STAC extensions specified for the collection/item?

[danielfdsilva]

@oliverroick That's a good point. It's going to be difficult to capture all edge cases but each instance will always be able to make tweaks to suit its needs.

I was thinking that the config could also accept functions to dynamically determine which plugins to "activate" for a given item/collection. You could make this check as complex as you'd want.

For example:

{
 // Item level plugins.
  itemPlugins: [
    new PluginMeta(),
    new PluginExtension(),
    // Where `data` is the current item being edited.
    (data) => data.stac_extensions?.some((e) => e.includes('/render/'))
      ? new PluginRender()
      : null
    ]
}

Only activate the render plugin if the given item has the render collection.

[j08lue]

We are documenting the above in an Architecture Decision Record (ADR) here:

• https://github.com/developmentseed/stac-admin/pull/17

[danielfdsilva]

After our last meeting about dynamic form sections I did some experiments on how that could work.

For context: A dynamic form section is a plugin whose edit schema depends on the value from another field.

An application for this could be allowing the user to enable different extensions on a collection. Enabling a given extension could cause the form to display a new section with specific fields. For example, enabling the render extension the form would allow the user to create render presets.

This would be done by returning the edit schema depending on form values.

  editSchema(formData) {
    if (formData?.stac_extensions?.some((e: string) => e.includes('/render/'))) {
      return {
        type: 'root',
        properties: {
          renders: {
            type: 'array',
            label: 'Renders',
            items: { /* fields */ }
        }
      } as SchemaFieldObject;
    }

    // Let the form know that there's nothing to show for now.
    return Plugin.HIDDEN;
  }

For performance reasons the form doesn't re-render whenever a value changes, therefore we need to watch the dependent fields to ensure the schema is recalculated whenever those fields change. This is just a matter of specifying the field names to watch.

Full example:

export class PluginRender extends Plugin {
  name = 'Render';

  watchFields = ['stac_extensions'];

  editSchema(formData) {
    if (formData?.stac_extensions?.some((e: string) => e.includes('/render/'))) {
      return {
        type: 'root',
        properties: {
          renders: {
            type: 'array',
            label: 'Renders',
            items: { /* fields */ }
        }
      } as SchemaFieldObject;
    }

    // Let the form know that there's nothing to show for now.
    return Plugin.HIDDEN;
  }

  enterData(data) {
    return {
      renders: Object.entries(data.renders).map(([key, render]) => ({
        key,
        assets: render.assets.map((asset) => ({ value: asset }))
      }))
    };
  }

  exitData(data) {
    return {
      renders: data.renders.reduce((acc, render) => {
        return {
          ...acc,
          [render.key]: {
            assets: render.assets.map((asset) => asset.value)
          }
        };
      }, {})
    };
  }
}

[j08lue]

I wonder what role the stac_extensions array in the STAC collection metadata should play.

Collections can have arbitrary additional metadata {shortname}:{key}={value}, without actually referencing a well defined extension / JSON schema for that data.

For example, NASA VEDA STAC has some "dashboard:is_periodic": true keys

and Microsoft Planetary Computer STAC has things like "msft:storage_account": "daymeteuwest", to my knowledge without specifying anywhere a schema for these properties.

Interestingly, STAC Explorer somehow magically knows how to render these fields:

Just by guessing and some string manipulation, or does STAC Explorer get types etc from somewhere? 🤔

[oliverroick]

STAC Explorer is using stac-fields internally to render the values. Stac-fields doesn't rely on extension spec but a list of hard-coded field options, which is maintained manually. That list has supports for these fields (example). (I've used stac-fields as well in the stac-admin prototype to render the properties on the item pages.

If we wanted to support the same, we could write plugins for those platform-specific fields, there could be a Planetary Computer and VEDA STAC plugin.

[j08lue]

There is no magic then 😞

[j08lue]

While the details will still be refined, the decision for a plugin system has been made.

• https://github.com/developmentseed/stac-admin/pull/17