For a hub to be successfully accessed as an arrow dataset, column data types should not change from round to round.
Generally many task IDs that are covered by our schema shouldn't change data type in further rounds as that's somewhat fixed by the schema. Custom task IDs however, which are beyond our control, and the output_type_id column have the potential to change and this could indeed cause problems downstream. This is mainly a problem for parquet files (but has a small chance to cause problems in csvs too).
This is why early on when we had lots of discussions about this we allowed for a hub to override the automatic detection of the output_type_id data type in the hubData::create_hub_schema() function, used to determine the overall hub schema from the tasks.json config file. To future proof the output_type_id admins could set the value of output_type_id_datatype to the safest, most future proof data type, i.e. character.
Set value of arg output_type_id_datatype in config
To enable hub admins to be able to configure and communicate this setting at a hub level, I propose an output_type_id_datatypeenum optional property at the top level of tasks.json config (i.e. sibling to rounds and schema_version) that would accept valid hubData::create_hub_schema()output_type_id_datatype argument values. If this property exists in the config, the default behaviour (unless specifically overridden when calling) of hubData::create_hub_schema() would be to set the output type id column to the data type specified in the config.
This would give admins the ability to future proof their hubs by setting the column to character if they are unsure whether they may start collecting an output type that could affect the schema.
Background
For a hub to be successfully accessed as an arrow dataset, column data types should not change from round to round. Generally many task IDs that are covered by our schema shouldn't change data type in further rounds as that's somewhat fixed by the schema. Custom task IDs however, which are beyond our control, and the output_type_id column have the potential to change and this could indeed cause problems downstream. This is mainly a problem for parquet files (but has a small chance to cause problems in csvs too).
This is why early on when we had lots of discussions about this we allowed for a hub to override the automatic detection of the
output_type_iddata type in thehubData::create_hub_schema()function, used to determine the overall hub schema from thetasks.jsonconfig file. To future proof theoutput_type_idadmins could set the value ofoutput_type_id_datatypeto the safest, most future proof data type, i.e. character.Set value of arg
output_type_id_datatypein configTo enable hub admins to be able to configure and communicate this setting at a hub level, I propose an
output_type_id_datatypeenumoptional property at the top level oftasks.jsonconfig (i.e. sibling toroundsandschema_version) that would accept validhubData::create_hub_schema()output_type_id_datatypeargument values. If this property exists in the config, the default behaviour (unless specifically overridden when calling) ofhubData::create_hub_schema()would be to set the output type id column to the data type specified in the config.This would give admins the ability to future proof their hubs by setting the column to character if they are unsure whether they may start collecting an output type that could affect the schema.