search

IO500 / webpage

Sources for the io500.org website pages.
6 stars 4 forks source link

JSON for past lists incompatible with current format #278

Open gflofst opened 5 months ago

gflofst commented 5 months ago

The JSON you can download for the current year is understandable. For previous years, it is not updated to the current format and it is difficult to impossible to parse and understand what the data is. We need to update all of the old JSON files to the current format and organization to enable data analysis more easily.

gflofst commented 5 months ago

@JulianKunkel proposal:

  1. Use the https://github.com/VI4IO/cdcl-schema-tools tool to convert the old JSON files to the newest format
  2. Add flat schema to the new JSON schema
  3. add the conversion rules from the complex schema to the flat schema
  4. Use the existing tools to convert schema into a csv/excel schema to enable downloading a format for easy analysis
gflofst commented 5 months ago

@seattleplus We need to ensure that schema changes are reflected automatically in the tools such that new fields are automatically in the excel version rather than having to modify tools @JulianKunkel we need to add a schema version to make future conversions easier.

@gflofst says we need to document the data conversion process to see what we can do on the server vs. asking the client to do via examples. @gflofst we need to have a checklist for each list to include things like updating the schema and tools.