Csv-geo-au is growing beyond csv, geo and au

[RacingTadpole]

We are now using csv-geo-au for things:

• beyond “au” (eg. any lat/lon file)
• beyond “geo” (for the date formats allowed in charts)
• beyond “csv” (eg. The ABS has agreed to provide region aliases such as lga_code_2011 in their SDMX-JSON format)

I wonder if we should pull out each of these pieces into their own standards, and define csv-geo-au in terms of them?

[RacingTadpole]

@LegoStormtroopr suggested in another channel:

What you are describing is recording all of the structural metadata around particular columns.

We could record all of that metadata around those Data Elements, and then with very minor tweaking NationalMap could then interpret that metadata to understand how to display data on NationalMap.

Let me see if I understand… do you mean we could format the wordy spec at https://github.com/TerriaJS/nationalmap/wiki/csv-geo-au into a machine-readable form, which TerriaJS can then read?

To which @LegoStormtroopr replied

At the least, for the columns yes.

[RacingTadpole]

Cool! I’m keen to give it a try (with a side motivation to help me get my head around the concept of metadata). How do I start?

[tobybellwood]

This sort of approach came up in a conversation with ABS geography around the concepts of statistical geography yesterday.

I've always been keen to see the csv-geo-* get legs, but hadn't thought of csv-- !

I'm also interested in whether data packages could be used to carry this extended meaning (possibly generated by future dga?) http://frictionlessdata.io/data-packages/

[stevage]

Just fwiw, "geocsv" is some kind of de facto standard used by Mapbox and some others: https://github.com/mapbox/detect-geocsv

[RacingTadpole]

Looks like the crux of that is, first establish that it is csv, then check for these column names:

hasOne(lowerCaseNames, ['wkt', 'geom', 'geometry', 'geojson']) ||
        ((hasOne(lowerCaseNames, ['x', 'lon', 'lng', 'long']) ||
            hasOneThatContains(lowerCaseNames, 'longitude')) &&
        (hasOne(lowerCaseNames, ['y', 'lat']) ||
            hasOneThatContains(lowerCaseNames, 'latitude')));

[RacingTadpole]

@tobybellwood thanks for the data-packages link. That could be useful as another way to tell terriajs how to interpret other files (like our custom "init" json-format does now, but easier to construct). Is that what you meant? I can't see how to use data-packages to define a standard though.

[stevage]

Ok @tobybellwood so I'll just recap how the Data Packages work for everyone's benefit:

• Data Package is a spec that says "you put files together in a place with a datapackage.json that describes them"
• Tabular Data Package is a profile (more specific version) of that spec that says "you put one more CSV files together with a datapackage.json that describes them and includes a JSON Table Schema definition of every field in every CSV"
• JSON Table Schema is a structure that can closely define each field with data types, acceptable values, descriptions etc.

All these structures are extensible, and it's ok to include extra attributes not explicitly defined in the spec.

So, what can we do with it?

• TerriaJS could support Tabular Data Package, so that given the appropriate datapackage.json, it would load one (or or more) CSV files, and know how to style it, describe the columns in the interface etc. This would be an alternative to a data custodian having to produce a full init file, and would probably be more widely useful. We could either allow tableStyle attributes in those data packages, or support something a bit more data-packagey.
• We could define our own profile of TDP for csv-geo-au. (It's not possible to define CSV-geo-au as a tabular data package, for instance, because all fields must be defined.) Our profile would define the possible region types and how they should be referred to in the JSON file. I'm not sure what the benefits would be exactly. Maybe, having done this, when someone publishes a "csv-geo-au data package", they wouldn't need to be as explicit about what some of the fields mean.

[LegoStormtroopr]

I'd go a bit further an suggest that if data.gov.au v2.0 is going to have a metadata registry associated with it, defining and publishing all of the structural metadata would be beneficial for a number of reasons:

1. They would help us demonstrate why metadata is useful to describe in a very granular way.
2. Means geo-csv is less constrained by column headers - if a column maps to a data element which maps to the latitude concept, then NationalMap can utilise it. Likewise, if it is a state, statistical boundary, etc...
3. We could (lossy) publish tabular JSON from the metadata

What we'd need to accomplish this are services to:

• bulk produce structural metadata
• map data to structural metadata
• allow NationalMap to query a metadata store and determine geospatial columns (distinct from
• export TDP schemas from a metadata registry