search

TerriaJS / terriajs

A library for building rich, web-based geospatial data platforms.
https://terria.io
Apache License 2.0
1.16k stars 360 forks source link

Model Architecture: Implement SdmxCatalogItem #3524

Closed steve9164 closed 3 years ago

soyarsauce commented 4 years ago

Ping Rowan or Nanda if the master sdmx stuff is still unclear

AnaBelgun commented 4 years ago

ABS has released the latest data api based on SDMX JSON v2.1 Here are the details to test: https://docs.google.com/document/d/1J3con-CBdBGlQLSelZ_eYwesX2cjYI5UnGfSLMNZg9o/edit# ABS would like for an APIkey to be used with the calls

nf-s commented 4 years ago

SDMX learning materials

REST cheat sheets:

https://raw.githubusercontent.com/sdmx-twg/sdmx-rest/master/v2_1/ws/rest/docs/rest_cheat_sheet.pdf https://sis-cc.gitlab.io/dotstatsuite-documentation/using-api/restful/

Reference

For structure messages see - https://github.com/sdmx-twg/sdmx-json/blob/structure-message-release/structure-message/docs/1-sdmx-json-field-guide.md#agencyscheme

SDMX group structure

Something similar to this - https://ilostat.ilo.org/data/sdmx-query-builder/

https://registry.sdmx.org/webservice/data.html

https://github.com/sdmx-twg/sdmx-json/blob/structure-message-release/structure-message/requirements/browse-by-topic.md

Steps:

APIs

https://sdd-dotstat-api-gateway.portal.azure-api.net/concepts

See https://github.com/sdmx-twg/sdmx-rest for more

Thoughts

Maybe use sdmx-csv instead? - see https://github.com/sdmx-twg/sdmx-csv/blob/master/data-message/docs/sdmx-csv-field-guide.md

Js libs

Started with sdmx-rest, but is only really useful for creating requests.

Master used sdmxjsonlib, but it is quite basic, isn't maintained

dotstatsuite-sdmxjs which is used by .stat suite - looks like the most useful.

At this stage I am not using any libs - as the type definitions do all the hard work. I may reconsider this as I create the SdmxJsonCatalogItem

nf-s commented 4 years ago

I have types for all SDMX data messages, and some SDMX structure messages - https://github.com/nf-s/sdmx-json-types

nf-s commented 4 years ago

Groups are now working in this order:

nf-s commented 4 years ago

image

nf-s commented 4 years ago

SDMX item structure:

All of this information is given by = https://stats-nsi-stable.pacificdata.org/rest/dataflow/SPC/DF_CPI?references=all It may be more efficient to fetch Codelists and concepts at the SdmxGroup level - but this is easier for the moment

nf-s commented 4 years ago

Very basic SDMX item is now working image

Some TODOs:

nf-s commented 4 years ago

Basic region mapping is working (time selector isn't working yet) image

But this isn't a very good visualisation for pacific countries...

nf-s commented 4 years ago

Issue with region-mapping https://github.com/TerriaJS/terriajs/issues/4676 and time-dimension #4674

nf-s commented 4 years ago

ABS region mapping now working image

nf-s commented 4 years ago

It is now working on CI-deployment http://ci.terria.io/sdmx-next/#clean&https://gist.githubusercontent.com/nf-s/24b9288c6cdb6ba41d12f3d269318a0d/raw/c5ed64071c8a651ae458b4c0d3dc79c8618cb86d/sdmx.json

Not all layers will work (especially some ABS ones), but quite a few do. Note issues with region-mapping #4676 and time-dimension #4674

nf-s commented 4 years ago

Some layers will require setting conceptTraits to work correctly - (usually for region-mapping).

Setting up regions

Information about that can be found here - https://github.com/TerriaJS/terriajs/blob/c351cb6c598ce66dc1ed373e7cc811a88766f0c5/lib/Traits/SdmxCommonTraits.ts

RegionType is determined by sdmxJsonDataflowStratum.column in this order:

  1. conceptTraits.regionType
  2. if conceptTraits.type === 'region' and there exists another conceptTraits with type === 'region-type' in the datastructure, therefore region-type will be set from this concept
  3. the dimension id
  4. the concept name
  5. the concept id (the actual string, not URN form)

For example SPC - Option 1 above

The following catalog json specifies that the CS_COMMON(2.0).GEO_PICT concept corresponds to regionType CNT2 (2-digit country code). So whenever this concept is used in a dataflow, the region type will be set to CNT2

 "conceptOverrides": [
        {
          "id": "urn:sdmx:org.sdmx.infomodel.conceptscheme.Concept=SPC:CS_COMMON(2.0).GEO_PICT",
          "type": "region",
          "regionType": "CNT2"
        },
...

Another example, ABS - Option 2 above

Some dataflows use REGION_TYPE concept to specify the region type, and REGION concept to specify the region. If the REGION_TYPE concept is found within a dataflow, it's selectedId will be used as the regionType for the REGION concept.

 "conceptOverrides": [
        {
          "id": "urn:sdmx:org.sdmx.infomodel.conceptscheme.Concept=ABS:CS_C16_COMMON(1.0.0).REGION_TYPE",
          "type": "region-type",
          "selectedId": "SA3"
        },
        {
          "id": "urn:sdmx:org.sdmx.infomodel.conceptscheme.Concept=ABS:CS_C16_COMMON(1.0.0).REGION",
          "type": "region"
        },
...

Options 3 - 5

These are a bit more obvious, and are kind of handled in the same way a column title is used to find the corresponding regionType - for example lga_code_2016 - but it uses the first option which matches a valid regionType

Other ConceptTraits

ConceptTraits extends DimensionTraits so you also get the following properties you can override:

@primitiveTrait({
    type: "string",
    name: "Name",
    description: "Dimension name (human-readable)"
  })
  name?: string;

  @objectArrayTrait({
    type: DimensionOptionTraits,
    idProperty: "id",
    name: "Options",
    description: "Dimension options"
  })
  options?: DimensionOptionTraits[];

  @primitiveTrait({
    type: "string",
    name: "Selected ID",
    description: "Selected Option's ID"
  })
  selectedId?: string;

  @primitiveTrait({
    type: "boolean",
    name: "Allow undefined",
    description: "Allow dimension to be undefined"
  })
  allowUndefined?: boolean;

  @primitiveTrait({
    type: "boolean",
    name: "Disable dimension",
    description: "Hides dimension"
  })
  disable?: boolean;
nf-s commented 4 years ago

Some TODOs:

nf-s commented 4 years ago

HTTP 400 issue may be related to lack of IPv6 support.

nf-s commented 4 years ago

I might add ColumnTraits to ConceptTraits so you can directly add column style/etc... properties to concepts

nf-s commented 4 years ago

Update!

I consider this finished (as a first pass anyway) - I just need to write tests.

There are a few things missing from master implementation:

Other possible features

soyarsauce commented 4 years ago

rowan: test some of the ways we use in nationalmap for it - and if we get the same results then we are good.

nick: main difference w/ v7 implementation, we have sdmx groups, instead of creating a catalog item for every single data flow ideally we just 'browse a sdmx'

@nf-s to update post-Wedneday

nf-s commented 4 years ago

Feedback from meeting

nf-s commented 4 years ago

Other thoughts

nf-s commented 3 years ago

I'm going to close this as https://github.com/TerriaJS/terriajs/pull/4659 has been merged. We can create new issue(s) after we have received more feedback - I will make issues on current feedback

nf-s commented 3 years ago

I'm going to re-open this for - NationalMap - Beta release (V8) - Feb 2021 https://github.com/TerriaJS/nationalmap/issues/997

I wanted to create separate tickets, but I didn't have time

nf-s commented 3 years ago

This is not great: image

Maybe write a featureInfoTemplate using region feature info from https://github.com/TerriaJS/terriajs/pull/5211

nf-s commented 3 years ago
nf-s commented 3 years ago
nf-s commented 3 years ago
nf-s commented 3 years ago

Update!

image

nf-s commented 3 years ago

URL to test it out!
http://ci.terria.io/sdmx-fixes-loading/#clean&https://gist.githubusercontent.com/nf-s/1e05f005a93d7b9ab6f1723b8b40d1d2/raw/f74e8c68f849db10cccfa96b05943cdb9cdb635b/abs-sdmx.json

Known dataset issues

nf-s commented 3 years ago

TODO

Future Improvements

Chart "expanding"

At the moment, when you expand a chart you don't get any information about dimensions

image

nf-s commented 3 years ago

ABS Dataflow issues

Supply Use Table

UNIT_MEASURE is set to OBS_COMMENT concept.
https://api.data.abs.gov.au/dataflow/ABS/ABS_SU_TABLE_2018?references=all

nf-s commented 3 years ago

Issues

nf-s commented 3 years ago

Closing again! Due to https://github.com/TerriaJS/terriajs/pull/5234