DfT-Journey-times-to-key-services-by-lower-super-output-area

LPerryman commented 4 years ago

4 ODS files. 1 Sheet per file but still a table that needs to be transformed

LPerryman commented 3 years ago

GitHub https://github.com/GSS-Cogs/family-towns-and-high-streets/tree/master/datasets/DfT-Journey-times-to-key-services-by-lower-super-output-area Jenkins https://ci.floop.org.uk/job/GSS_data/job/towns-high-streets/job/DfT-Journey-times-to-key-services-by-lower-super-output-area/

LPerryman commented 3 years ago

Having massive issues with transforming as data is huge especially the first file JTS0501: Travel time, destination and origin indicators for Employment centres by mode of travel, Lower Super Output Area (LSOA), England (ODS, 67.3MB)

LPerryman commented 3 years ago

Have published journey times from first dataset: JTS0501: Travel time, destination and origin indicators for Employment centres by mode of travel, Lower Super Output Area (LSOA), England (ODS, 67.3MB) 295596 observations

LPerryman commented 3 years ago

All 9 Journey time datasets have been published on PMD4. note only the journey time has been published other information has been left out at this time. Can only publish one measure type at present.

Journey times to key services by lower super output area: Employment Centres - JTS0501 https://staging.gss-data.org.uk/cube/explore?uri=http%3A%2F%2Fgss-data.org.uk%2Fdata%2Fgss_data%2Ftowns-high-streets%2Fdft-journey-times-to-key-services-by-lower-super-output-area%2Femployment-centres-catalog-entry

Journey times to key services by lower super output area: Primary Schools - JTS0502 https://staging.gss-data.org.uk/cube/explore?uri=http%3A%2F%2Fgss-data.org.uk%2Fdata%2Fgss_data%2Ftowns-high-streets%2Fdft-journey-times-to-key-services-by-lower-super-output-area%2Fprimary-schools-catalog-entry

Journey times to key services by lower super output area: Secondary Schools - JTS0503 https://staging.gss-data.org.uk/cube/explore?uri=http%3A%2F%2Fgss-data.org.uk%2Fdata%2Fgss_data%2Ftowns-high-streets%2Fdft-journey-times-to-key-services-by-lower-super-output-area%2Fsecondary-schools-catalog-entry

Journey times to key services by lower super output area: Further Education - JTS0504 https://staging.gss-data.org.uk/cube/explore?uri=http%3A%2F%2Fgss-data.org.uk%2Fdata%2Fgss_data%2Ftowns-high-streets%2Fdft-journey-times-to-key-services-by-lower-super-output-area%2Ffurther-education-catalog-entry

Journey times to key services by lower super output area: General Practices - JTS0505 https://staging.gss-data.org.uk/cube/explore?uri=http%3A%2F%2Fgss-data.org.uk%2Fdata%2Fgss_data%2Ftowns-high-streets%2Fdft-journey-times-to-key-services-by-lower-super-output-area%2Fgeneral-practices-catalog-entry

Journey times to key services by lower super output area: Hospitals - JTS0506 https://staging.gss-data.org.uk/cube/explore?uri=http%3A%2F%2Fgss-data.org.uk%2Fdata%2Fgss_data%2Ftowns-high-streets%2Fdft-journey-times-to-key-services-by-lower-super-output-area%2Fhospitals-catalog-entry

Journey times to key services by lower super output area: Food Stores - JTS0507 https://staging.gss-data.org.uk/cube/explore?uri=http%3A%2F%2Fgss-data.org.uk%2Fdata%2Fgss_data%2Ftowns-high-streets%2Fdft-journey-times-to-key-services-by-lower-super-output-area%2Ffood-stores-catalog-entry

Journey times to key services by lower super output area: Town Centre - JTS0508 https://staging.gss-data.org.uk/cube/explore?uri=http%3A%2F%2Fgss-data.org.uk%2Fdata%2Fgss_data%2Ftowns-high-streets%2Fdft-journey-times-to-key-services-by-lower-super-output-area%2Ftown-centre-catalog-entry

Journey times to key services by lower super output area: Pharmacies - JTS0509 https://staging.gss-data.org.uk/cube/explore?uri=http%3A%2F%2Fgss-data.org.uk%2Fdata%2Fgss_data%2Ftowns-high-streets%2Fdft-journey-times-to-key-services-by-lower-super-output-area%2Fpharmacies-catalog-entry

LPerryman commented 3 years ago

Due to the amount of data being transformed and time issues the source data has been processed offline and output to google docs. The python script processes these files instead of from source. Will cause issues with new releases but DEs are experimenting with code to speed thing up.

Tracey-B commented 3 years ago

BA comments: The dataset has been split into 9 different datasets on PMDv4 due to the size of the data and only currently contains travel times due to issues with multiple measures. Stage 1 & 2 tech spec combined There are no active filters. Incorrect contents issue date of 11/12/2018 has been used the latest published date is 19/12/2019. Is Field Code the correct name for a dimension?
Travel time to 15+ decimal points?

LPerryman commented 3 years ago

'Employment Centre Size' dimension has been added to the 'Employment Centre' dataset and 'Field Code' dimension renamed 'Mode of Travel' and codelist altered to ('PT/Walk', 'Cycle', 'Car'). All 9 datasets have been republished.

Robsteranium commented 3 years ago

Hi.

It doesn't look like the pickers are working for any of these datasets: JTS0501 for example.

Indeed it looks like the dimension properties haven't been defined (or at least any that are defined aren't using the same URIs as those in the DSD).

Robsteranium commented 3 years ago

Also seems to be a typo with the employment-centre-size codelist:

?code skos:inScheme <http://gss-data.org.uk/def/concept-scheme/lemployment-centre-size>

Ought to be:

?code skos:inScheme <http://gss-data.org.uk/def/concept-scheme/employment-centre-size>

example

The slug looks right in code and graph URIs. I wonder if it's possible to DRY-up the generation process to avoid copy-paste mistakes?

GSS-Cogs / family-towns-and-high-streets

DfT-Journey-times-to-key-services-by-lower-super-output-area #2