DfE-special-educational-needs

GSS-Cogs / family-disability

Family of disability datasets

1 stars 1 forks source link

DfE-special-educational-needs #15

Open grace-spitzer-wong opened 5 years ago

grace-spitzer-wong commented 5 years ago

[x] Create Jenkins job (in Jenkins).
- [x] Create a Jenkinsfile.
- [x] Figure out landing page.
- [x] Create main.py and check scraper works.
- [x] Determine eventual Tidy data column names and add to info.json.
- [x] Map column names to codelists (existing and newly required), adding mapping to info.json.
- [ ] Add definition to reference/columns.csv
- [ ] Add newly required codelists (under reference/codelists) and add info to reference/codelists-metadata.json
- [ ] Add component definitions to reference/components.csv
- [ ] Write a transformation to Tidy CSV and output metadata dataset.trig and schema.

grace-spitzer-wong commented 5 years ago

Codelist created for educational establishments.

grace-spitzer-wong commented 4 years ago

Hi @VTula2000 - the geography is ONS geography. I'll create codelists for the others

VTula2000 commented 4 years ago

SEN_2019_Local_Authority_tables - Observations skipped in final tidy data where LA codes were missing in various tabs eg: NORTH EAST Dimension Education provider includes School type and Academy type Two Dimension Special support type, Special need type, forged to fit multiple tab data into one csv final tidy file

ajtucker commented 4 years ago

There are quite a few duplicate rows in the output, see https://ci.floop.org.uk/job/GSS_data/job/Disability/job/DfE-special-educational-needs/22/execution/node/58/log/

ajtucker commented 4 years ago

Failing some integrity constraints, see https://ci.floop.org.uk/job/GSS_data/job/Disability/job/DfE-special-educational-needs/29/testReport/

ajtucker commented 4 years ago

@VTula2000 , can you change Dfe-Sex to Sex and use the M, F, T values instead of B (boy), G (girl), T and all?

For the Special Education Provider column, would you be able to generate a codelist as part of the transformation? Is there a way to figure out from the spreadsheet what the hierarchy is?

LPerryman commented 4 years ago

Codelists created and data published, structure of data needs review

grace-spitzer-wong commented 4 years ago

Checking dimensions- works and showing on PMD but some dimensions likely to need to be split up

VTula2000 commented 4 years ago

This is more about user acceptance of final output compare to splitting dimension, a review of reference table to describe observations is required like eg: In dimension DfE Special Education Support Type - 1 label to be represented as school year 1 etc

MartynBSpooner commented 4 years ago

Require clarification of dimensions - possibly with publisher - http://gss-data.org.uk/dataset?uri=http%3A%2F%2Fgss-data.org.uk%2Fdata%2Fgss_data%2Fdisability%2Fdfe-special-educational-needs