search

open-innovations / jrf-insight

JRF North England Insight Finder
https://open-innovations.github.io/jrf-insight/
MIT License
1 stars 0 forks source link

Work out how to data harvest from Stat-Xplore #6

Closed luke-strange closed 1 year ago

luke-strange commented 1 year ago

Pull all 6 tables, do not make any transformations yet.

HBAI = Households below average income

luke-strange commented 1 year ago

Need to figure out how to iterate through the different variables in each data set.

store and name them as Datatset1_table1_wafer1.

luke-strange commented 1 year ago

we want the lowest geography level possible.

luke-strange commented 1 year ago

So far I have written a script called probe.py located in pipelines/extract/. This script requests a list of the folders in statXplore, the databases in these folders, and the variables within each database. The variables include counts and measures (facts), and groups/fields/ (dimensions). These are stored in named folders under data/lookups.