[DAPS training] TTWA Travel pipeline

[jaklinger]

NEVER commit to the dev or master branch directly.

• [x] Clone dev, create a new branch with a short sensible name e.g. 193_ttwa, prefixed with this issue number
• [x] Create an empty package under nesta.packages.geographies called uk_centroids.py, and under nesta.packages.travel_time called travel_matrix.py.
• [x] Commit and create a draft PR for this branch.

Using TDD, commit regularly after this stage, regularly merging in dev and feature branch (193_ttwa). NEVER merge into dev, always merge from dev

• [x] Write package code which which maps TTWAs --> LSOAs by id, and maps LSOA --> OA by id, based on open data e.g. https://geoportal.statistics.gov.uk/search?collection=Dataset&sort=name&tags=LUP_LSOA_TTWA without hardcoding!, including tests [ ]Outstanding: lookup LSOA --> OA for Scotland and Northern Ireland
• [x] Write package code which generates the OA centroids from shapefiles on the fly (e.g. using shapely and https://geoportal.statistics.gov.uk/datasets/output-area-december-2011-full-clipped-boundaries-in-england-and-wales), including tests [ ] Extend to whole of the UK [ ] Batch the get request
• [x] Write a ORM to store all of the above data, including mapping lookups.
• [ ] Write package code to iterate over the points, according to some strategy
• I would personally say each nearest n neighbours could use https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.cKDTree.html#scipy.spatial.cKDTree and store/retrieve the tree in S3 by pickling if required)
• Maybe use n = 3 by default to be conservative at first?
• In addition to the KNN search, do a random 'coarse' search
• Hit the https://openrouteservice.org API. I would mock the response using tests to avoid actually hitting the API too many times whilst in development.
• [ ] Write an ORM mapping the journey time between OAs.
• [ ] Write a luigi routine to collect the geographies (dev mode then production)
• [ ] Write a luigi routine to collect the time matrix (dev mode then production)
• [ ] Referring to issue #192 see their implementation for converting SQL data to neo4j. I would open a new issue at this point, and develop your codebase from there!

[stefgrs]

Just leaving it as a comment before I forget. At some point, I'll add to the list:

• collect info from Nomis (population, employment, etc.) per OA / LSOA
• address second level TODOs (e.g. extend LSOA --> OA to the whole of the UK)
• optimise code if it's too slow