Internal migration

[crangelsmith]

Internal migration implementation based on this document. This is a meta-issue.

The implementation is in the following steps:

• [x] Get a "pool" of individuals that migrate from a local region (based on the probability of internal migration here) (PR #45)

• [x] Individual that are in that pool get assigned to a new location. (PR #45)

  • Assignment happens with a matrix that uses the probability of one person from an area X to migrate to are Y. Memory of the person movement is kept.

• [x] Implement more complex migration matrix (PR #?)

  • Currently the new location allocation does not depend on the individual age, gender or ethnicity, this is to be included in the next PR.

Note: we use csv2sparse function in utilities.py to convert the OD matrices to sparse representations.

Refer to #48 , these are already implemented:

• [x] Read the correct OD matrix given the migrant info (age and gender)
• [x] Efficient ways to read OD matrices (e.g., sparse?), use scipy.sparse.coo_matrix to save npz files for each OD matrix (in the preprocessing step), we then load these files in the iternal_migration code.
• [x] Replace MSOA_LAD_indices with the first column of OD matrices, this should be created in the preprocessing step as the previous step
• [x] ~Load the matrices outside of the class~
• [x] ~Upload the large files to git LFS?, Fix CI accordingly~
• [x] ~in test_internal_migration, we download persistant_data/OD_matrix_EW_no_intra_flows.csv using wget, this should be replaced by the new matrices~
• [x] Test the components

[crangelsmith]

Topics to discuss in meeting of 117th August:

1. VPH framework:

   • Implement internal migration relocation in the same component that chooses the potential migrants
   • A function that reads the migrant initial location, age, gender and ethnicity and returns a new location
   • In the output data the location field gets re-writen but we have a new variable that saves the old locations and the time of migration.

2. The assignment function:

• Multidimensional matrix with values of observed migration
• Colapse matrix 1 dimension based on the input migrant information
• Softmax to build the distribution to a PDF -> or just use simple rates for simpler computation.
• Sample from PDF to get a location

3. The data used to build the matrix.

• Format of the data: MSOA to MSOA level, by age and gender (ethnicity might be added)
• Matrix format considerations: Dictionary of matrices.
• Can we start with a simpler model? (E.g only using input and output location numbers?)

[crangelsmith]

[

](url)

[kasra-hosseini]

@crangelsmith please take a look and feel free to edit as you see fit.

In the meeting today (17th Aug), we talked about the following components:

1. A data structure to efficiently extract a "distribution over destinations" for each (sex, age group, location).

input: Rows with internal_outmigration == True in:

The relevant columns (sex, age, location, ...) should be selected, converted to meaningful keys for the data structure and finally:

output:

matrix:
Shape: N x d
N: number of rows with `internal_outmigration == True`
d: number of possible destinations
values: weights assigned to each possible destination

2. The second component reads in the above matrix and performs the following steps:

• convert each row to a probability distribution (see above for more info)
• sampling, one column for each row

output:

vector:
Shape: N x 1
N: number of rows with `internal_outmigration == True`
values: ids of next destination

[crangelsmith]

Notes from meeting on 19 August:

• Internal migration matrices are in the ward level. This means the following steps (@kasra-hosseini please feel free to add/comment/modify these):

  1. Add ward information to the initial data, on the data preparation step
  2. Use matrix assignment are going to be on ward to ward level (the same procedure as described in the previous comment applies).
  3. Add extra step that assigns MSOA back from ward level (use rates obtained from nomis data)
  4. Re-write ward and MSOA to the migrant data, save previous locations in a new field.

I'll will try to have all steps implemented in the code (with step 2 and 3 using dummy matrices) before I leave on holiday.

[kasra-hosseini]

Currently, we use one OD matrix (regardless of age/sex) to assign new MSOAs to migrants.

TODO:

• [ ] Use int_migration_pool to detect correct OD matrices and their rows, this depends on the origin, sex and age.

NOTE: as already mentioned, we use one OD matrix now.

The plan is to use IPF to generate one OD matrix for each (age_group, sex). In the internal migration component and by using int_migration_pool, we should first detect the correct OD matrix (based on age_group and sex), find the relevant rows (based on origins), extract the rows and go to next step.

• [x] This results in a matrix of n x m where (n is the number of migrants in the pool, m is the number of potential MSOA they can be assign to).
• [x] Normalise the matrix by total number of counts in each row, to obtain rates
• [x] Return the rate matrix of n x m

[kasra-hosseini]

I also added a new file: https://github.com/alan-turing-institute/vivarium_public_health_spenser/blob/features/44-internal-outmigration/persistant_data/Middle_Layer_Super_Output_Areas__December_2001__Population_Weighted_Centroids.csv

Using this file, we can convert migrant's MSOAs to geographic coordinates, so we can track the locations of migrants. In the following figure, the left column shows origins and the right column shows migrant's destinations.

[kasra-hosseini]

I had to add a very large file (~100MB) to the repo as well: https://github.com/alan-turing-institute/vivarium_public_health_spenser/blob/features/44-internal-outmigration/persistant_data/OD_matrix_EW_no_intra_flows.csv

Maybe we should remove this from the commit history and upload it somewhere else?