Mongolia - local model steps to build

[ChristinaLast]

Steps to build first model version

• [x] finalize columns that need to be generated for Robert historic data and Purple air sensors
• [x] create master table of columns from all sensors
• [x] diagnose issues in building features table (e.g. satellite data not varying).
• [x] Select the test validation set protocol
• [x] Train model.

[ChristinaLast]

2. Creating a master table

• we are treating the two tables as separate, we are going to generate the following structure from robert's data:

  1   locationId  float8  YES NULL    NULL        NULL
  2   location    text    YES NULL    NULL        NULL
  3   parameter   text    YES NULL    NULL        NULL
  4   value   float8  YES NULL    NULL        NULL
  5   date    text    YES NULL    NULL        NULL
  6   unit    text    YES NULL    NULL        NULL
  7   coordinates text    YES NULL    NULL        NULL
  8   country text    YES NULL    NULL        NULL
  9   city    text    YES NULL    NULL        NULL
  10  isMobile    bool    YES NULL    NULL        NULL
  11  isAnalysis  text    YES NULL    NULL        NULL
  12  entity  text    YES NULL    NULL        NULL
  13  sensorType  text    YES NULL    NULL        NULL

Ensuring that the locationId for Amgalan and US Diplomatic Post are the same as in the cohorts_MN table.

• We will then append the tables and process them with cohort_builder, this will generate the windows (e.g. cohort columns), and extract the date utc and x y from coordinates.
• Using the combined table we will write a separate cohorts_all_MN table and generate a uniqueId which is a combination of the locationId, timestamp_utc and cohort column.

[ChristinaLast]

The text description of how to translate Rob's columns to the API columns: Key differences:

1. the date columns should retain the utc and local {"utc": "2024-01-07T18:00:00+00:00", "local": "2024-01-08T02:00:00+08:00"}
2. the unit column should be µg/m³ rather than µg/m³
3. the location column should contain the location in Ulaanbaatar Particulate Matter Sensor Data
4. the mobile should be FALSE if false (which is currently a type str when it should be processed to boolean in postgres
5. isAnalysis column should be null
6. sensorType should be "reference grade"
7. entity columns should be assigned from sourcetype