openaq / openaq-ingestor

1 stars 1 forks source link

Source id value given to sensors during fetch is not consistent across fetchers #5

Open caparker opened 1 year ago

caparker commented 1 year ago

When a fetcher pulls data in from a source it will typically have a source site identifier as well as an identifier for the parameter (and units). This parameter value is then mapped to a name that corresponds with the name used in the openaq database.

Different fetchers are handling this differently. Purple air is using the source parameter identifier in the final sensor source id name that is sent to the server while the clarity fetcher is using the openaq name, but without the units.

Example source_id for sensor: PurpleAir-35053-10.0_um_count

const lookup = {
    // input_param: [measurand_parameter, measurand_unit]
    'pm1.0': ['pm1', 'µg/m³'],
    'pm2.5': ['pm25', 'µg/m³'],
    'pm10.0': ['pm10', 'µg/m³'],
    '0.3_um_count': ['um003', 'pp100ml'],
    '0.5_um_count': ['um005', 'pp100ml'],
    '1.0_um_count': ['um010', 'pp100ml'],
    '2.5_um_count': ['um025', 'pp100ml'],
    '5.0_um_count': ['um050', 'pp100ml'],
    '10.0_um_count': ['um100', 'pp100ml'],
    'humidity': ['humidity', '%'],
    'temperature': ['temperature', 'f'],
    'pressure': ['pressure', 'mb'],
    'voc': ['voc', 'iaq'],
    'ozone1': ['ozone', 'ppb']
};

And in clarity we are using the first entry in the value Source id for sensor: AX5CC5K1-pm10

const lookup = {
    relHumid: ['relativehumidity', '%'], // RelativeHumidity
    temperature: ['temperature', 'c'], // Temperature
    pm2_5ConcMass: ['pm25', 'μg/m3'], //    PM2.5 mass concentration
    pm1ConcMass: ['pm1', 'μg/m3'], //   PM1 mass concentration
    pm10ConcMass: ['pm10, 'μg/m3'], //  PM10 mass concentration
    no2Conc: ['no2', 'ppb'], // NO2 volume concentration
    windSpeed: ['windspeed', 'm/s'], // Wind speed
    windDirection: ['winddirection', 'degrees'] //  Wind direction, compass degrees (0°=North, then clockwise)
};

One of the issues that this creates is that we cannot create a new node/system/sensor directly from the measurements file