qc checks: e.g. smoke_2SD may have errors, might get missing or 0 pm25_pred, others?

[ivanhanigan]

this is supposed to be

1: remainder for FID-Date > 2 standard deviations (SD) of remainder across each FID
0: remainder for FID-Date <= 2 standard deviations (SD) of remainder across each FID

but when extracting data for SA1s in Canberra to test I found odd result

[truth-quark]

Which part is odd? The location of yellow/orange dots or the smoke/PM2.5 readings? (Y axis)

There is also a potential explanation from early Dec 2019 onwards:

• Large amounts of smoke blew over Canberra from the Tallaganda forest fire (from at least 8/12/2019). This lasted for a few weeks, throughout the January holiday period. The smoke was like fog on some days.
• From 27/1/2020, Namadgi National park was on fire, spreading more smoke around.

[ivanhanigan]

@truth-quark the dates in February have high PM2.5 but were not flagged as > smoke_2SD (two standard deviations)

[ivanhanigan]

I note these days in ACT have 0 pm25_pred but that is impossible.

[1] "2003-01-01" "2003-04-02" "2003-05-22" "2003-05-23" "2003-05-24" "2003-05-25" "2003-05-26" "2003-05-27" "2003-05-28" [10] "2003-05-29" "2003-05-30" "2003-05-31" "2003-06-01" "2003-06-02" "2003-06-03" "2003-06-04" "2003-06-05" "2003-10-16"

 

| >

[ivanhanigan]

THis is the whole 20 years. I created the green line manually from the data. It shows that some times smoke_2SD is under the true threshold

[truth-quark]

Here's an ncview image of smoke 2SD for 2003-01-01:

This looks wrong, given the large angular data block in SE NSW & the ACT. My assumption is the smoke data should look uneven. The shape looks like swaths from a passing satellite. Do we have the source data used to create the smoke_2SD layer?

[cassandrayuen]

Here's an ncview image of smoke 2SD for 2003-01-01:

This looks wrong, given the large angular data block in SE NSW & the ACT. My assumption is the smoke data should look uneven. The shape looks like swaths from a passing satellite. Do we have the source data used to create the smoke_2SD layer?

Most probably the blockiness is from the MERRA-2 satellite resolution (roughly 55-70km). The MERRA-2 satellite variables make up a decent portion of predictors so that distinctive square grid is a visible artefact even in the predicted PM2.5.

[ivanhanigan]

But smoke_2sd is binary 1 or zero so it has to have a hard boundary. It is not representing smoke PM

[truth-quark]

Ok, I assumed the resolution would be much better than 55-70km.

[truth-quark]

THis is the whole 20 years. I created the green line manually from the data. It shows that some times smoke_2SD is under the true threshold [image removed]

For debugging, is it possible to rejig the R graphing code? Thoughts:

• general change: plot the red & orange points last/on top of the lines
• general change: change the vertical red lines colour to help distinguish red points & vert lines
• (1) Plot just the 2003 data (smaller area to focus debugging to specific days)
• (2) Plot only 2019 in the combined chart style

Late 2009 also has a data gap (there's a slight hole showing in the 20 year plot).