search

GEUS-Glaciology-and-Climate / pypromice

Process AWS data from L0 (raw logger) through Lx (end user)
https://pypromice.readthedocs.io
GNU General Public License v2.0
14 stars 4 forks source link

OOL dependencies #21

Closed mankoff closed 1 year ago

mankoff commented 3 years ago

https://github.com/GEUS-PROMICE/PROMICE-AWS-processing/blob/main/variables.csv

OOL flags have dependencies. For example, if t_1 is OOL, then also flag rh_cor cc dsr usr z_boom z_stake.

Do we care about circular dependencies? What about order? What to do if the variables DB, processed in order, specifies:

if OOL(B) then flag C If OOL(A) then flag B.

What happens to C in this case, where B is good, until flagged by A?

I think a simple brute-force way to make sure all dependency chains get resolved is to re-run the flagging function n times, where n is the length of the longest OOL dependency list: max([len(_.split(" ")) for _ in df['OOL'].dropna()]) (ans: 6)

PennyHow commented 1 year ago

OOL flag dependencies are run at two different points in the pypromice workflow currently:

https://github.com/GEUS-Glaciology-and-Climate/pypromice/blob/2331c02e1e121648c06dd054d370f0b99a6a8f6d/src/pypromice/aws.py#L96-L115

Therefore, if it is not caught in the first run, then it will be in the second. I think this effectively catches these dependency flags for now.