search

emit-sds / emit-ghg

Mapping of greenhouse gases with EMIT.
Apache License 2.0
34 stars 14 forks source link

Saturation clouds flare masking only #5

Closed jfahlen closed 11 months ago

jfahlen commented 1 year ago

This PR adds cloud, water, flare, and saturation masking to the MF mean and covariance estimation. It also adds a field to the MF output file's header file called "fraction_of_lines_per_column_used_in_mf_mu_cov_estimation".

Testing is limited as I'm having trouble getting the set of flare cases that @pgbrodrick sent me to run. It is possible that there is a bug, but I can't seem to reproduce it and I suspect it is a ray issue. Be careful!

jfahlen commented 11 months ago

Updates:

1) Fixed run_bulk_ghg.py to match the new ghg_process.py arguments 2) Renamed mask variables in run_bulk_ghg, ghg_process, and parallel_mf to be explicit about which mask file is which 3) Reworked the masking code in parallel_mf.py to be clearer; no change to algorithm

Testing: 1) Ran the original set of flares cases that Phil sent me a while ago. The current PR reproduces those results exactly for all cases: np.allclose(original_mf, new_mf) is True. I didn't do this for the refined MF outputs because I didn't touch those. outputs here: Current PR: /beegfs/scratch/jfahlen/flare_cases_new, original set before these updates: /beegfs/scratch/jfahlen/flare_cases_only 2) Tested updates to run_bulk_ghg by running the first 3 of the cases from 20230621. The runs completed successfully here: /beegfs/scratch/jfahlen/test_bulk_ghg

pgbrodrick commented 11 months ago

Thanks Jay!