search

gulfofmaine / sdm_workflow

A repository to help streamline the species distribution model development and prediction workflow.
MIT License
2 stars 0 forks source link

Add NaN step in the Zarr process #39

Closed dzaugis closed 2 years ago

dzaugis commented 3 years ago

Find fill values and make them NaNs in the initial processing step

adamkemberling commented 3 years ago

Just for documentation sake here following some slack messages with individuals:

OISST's fill value is 32767, I don't see that coming through in anything I've seen, in either the climatology we have or the timelines etc. Feel good here.

I think I should check the SODA data and re-run the SODA climatologies. That way any strange fill values don't bake-in to the climatologies and then impact everything downstream.

Quick fix on the CMIP6 data screening is to set a boundary (say -50 to 100), outside of which we would assign NA. Same story here, just apply that screen as early as possible so that it doesn't impact downstream steps.

aallyn commented 3 years ago

Awesome! I feel like this part was a bit confusing with all three of us involved and I honestly can't remember the exact process. I think I remember collecting all the individual yearly SODA files and then passing along the collected file for each variable, but cropped to our study region of interest? Those are then in Box/RES_Data/SODA? From there...@dzaugis ran things through to get the bottom values for salinity and temp? And then....@adamkemberling did the calculation of the climatology?

adamkemberling commented 3 years ago

Just to follow up with some documentation severalmonths later. I screen each CMIP, SODA, and OISST stack for values outside a reasonable range.

Here is what that looks like for the OISST and CMIP bias corrections: https://github.com/gulfofmaine/sdm_workflow/blob/main/CMIP6_processing/R/CMIP_OISST_bias_corrections.Rmd#L103-L120

And for SODA & CMIP Bias corrections: https://github.com/gulfofmaine/sdm_workflow/blob/main/CMIP6_processing/R/CMIP_SODA_bias_corrections.Rmd#L126-L157

As of now I am unsure whether they are done "upstream" in the Zarr steps, but we have checks against crazy fill values there.