search

noaa-oar-arl / monetio

The Model and ObservatioN Evaluation Tool I/O package
https://monetio.readthedocs.io
MIT License
17 stars 30 forks source link

Add Gridded VIIRS from the NOAA AWS bucket and more #164

Open bbakernoaa opened 8 months ago

bbakernoaa commented 8 months ago

Added a new function to stream data from AWS from the noaa-jpss bucket into xarray for monetio. This should be useful for satellite comparisons.

It can handle both SNPP and NOAA20 satellites as well as varying resolutions (0.05, 0.1, 0.25 degree) and different time aggregations that are preprocessed by NESDIS (daily, weekly, monthly).

Data availability is from 2012 -> current.

to use:

nests_viirs_aod_gridded.open_dataset('2013-01-01', 'SNPP', '0.100', 'daily')

nests_viirs_aod_gridded.open_mfdataset(dates, 'SNPP', '0.050', 'daily')

@angehung5

bbakernoaa commented 8 months ago

just a few seconds at most

bbakernoaa commented 8 months ago

Can you run the pre-commit hooks?

How long does it take to get a small amount of data (for test)?

Not quite sure how to do that. Can you provide a little instruction?

bbakernoaa commented 8 months ago

I added back in the NRT data. Seems the AWS server didn't have a the latest data from the past 3 years. in the NRT data we don't have the 0.05 data but we do have the 0.1 and 0.25 degree data.

zmoon commented 8 months ago

Not quite sure how to do that.

  1. Install pre-commit to your env if you don't have it available already (it's available on conda-forge)
  2. pre-commit install --install-hooks
  3. pre-commit run --all-files
zmoon commented 8 months ago

just a few seconds at most

Sweet, do you want a write a test? I can give it a go if not.

bbakernoaa commented 8 months ago

Not quite sure how to do that.

  1. Install pre-commit to your env if you don't have it available already (it's available on conda-forge)
  2. pre-commit install --install-hooks
  3. pre-commit run --all-files

Thanks! Added it and got these rolling. Seems the tests are failing but in unrelated spots where the code was changed

zmoon commented 3 months ago

Current NDVI failure seems to be an issue decoding the times in current xarray (works in my local older version) https://github.com/pydata/xarray/issues/8957 TIMEOFDAY variable is the offender.

zmoon commented 2 months ago

@bbakernoaa besides the question of whether the module names need to have _gridded, I also wondered if you wanted to add a bit more documentation about the data sources, some links or something.

bbakernoaa commented 2 months ago

@bbakernoaa besides the question whether the module names need to have _gridded, I also wondered if you wanted to add a bit more documentation about the data sources, some links or something.

sure where you like me to add these?

zmoon commented 2 months ago

sure where you like me to add these?

It could be in module docstring at the top (probably best), or in the docstring of the function that gets the data, or some combination. Or send me or point me to the relevant info and I can put it in.

zmoon commented 5 days ago

@bbakernoaa any chance we can back up a bit and just get the docstring info enhancements so we can get this in faster? We can preserve the typing additions and such in a branch for after we drop 3.6.