search

Applied-GeoSolutions / gips

Geospatial Image Processing System
GNU General Public License v3.0
17 stars 5 forks source link

consistent CDL asset creation #38

Open dchowdhury opened 8 years ago

dchowdhury commented 8 years ago

dchowdhury@nile:/titan/data/cdl/stage/dc$ gips_archive cdl -v5 GIPS Data Archive Utility (v0.8.2) Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/gips/scripts/archive.py", line 49, in main cls.Asset.archive(**vars(args)) TypeError: archive() got an unexpected keyword argument 'recursive'

Data archive error: archive() got an unexpected keyword argument 'recursive'

dchowdhury commented 8 years ago

CDL data seems to be missing NoData Value in .tif files. Not sure if this was always the case but if data is manually placed in archive dir and gips_project-ed, nodata value is off in product.

ircwaves commented 8 years ago

The missing-no-data issue is a new thing for NASS. Perhaps it is a bug. If not, it requires that CDL either implement archive and augment it with no data, OR that we stop expecting that CDL has a no-data value.

ircwaves commented 7 years ago

@dchowdhury: did we hear back from NASS on this one? I feel like we conversed about it, but github is my memory...and I don't see it here.

dchowdhury commented 7 years ago

Applied Geosolutions Mail - NASS CDL products.pdf

ircwaves commented 7 years ago

So, gips.data.cdl had not implemented archive previously, and NASS used to set NODATA and have consistent color tables. This email indicates that they may no longer have a consistent color table. Seems like a review of the CDL metadata is in order, and possibly an implementation of an archive method, since we may need to set our own NODATA value.

ircwaves commented 7 years ago

Please review if the color tables and class codes have changed for Indiana between 2014 and 2016

https://www.nass.usda.gov/Research_and_Science/Cropland/metadata/meta.php

ircwaves commented 6 years ago

@naterubin -- I believe this has been addressed...reopen if you disagree.

bhbraswell commented 6 years ago

CDL assets should be unified and versioned. There should be one CDL asset (not two as currently implemented). There is strong evidence that the interpretation of crop code values has not changed over time, though there are minor differences in the legend that can be resolved unambiguously.

Thus we believe we don't need the mkii asset. However it is clear that there are changes in pixel values occasionally due to reprocessing. Reprocessed data are not explicitly versioned, so I suggest we version by fetch date. See #486

Note also that the asset will revert to being a single tif file and the dbf should not be needed. See the updated file CDL_Legend.csv in the CDL archive top level, which I suggest should be in the repo. On fetch, we can check the dbf against this file and at that point the dbf can be discarded.