search

ariesteam / aries

http://www.ariesonline.org
GNU General Public License v3.0
6 stars 1 forks source link

overlap problems on NLCD rasters #32

Closed kbagstad closed 12 years ago

kbagstad commented 12 years ago

@fvilla

kbagstad commented 12 years ago

So we have (in theory) a seamless NLCD LULC, impervious surface, and tree canopy cover dataset for the whole US. It's not tiled, it's done by zones that have irregular boundaries but which match up perfectly. However, when we call for one of these layers using a context that crosses two zones, we get weird gaps in the outputs. The two commands below will generate these gaps. Is there any way to remove them so that the layers properly produce seamless outputs? I'm going to email an image of this as github is a serious pain in the *ss for attaching images to issues.

model -o canopytest.nc core.models.water-colorado/percent-canopy-cover core.contexts.beta/co_south_platte

model -o canopytestaz.nc core.models.water-colorado/percent-canopy-cover core.contexts.beta/arizona

fvilla commented 12 years ago

oh shit, this is something I was afraid of at the time I implemented that, and didn't have a good test case. I haven't checked yet, but I'm inclined to think it's a problem with reprojected data - basically the two layers are in the same projection but are warped to the common one for the model before merging, and that creates well-known edge effects. The only way to fix it is to merge unprojected and reproject the merge, which sounds easy... but the problem is, geoserver does the reprojection and thinklab the merging, and the fix would require thinklab to do an in-memory reproject which is highly complex, specialized code that isn't available at the moment; or geoserver to do the merge, which I'm pretty sure is a use case they haven't thought about.

So unless it's something simpler (and I don't really think so) it will be a pretty big deal to fix. I'll scour the libraries for this kind of code and ask on the geoserver list for merging options I may have missed. Hope it's not a next-day criticality.

kbagstad commented 12 years ago

Here's the deal: Layers were manually reprojected to a common projection (WGS84) before entering geoserver, so GS/thinklab should not have to do that, though they do have to do a merge. How do those facts affect your understanding of the problem?

There will (eventually) be a single nationwide data layer for the NLCD data but of course this problem could and likely will bite us in other cases. Manual merging beforehand gets tough, as these are giant datasets, with limitations in Arc, as Brian and I are rapidly discovering.

This is indeed a semi-urgent problem - if there's a quick fix, great, if not, let's keep it on our radar and think about longer-term fixes.

Ken

On Wed, Nov 16, 2011 at 2:40 AM, Ferdinando Villa reply@reply.github.com wrote:

oh shit, this is something I was afraid of at the time I implemented that, and didn't have a good test case. I haven't checked yet, but I'm inclined to think it's a problem with reprojected data - basically the two layers are in the same projection but are warped to the common one for the model before merging, and that creates well-known edge effects. The only way to fix it is to merge unprojected and reproject the merge, which sounds easy... but the problem is, geoserver does the reprojection and thinklab the merging, and the fix would require thinklab to do an in-memory reproject which is highly complex, specialized code that isn't available at the moment; or geoserver to do the merge, which I'm pretty sure is a use case they haven't thought about.

So unless it's something simpler (and I don't really think so) it will be a pretty big deal to fix. I'll scour the libraries for this kind of code and ask on the geoserver list for merging options I may have missed. Hope it's not a next-day criticality.

Reply to this email directly or view it on GitHub: https://github.com/ariesteam/aries/issues/32#issuecomment-2757613

kbagstad commented 12 years ago

One other potentially useful bit of information: there is a nodata border around each individual layer that is likely causing the problem (so in that sense you're right, the data aren't seamless, well, the useful data are but for some annoying reason they put a nodata border around the good data). I believe it's value 127, which is BS, and which we set as a nodata value in the .xml. My guess is that the nodata show through and cancel out the good data within that overlap region. So if there's an easy way to prioritize the good data over the nodata, that would be a fantastic and exciting fix to all of this...

On Wed, Nov 16, 2011 at 8:14 AM, Ken Bagstad kbagstad@gmail.com wrote:

Here's the deal: Layers were manually reprojected to a common projection (WGS84) before entering geoserver, so GS/thinklab should not have to do that, though they do have to do a merge.  How do those facts affect your understanding of the problem?

There will (eventually) be a single nationwide data layer for the NLCD data but of course this problem could and likely will bite us in other cases.  Manual merging beforehand gets tough, as these are giant datasets, with limitations in Arc, as Brian and I are rapidly discovering.

This is indeed a semi-urgent problem - if there's a quick fix, great, if not, let's keep it on our radar and think about longer-term fixes.

Ken

On Wed, Nov 16, 2011 at 2:40 AM, Ferdinando Villa reply@reply.github.com wrote:

oh shit, this is something I was afraid of at the time I implemented that, and didn't have a good test case. I haven't checked yet, but I'm inclined to think it's a problem with reprojected data - basically the two layers are in the same projection but are warped to the common one for the model before merging, and that creates well-known edge effects. The only way to fix it is to merge unprojected and reproject the merge, which sounds easy... but the problem is, geoserver does the reprojection and thinklab the merging, and the fix would require thinklab to do an in-memory reproject which is highly complex, specialized code that isn't available at the moment; or geoserver to do the merge, which I'm pretty sure is a use case they haven't thought about.

So unless it's something simpler (and I don't really think so) it will be a pretty big deal to fix. I'll scour the libraries for this kind of code and ask on the geoserver list for merging options I may have missed. Hope it's not a next-day criticality.

Reply to this email directly or view it on GitHub: https://github.com/ariesteam/aries/issues/32#issuecomment-2757613

kbagstad commented 12 years ago

So never heard back - is this an easy fix or not?

kbagstad commented 12 years ago

Note that this will also present a problem in training the carbon models: the National Biomass and Carbon Dataset is another "irregularly tiled" dataset that may be subject to the same problem.

kbagstad commented 12 years ago

This issue was addressed in issue #60. A big fix on the geoserver end, so will have to find a data work-around. Closing.