holes in the mean days precip layer

[kbagstad]

OK, just another one that will limit our credibility when we can't properly import a layer.

model -o /raid/nc_outputs/june_11/preciptest.nc core.models.flood-puget/mean-days-precipitation-annual core.contexts.beta/western_wa2048

produces a map with noData "holes" when there clearly are data, viewed in both geoserver and the underlying data. Underlying .xml and defmodel statement are incredibly simple.

[fvilla]

All I can see is that the number of no-data values in the file coming from GS is the same as in the classified layer (can be seen easily with -d). That means that probably the no-data statements in the original geotiff are broken (therefore they need to be fixed in the XML) - I'd check what values come from GS where there should be data you're not seeing and ensure they're not the default no-data (probably 0 or 255). The output of geoserver is imported and reclassified correctly, I don't know what else to do here (or how the nodata would limit our credibility).

[kbagstad]

Ahh if only it were that simple! There are no "nodata" set for this layer, as all data are good (it's a very simple vector layer that just selects the attribute "gridcode", which ranges from 1-9, as reported in kb.xml. Checking the layer in geoserver's layer preview will demonstrate that there are no valid "nodata" values). If you then look at flood-puget.clj to the mean-days-precipitation-annual defmodel statement, you'll see that it's a very simple reclassification of the values 1-9. Here's the problem: the nodata "holes" are all values 1-9, and you'll even see holes where there should be values of "7" or "8" where elsewhere in the layer values of 7 or 8 are reported or reclassified just fine! This is where I continue to be at a loss. You can see the underlying data yourself at raid/geodata/data/united_states/meanDaysPrecip/Annual.shp - try a model -o then compare the holes in the layer to the actual data itself, and you may wind up as stumped as I am.

I could also show this to you via webex but I know times to do that are hard to come by...

[fvilla]

OK, it's a vector file then - all thinklab does is to take one polygon at a time and burn in the values in a raster using image paint functions. Which it does correctly, as that's its main function and if it didn't work it wouldn't just poke holes here and there and only show problems with this layer. It can be caused by a couple things - one being the data having complex polygons whose edges intersect or cause holes - those require very sophisticated rasterization code which is a nightmare to write. The other is by having overlapping polygons so when one value has been burned in correctly, another polygon with a hole comes out and empties it. Nothing that's easy to debug or fix, anything involving 2d topologies doesn't really lend itself to spot debugging unfortunately. If it's not too much work, could you try rasterizing that area in arc and see what you get from there? "My" rasterizer is industry standard, meaning it uses the same algorithm as GRASS, Sextante and Geotools, but dealing with degenerate polygons and all that is a major undertaking and very few softwares do it right.

[kbagstad]

Fixed by rasterizing the damn layer. Another manual fix to a lousy 3rd party data layer, but at least it's working fine now.

[fvilla]

Must have been a degenerate vector layer then - the triangular shape of the holes means that most likely there were crossing boundaries, creating the infamous "papillon" shape that is a nightmare for rasterization. ARC must have a pretty good rasterizer if it does that right. I'll open a general thinklab issue to do more checking when things are calmer.