use copy of PNG data for rendering, so we get more control over colors

[robinkraft]

Someone at Vizzuality said that we should actually be able to make a copy of the PNG data and manipulate it without destroying the original - needed for scrubbing backwards in time.

This is worth doing if only to get better control over the color scheme.

We don't want flicker, of course, and have to be able to scrub forwards and backwards in time.

[hhuuggoo]

this is definitely true, when we were working on it earlier, we were having trouble because our entry point that we were trying to copy was some object that was alot of trouble to copy.

if we find where the the data is loaded, we should separate it into data and image, and use the data to manipulate the image.

[brianhonohan]

I'm looking into this now... their live version of the demo does not use the same code they committed to github. http://visualraster.appspot.com/

I've grabbed the live app.js and canvas_tile_layer.js and looking to incorporate our changes (or better yet, parameterize the functions so that a configuration array and necessary callbacks can be passed in).

[brianhonohan]

I've isolated where this an issue... and have a working version (with no residual Blue) up at: http://elementstream.com/forma-vis/test_pixel/maps_threshold_NEW.html

Robin, can you double check the permissions on the github repository? I'm getting 403's when trying to push... though I'm wondering if that is an issue of me trying to push into your repository... perhaps I need to fork it, and make updates to my own copy. (Note the commits I made earlier in the day were via the "Edit this file" feature within the github.com site).

Note, I'm seeing what I believe to be data artifacts in the PNG when the map is zoomed out, in so much that the "Red" seems to bleed all over, but when you zoom in (without adjusting the slider), it is isolated to particular areas. Also, there appear to be single pixel red outlines around some of the more sparse areas, where the alpha goes to zero... this gets amplified when zooming out.

[robinkraft]

Hey Brian, be sure you used the ssh clone URL, as explained here:

http://help.github.com/remotes/#clone

On Nov 12, 2011, at 6:01 PM, Brian Honohan wrote:

I've isolated where this an issue... and have a working version (with no residual Blue) up at: http://elementstream.com/forma-vis/test_pixel/maps_threshold_NEW.html

Robin, can you double check the permissions on the github repository? I'm getting 403's when trying to push... though I'm wondering if that is an issue of me trying to push into your repository... perhaps I need to fork it, and make updates to my own copy. (Note the commits I made earlier in the day were via the "Edit this file" feature within the github.com site).

Note, I'm seeing what I believe to be data artifacts in the PNG when the map is zoomed out, in so much that the "Red" seems to bleed all over, but when you zoom in (without adjusting the slider), it is isolated to particular areas. Also, there appear to be single pixel red outlines around some of the more sparse areas, where the alpha goes to zero... this gets amplified when zooming out.

---

Reply to this email directly or view it on GitHub: https://github.com/robinkraft/forma-vis/issues/4#issuecomment-2719855

[robinkraft]

Brian, you're spot on about the artifacts. Check out this tile:

http://elementstream.com/forma-vis/tiles/newecohack/10/824/515.png

If you zoom way in you can see 'borders' on the pixels.

The good news is that it's not a canvas-related problem. Nor is it an error in the original data. Turns out it's introduced by gdal2tiles.py, which by default resamples the original image using the average of the pixel neighborhood. I can regenerate the tiles using the nearest neighbor option, which should not change the pixel values (though it could shift them slightly - not enough to matter).

Thanks for flagging this. I'll add an issue.

On Nov 12, 2011, at 6:01 PM, Brian Honohan wrote:

I've isolated where this an issue... and have a working version (with no residual Blue) up at: http://elementstream.com/forma-vis/test_pixel/maps_threshold_NEW.html

Robin, can you double check the permissions on the github repository? I'm getting 403's when trying to push... though I'm wondering if that is an issue of me trying to push into your repository... perhaps I need to fork it, and make updates to my own copy. (Note the commits I made earlier in the day were via the "Edit this file" feature within the github.com site).

Note, I'm seeing what I believe to be data artifacts in the PNG when the map is zoomed out, in so much that the "Red" seems to bleed all over, but when you zoom in (without adjusting the slider), it is isolated to particular areas. Also, there appear to be single pixel red outlines around some of the more sparse areas, where the alpha goes to zero... this gets amplified when zooming out.

---

Reply to this email directly or view it on GitHub: https://github.com/robinkraft/forma-vis/issues/4#issuecomment-2719855

[robinkraft]

I've taken care of the tiling artifacts in this new tile directory:

http://s3.amazonaws.com/forma/forma_low_zoom/

It only goes to zoom 7, but unless I increase the resolution of the input data we're going to see similar artifacts even if we use the nearest neighbor resampling algorithm. This is because gdal2tiles generates the base tiles first - which requires resampling past zoom level 7 for this dataset - and then uses four of those base tiles, and resamples the four stitched-together tiles to create the appropriate tile at the lower zoom level (e.g. 6). So the data artifacts introduced in the base tiles cascade through the rest of the zoom levels.

In short, for the moment we'll have to be satisfied with zoom level 7. We're prepping a new data release at 500m resolution for Indonesia, which will get us up to zoom level 9 I believe without requiring resampling for base tiles. Then if we want higher zooms we'll have to mess with the input image so that we have correctly-sampled base tiles at higher resolution.

-Robin

On Nov 12, 2011, at 6:49 PM, Robin Kraft wrote:

Brian, you're spot on about the artifacts. Check out this tile:

http://elementstream.com/forma-vis/tiles/newecohack/10/824/515.png

If you zoom way in you can see 'borders' on the pixels.

The good news is that it's not a canvas-related problem. Nor is it an error in the original data. Turns out it's introduced by gdal2tiles.py, which by default resamples the original image using the average of the pixel neighborhood. I can regenerate the tiles using the nearest neighbor option, which should not change the pixel values (though it could shift them slightly - not enough to matter).

Thanks for flagging this. I'll add an issue.

On Nov 12, 2011, at 6:01 PM, Brian Honohan wrote:

I've isolated where this an issue... and have a working version (with no residual Blue) up at: http://elementstream.com/forma-vis/test_pixel/maps_threshold_NEW.html

Robin, can you double check the permissions on the github repository? I'm getting 403's when trying to push... though I'm wondering if that is an issue of me trying to push into your repository... perhaps I need to fork it, and make updates to my own copy. (Note the commits I made earlier in the day were via the "Edit this file" feature within the github.com site).

Note, I'm seeing what I believe to be data artifacts in the PNG when the map is zoomed out, in so much that the "Red" seems to bleed all over, but when you zoom in (without adjusting the slider), it is isolated to particular areas. Also, there appear to be single pixel red outlines around some of the more sparse areas, where the alpha goes to zero... this gets amplified when zooming out.

---

Reply to this email directly or view it on GitHub: https://github.com/robinkraft/forma-vis/issues/4#issuecomment-2719855

[brianhonohan]

Cool, I'll check them out. I will also try to use the proper clone command to be able to commit my changes back up to github.

On Thu, Nov 17, 2011 at 12:53 PM, robinkraft < reply@reply.github.com

wrote:

I've taken care of the tiling artifacts in this new tile directory:

http://s3.amazonaws.com/forma/forma_low_zoom/

It only goes to zoom 7, but unless I increase the resolution of the input data we're going to see similar artifacts even if we use the nearest neighbor resampling algorithm. This is because gdal2tiles generates the base tiles first - which requires resampling past zoom level 7 for this dataset - and then uses four of those base tiles, and resamples the four stitched-together tiles to create the appropriate tile at the lower zoom level (e.g. 6). So the data artifacts introduced in the base tiles cascade through the rest of the zoom levels.

In short, for the moment we'll have to be satisfied with zoom level 7. We're prepping a new data release at 500m resolution for Indonesia, which will get us up to zoom level 9 I believe without requiring resampling for base tiles. Then if we want higher zooms we'll have to mess with the input image so that we have correctly-sampled base tiles at higher resolution.

-Robin

On Nov 12, 2011, at 6:49 PM, Robin Kraft wrote:

Brian, you're spot on about the artifacts. Check out this tile:

http://elementstream.com/forma-vis/tiles/newecohack/10/824/515.png

If you zoom way in you can see 'borders' on the pixels.

The good news is that it's not a canvas-related problem. Nor is it an error in the original data. Turns out it's introduced by gdal2tiles.py, which by default resamples the original image using the average of the pixel neighborhood. I can regenerate the tiles using the nearest neighbor option, which should not change the pixel values (though it could shift them slightly - not enough to matter).

Thanks for flagging this. I'll add an issue.

On Nov 12, 2011, at 6:01 PM, Brian Honohan wrote:

I've isolated where this an issue... and have a working version (with no residual Blue) up at: http://elementstream.com/forma-vis/test_pixel/maps_threshold_NEW.html

Robin, can you double check the permissions on the github repository? I'm getting 403's when trying to push... though I'm wondering if that is an issue of me trying to push into your repository... perhaps I need to fork it, and make updates to my own copy. (Note the commits I made earlier in the day were via the "Edit this file" feature within the github.com site).

Note, I'm seeing what I believe to be data artifacts in the PNG when the map is zoomed out, in so much that the "Red" seems to bleed all over, but when you zoom in (without adjusting the slider), it is isolated to particular areas. Also, there appear to be single pixel red outlines around some of the more sparse areas, where the alpha goes to zero... this gets amplified when zooming out.

---

Reply to this email directly or view it on GitHub: https://github.com/robinkraft/forma-vis/issues/4#issuecomment-2719855

---

Reply to this email directly or view it on GitHub: https://github.com/robinkraft/forma-vis/issues/4#issuecomment-2779323

[robinkraft]

Hey Brian, it looks like this got fixed given the nice colors. But I'm not sure - were you able to copy the image data somehow?

[brianhonohan]

Yep, I fixed this. Fairly certain it's related to the drawImage(...) call. The maps_threshold.html example tried to do some performance optimizations by disabling the redrawing of the image data.

CanvasTileLayer.prototype.filter_tile = function(canvas, args) {
    var ctx = canvas.getContext('2d');
    ctx.drawImage(canvas.image, 0, 0);  
    var I = ctx.getImageData(0, 0, canvas.width, canvas.height);
    this.filter.apply(this, [I.data, ctx.width, ctx.height].concat(args));
    ctx.putImageData(I,0,0);
}

[robinkraft]

Great!