search

publiclab / mapknitter-exporter-sinatra

A minimal Sinatra app to run MapKnitter exports in the cloud
GNU General Public License v3.0
3 stars 4 forks source link

Some images cropped/cut off #39

Open jywarren opened 5 years ago

jywarren commented 5 years ago

I see a couple edges which are not fuzzy, and so that means they were cut off partway through their composite processes.

image

image

https://mapknitter.org/maps/otter-view-park-west

I'm wondering how it happened, but it could be either a crash partway through, OR it could be that their new bounding boxes are not properly calculated and they're just cropped wrong. I'll have to check the intermediate images to be sure.

jywarren commented 5 years ago

This is one that got cut off: 47243/OVPKAP-20111028-85-170_medium.jpg

I'll try to make a special warpable.json for just that

jywarren commented 5 years ago 
[{"cm_per_pixel":9.71209,"created_at":"2011-10-29T01:30:36Z","deleted":true,"height":2448,"history":"","id":47243,"image_content_type":"image/jpeg","image_file_name":"OVPKAP-20111028-85-170.jpg","image_file_size":971714,"locked":false,"map_id":662,"nodes":[{"author":"662","body":null,"color":"black","created_at":"2011-10-29T01:33:00Z","description":"","id":98866,"lat":"44.0182263551","lon":"-73.178511401","map_id":0,"name":"","order":0,"updated_at":"2011-10-29T01:33:00Z","way_id":0,"way_order":0},{"author":"662","body":null,"color":"black","created_at":"2011-10-29T01:33:00Z","description":"","id":98867,"lat":"44.0161785757","lon":"-73.1784863821","map_id":0,"name":"","order":0,"updated_at":"2011-10-29T01:33:00Z","way_id":0,"way_order":0},{"author":"662","body":null,"color":"black","created_at":"2011-10-29T01:33:00Z","description":"","id":98868,"lat":"44.0163197466","lon":"-73.1806368023","map_id":0,"name":"","order":0,"updated_at":"2011-10-29T01:33:00Z","way_id":0,"way_order":0},{"author":"662","body":null,"color":"black","created_at":"2011-10-29T01:33:00Z","description":"","id":98869,"lat":"44.0184715903","lon":"-73.1804661828","map_id":0,"name":"","order":0,"updated_at":"2011-10-29T01:33:00Z","way_id":0,"way_order":0}],"parent_id":null,"thumbnail":null,"updated_at":"2011-10-29T01:36:19Z","width":3264,"src":"https://s3.amazonaws.com/grassrootsmapping/warpables/47243/OVPKAP-20111028-85-170.jpg","srcmedium":"https://s3.amazonaws.com/grassrootsmapping/warpables/47243/OVPKAP-20111028-85-170_medium.jpg"}]
jywarren commented 5 years ago

It's not cut off!

image

jywarren commented 5 years ago

So, i'll run this again and let's see how it goes - if it fails in the same places, or if this was a memory error and they'll get cut off differently:

http://export.mapknitter.org/export?url=https://mapknitter.org/maps/otter-view-park-west/warpables.json&scale=30

http://export.mapknitter.org/id/1558733043/status.json

jywarren commented 5 years ago

Uh oh, that's still stuck at 3 of 25. Not sure why. Will restart it.

{"status_url":"https://mapknitter-exports-warps.storage.googleapis.com/1558733043/status.json","status":"warping 3 of 25","tms":false,"geotiff":false,"zip":false,"jpg":false,"export_id":1558733043,"user_id":null,"size":null,"width":null,"height":null,"start_time":null,"run_time":null,"gem_version":"1.0.6","cm_per_pixel":null}

Here's the restarted one:

https://mapknitter-exports-warps.storage.googleapis.com/1558800253/status.json

jywarren commented 5 years ago

{"status_url":"https://mapknitter-exports-warps.storage.googleapis.com/1558800253/status.json","status":"warping 3 of 25","tms":false,"geotiff":false,"zip":false,"jpg":false,"export_id":1558800253,"user_id":null,"size":null,"width":null,"height":null,"start_time":null,"run_time":null,"gem_version":"1.0.7","cm_per_pixel":null}

So, on the new version. Already at number 3!

jywarren commented 5 years ago

OK, here - for comparison with above image:

image

jywarren commented 5 years ago

Earlier and later images, left to right; the new one is better, but still not complete:

image

jywarren commented 5 years ago

From notes here: https://github.com/publiclab/mapknitter-exporter-sinatra/issues/23#issuecomment-495953963

It looks like it happens on or just before this step, because images like this are copied in wrong, and then distorted, masked, and used in the final map:

https://github.com/publiclab/mapknitter-exporter/blob/c767780392342c109b26a887971b3ab6bcd98c94/lib/mapknitterExporter.rb#L158-L173

jywarren commented 5 years ago

Here's the start of the processing for the above image: https://console.cloud.google.com/logs/viewer?authuser=1&project=public-lab&minLogLevel=0&expandAll=false&timestamp=2019-05-27T23%3A11%3A47.765000000Z&customFacets&limitCustomFacetWidth=true&dateRangeStart=2019-05-20T23%3A13%3A27.528Z&interval=P7D&resource=k8s_container%2Fcluster_name%2Fmapknitter&scrollTimestamp=2019-05-26T04%3A59%3A26.881730593Z&dateRangeEnd=2019-05-27T23%3A13%3A27.528Z&advancedFilter=resource.type%3D%22k8s_container%22%0Aresource.labels.namespace_name%3D%22default%22%0Aresource.labels.location%3D%22us-east1%22%0Aresource.labels.project_id%3D%22public-lab%22%0Aresource.labels.cluster_name%3D%22mapknitter%22%0Aresource.labels.pod_name%3D%22mapknitter-exporter-kubes-6754bb8c54-q6qwt%22%0Aresource.labels.container_name%3D%22mapknitter-exporter-kubes%22%0Atimestamp%3D%222019-05-26T04%3A59%3A25.242514883Z%22%0AinsertId%3D%22f5rcvu9l8f5t5y0yt%22

132014

jywarren commented 5 years ago 
E  - -> /working/1558800531/w132014.tif 
convert -contrast-stretch 0 public/warps/1558824229-working/w132014-LPV-0483.JPG -crop 3648x3648+0+0! -flatten -distort Perspective '0,3648 0,522  0,0 227,0  2736,0 786,438  2736,3648 596,794' -flatten -crop 794x794+0+0! +repage public/warps/1558824229/w132014.png
convert +antialias -size 794x794  xc:none -draw "fill black stroke red stroke-width 30 polyline 0,522 227,0 786,438 596,794 0,522"  -alpha set -channel A -transparent red -blur 0x8 -channel R -evaluate set 0 +channel public/warps/1558824229/w132014-mask.png 
composite public/warps/1558824229/w132014-mask.png public/warps/1558824229/w132014.png -compose DstIn -alpha Set public/warps/1558824229/w132014-masked.png
gdal_translate -of GTiff -a_srs EPSG:4326  -gcp 0.0, 522.4933237396181, -73.1653081419, 44.0562634908 -gcp 227.05951564013958, 0.0, -73.1646962288, 44.0572754172 -gcp 786.0191691294312, 438.93269662186503, -73.1631898628, 44.056425326 -gcp 596.7493552267551, 794.3575743623078, -73.1636999347, 44.0557369574  -co "TILED=NO" public/warps/1558824229/w132014-masked.png public/warps/1558824229/w132014-geo-unwarped.tif
gdalwarp -of GTiff -t_srs EPSG:4326 public/warps/1558824229/w132014-geo-unwarped.tif public/warps/1558824229/w132014.tif
jywarren commented 5 years ago

So, i'd guess it's at this step, before mask generation and mask application:

2019-05-26 00:59:25.242 EDT

convert -contrast-stretch 0 public/warps/1558824229-working/w132014-LPV-0483.JPG -crop 3648x3648+0+0! -flatten -distort Perspective '0,3648 0,522 0,0 227,0 2736,0 786,438 2736,3648 596,794' -flatten -crop 794x794+0+0! +repage public/warps/1558824229/w132014.png

It's 20 mins until the next command, but i couldn't find any errors between them, besides a lot of garbage collection:

2019-05-26 01:20:52.329 EDT convert +antialias -size 794x794 xc:none -draw "fill black stroke red stroke-width 30 polyline 0,522 227,0 786,438 596,794 0,522" -alpha set -channel A -transparent red -blur 0x8 -channel R -evaluate set 0 +channel public/warps/1558824229/w132014-mask.png

jywarren commented 5 years ago

Maybe we can fetch this file next time: http://export.mapknitter.org/public/warps/1558824229/w132014.png

jywarren commented 5 years ago

Actually wait. Maybe we are seeing 2 types of cutoffs here. The one in the last image is diagonal bc the error was pre-distortion, i think? But the cutoff images in the big export outputs in this comment are all completely horizontal, as if they got cut off during the final compositing step.

jywarren commented 5 years ago

That final compositing step is:

gdalwarp -s_srs EPSG:3857 -te -73.1669712067 44.053772949 -73.1613063812 44.0582295713 public/warps/1558800531/w131950.tif public/warps/1558800531/w131954.tif public/warps/1558800531/w131958.tif public/warps/1558800531/w131962.tif public/warps/1558800531/w131966.tif public/warps/1558800531/w131970.tif public/warps/1558800531/w131974.tif public/warps/1558800531/w131978.tif public/warps/1558800531/w131982.tif public/warps/1558800531/w131986.tif public/warps/1558800531/w131990.tif public/warps/1558800531/w131994.tif public/warps/1558800531/w131998.tif public/warps/1558800531/w132002.tif public/warps/1558800531/w132006.tif public/warps/1558800531/w132010.tif public/warps/1558800531/w132014.tif public/warps/1558800531/w132018.tif public/warps/1558800531/w132022.tif public/warps/1558800531/w132030.tif public/warps/1558800531/w132034.tif public/warps/1558800531/w132038.tif public/warps/1558800531/w132042.tif public/warps/1558800531/w132046.tif public/warps/1558800531/w132118.tif public/warps/1558800531/w132150.tif public/warps/1558800531/w132154.tif public/warps/1558800531/w132158.tif public/warps/1558800531/w132162.tif public/warps/1558800531/w132166.tif public/warps/1558800531/w132170.tif public/warps/1558800531/w132178.tif public/warps/1558800531/w132182.tif public/warps/1558800531/w132186.tif public/warps/1558800531/w132190.tif public/warps/1558800531/w132194.tif public/warps/1558800531/w132198.tif public/warps/1558800531/w132202.tif public/warps/1558800531/w132206.tif public/warps/1558800531/w132210.tif public/warps/1558800531/w132214.tif public/warps/1558800531/w132218.tif public/warps/1558800531/w132222.tif public/warps/1558800531/w132226.tif public/warps/1558800531/w132230.tif public/warps/1558800531/w132234.tif public/warps/1558800531/w132238.tif public/warps/1558800531/w132242.tif public/warps/1558800531/1558800531.tif

here: https://console.cloud.google.com/logs/viewer?authuser=1&project=public-lab&minLogLevel=0&expandAll=false&timestamp=2019-05-27T23%3A11%3A47.765000000Z&customFacets&limitCustomFacetWidth=true&dateRangeStart=2019-05-20T23%3A21%3A11.415Z&interval=P7D&resource=k8s_container%2Fcluster_name%2Fmapknitter&scrollTimestamp=2019-05-26T07%3A15%3A23.632082370Z&dateRangeEnd=2019-05-27T23%3A21%3A11.415Z&filters=text%3A132014&advancedFilter=resource.type%3D%22k8s_container%22%0Aresource.labels.container_name%3D%22mapknitter-exporter-kubes%22%0Aresource.labels.namespace_name%3D%22default%22%0Aresource.labels.location%3D%22us-east1%22%0Aresource.labels.project_id%3D%22public-lab%22%0Aresource.labels.pod_name%3D%22mapknitter-exporter-kubes-6754bb8c54-q6qwt%22%0Aresource.labels.cluster_name%3D%22mapknitter%22%0Atimestamp%3D%222019-05-26T07%3A15%3A23.632082370Z%22%0AinsertId%3D%22f5rcvu9l8f5t5y6sr%22

jywarren commented 5 years ago

This is the big image cut off in the final composite: https://s3.amazonaws.com/grassrootsmapping/warpables/47243/OVPKAP-20111028-85-170_medium.jpg

So, we should restart and try to catch that image in .tif before it's composited to confirm it's at the compositing step that it fails.

jywarren commented 5 years ago

We can try increasing memory at https://github.com/publiclab/mapknitter-exporter-sinatra/blob/main/nolimit.xml

And/or set up a try/retry block?

jywarren commented 5 years ago

We could use this as a template: https://github.com/publiclab/mapknitter/blob/c391beb4e4b27a92bb0a2c11c68bba782145416a/nolimit.xml

jywarren commented 5 years ago

Oh, actually we've already applied that.