search

publiclab / mapknitter-exporter

The GDAL/ImageMagick-based exporter system from MapKnitter
GNU General Public License v3.0
5 stars 4 forks source link

simplify run_export #6

Closed jywarren closed 5 years ago

jywarren commented 5 years ago

Merge once we resolve everything and after #4; also refactor https://github.com/publiclab/mapknitter/ to use this, and make downstream adjustments in https://github.com/publiclab/mapknitter-exporter-sinatra/

Changes:

turned slug into id (will need test fixes), removed average_scale

def self.run_export(user_id, resolution, export, id, slug, root, average_scale, placed_warpables, key) became def self.run_export(user_id, resolution, export, id, root, placed_warpables, key)

def self.generate_perspectival_distort(pxperm, path, nodes_array, id, image_file_name, img_url, height, width, root = "https://mapknitter.org") became def self.generate_perspectival_distort(pxperm, id, nodes_array, image_file_name, img_url, height, width, root = "https://mapknitter.org")

jywarren commented 5 years ago

Getting stuck where it can't find the output of the previous file...


gdal_translate -of GTiff -a_srs EPSG:4326  -gcp 0.0, 0.0, -71.3983854668186, 41.8403113680142 -gcp 1500.0766798835248, 171.98331362381577, -71.3916477577732, 41.8397358653566 -gcp 1265.9882808048278, 1543.072508284822, -71.392699183707, 41.8351476451765 -gcp 47.773142671212554, 764.3702827505767, -71.3981708900974, 41.8377535388085  -co "TILED=NO" public/warps/1/1-masked.png public/warps/1/1-geo-unwarped.tif
nice: gdal_translate: No such file or directory
gdalwarp -of GTiff -t_srs EPSG:4326 public/warps/1/1-geo-unwarped.tif public/warps/1/1-geo.tif
nice: gdalwarp: No such file or directory
rm: cannot remove ‘public/warps/1/1-geo.tif’: No such file or directory

I fixed a few but have to step away; @icarito would you be able to take this forward by looking at the output of each step as it goes to the next?

jywarren commented 5 years ago

Now let's rebase this and complete the simplification...

jywarren commented 5 years ago

attempted a rebase at https://github.com/publiclab/mapknitter-exporter/pull/9, but kept it separate just to compare to be sure i've done it right.

jywarren commented 5 years ago

OK, the rebase was good; force pushing it here and closing the other PR: #9

jywarren commented 5 years ago

Hmm, this looks like an issue with the .travis.yml


Warning: apt-key output should not be parsed (stdout is not a terminal)
Executing: /tmp/apt-key-gpghome.KnIUpDVdm4/gpg.1.sh --keyserver fks.pgpkeys.eu --recv-keys BF26EE05EA6A68F0
gpg: keyserver receive failed: No data
The command '/bin/sh -c apt-key adv --keyserver fks.pgpkeys.eu --recv-keys BF26EE05EA6A68F0' returned a non-zero code: 2
jywarren commented 5 years ago

Getting there!

  1) Error:
ExporterTest#test_all_functions:
Errno::ENOENT: No such file or directory @ rb_sysopen - demo.png
jywarren commented 5 years ago

Somehow this is failing on line 85:

FileUtils.cp(img_url, local_location)

Hmm. I'll log out the vars.

icarito commented 5 years ago

Hmm, this looks like an issue with the .travis.yml


Warning: apt-key output should not be parsed (stdout is not a terminal)
Executing: /tmp/apt-key-gpghome.KnIUpDVdm4/gpg.1.sh --keyserver fks.pgpkeys.eu --recv-keys BF26EE05EA6A68F0
gpg: keyserver receive failed: No data
The command '/bin/sh -c apt-key adv --keyserver fks.pgpkeys.eu --recv-keys BF26EE05EA6A68F0' returned a non-zero code: 2

There is a fix for this in main branch - using ipv4 pool (ipv4.pool.sks-keyservers.net) instead!

icarito commented 5 years ago

Ah sorry that fix is in mapknitter main repo

jywarren commented 5 years ago

😄 


gdalwarp -s_srs EPSG:3857 -te -71.3983854668186 41.8351476451765 -71.3916477577732 41.8403113680142 public/warps/1/1-geo.tif public/warps/1/1-geo.tif
ERROR 1: Output dataset public/warps/1/1-geo.tif exists,
but some command line options were provided indicating a new dataset
should be created.  Please delete existing dataset and run again.
gdal2tiles.py -k --s_srs EPSG:3857 -t 1 -g AIzaSyAOLUQngEmJv0_zcG1xkGq-CXIPpLQY8iQ public/warps/1/1-geo.tif public/tms/1/
Generating Base Tiles:
0...10...20...30...40...50...60...70...80...90...100
rm: cannot remove '1.zip': No such file or directory
convert-im6.q16: unable to open image `/public/warps/1/1-geo.tif': No such file or directory @ error/blob.c/OpenBlob/2701.
convert-im6.q16: no images defined `/public/warps/1/1.jpg' @ error/convert.c/ConvertImageCommand/3258.
F
jywarren commented 5 years ago

Now:


gdalwarp -s_srs EPSG:3857 -te -71.3983854668186 41.8351476451765 -71.3916477577732 41.8403113680142 public/warps/1/1-geo.tif public/warps/1/1-geo.tif
ERROR 4: public/warps/1/1-geo.tif: No such file or directory
gdal2tiles.py -k --s_srs EPSG:3857 -t 1 -g AIzaSyAOLUQngEmJv0_zcG1xkGq-CXIPpLQY8iQ public/warps/1/1-geo.tif public/tms/1/
Usage: gdal2tiles.py [options] input_file [output]
gdal2tiles.py: error: The provided input file public/warps/1/1-geo.tif does not exist or is not a file
jywarren commented 5 years ago

Also noting identify-im6.q16: unknown image property "%[exif:Orientation]" @ warning/property.c/InterpretImageProperties/3831.

jywarren commented 5 years ago

Well, there we go. There are some issues to resolve, though:

convert-im6.q16: unable to open image `/public/warps/1/1-geo.tif': No such file or directory @ error/blob.c/OpenBlob/2701.
convert-im6.q16: no images defined `/public/warps/1/1.jpg' @ error/convert.c/ConvertImageCommand/3258.

And i'm not sure why the test doesn't catch that - any ideas, @icarito?

jywarren commented 5 years ago

Some of these warnings about not being able to delete things are because some of the methods try to delete anything in the location before trying to make something in that location, and I think that's OK to leave:

rm: cannot remove '/public/warps/1/': No such file or directory
rm: cannot remove '/public/tms/1': No such file or directory
jywarren commented 5 years ago

I think this is the last critical bug to resolve --

convert-im6.q16: unable to open image `/public/warps/1/1-geo.tif': No such file or directory @ error/blob.c/OpenBlob/2701.
convert-im6.q16: no images defined `/public/warps/1/1.jpg' @ error/convert.c/ConvertImageCommand/3258.

It's because the final command still runs against /public/... instead of .public/..., so i passed in '.' as root to try to fix it. I'd still like to be able to know that it failed, in a test, though.

jywarren commented 5 years ago

But, if this passes, we can merge it!

jywarren commented 5 years ago

I think we need to address and simplify the usage of "root" -- is it even relevant anymore for how it's structured now?

I think it originally was trying to use Rails.root but now we're mostly using it locally.

I fixed it again. We can do "root" cleanup in a follow-up PR.