springmeyer commented 12 years ago

Like MrSid, that is a common distribution format for NAIP imagery. This will require compiling against the lizardtech (or otherwise) SDK's and providing a separate download, then setting GDAL_DRIVER_PATH to point to the libraries

pnorman commented 10 years ago

I have gone to great efforts to get gdal 1.10 compiled with support for all the formats I need. It would be nice if I could use the system gdal, which has support for mrsid, jpeg2000, ecw and so on.

springmeyer commented 10 years ago

@pnorman - you can do that: you just need to 1) ensure your Mapnik is built against that gdal version and ) 2 pass 'type=gdal' to the advanced options when creating your layers so that the Mapnik gdal plugin is used. TileMill currently depends on millstone to declare which file extensions are known and how they map to which Mapnik datasource plugins. The 'type=gdal' is needed for extensions like ecw that would otherwise not be known.

pnorman commented 10 years ago

What's the best way to check what gdal location mapnik is built against?

springmeyer commented 10 years ago

What's the best way to check what gdal location mapnik is built against?

$ ldd `mapnik-config --input-plugins`/gdal.input | grep libgdal

or on OS X:

$ otool -L `mapnik-config --input-plugins`/gdal.input | grep libgdal
    /usr/local/lib/libgdal.1.dylib (compatibility version 19.0.0, current version 19.1.0)

pnorman commented 10 years ago

will mapnik-config always report the details for the mapnik install being used by tilemill?

Anyways, it looks like mapnik 2.2 from PPA is relying on /usr/lib/libgdal1.7.0.so.1 from libgdal1-1.7.0, not the 1.10 gdal I installed or the 1.9.0 in the postgres repo. This explains why tilemill can't display ecws, because mapnik is built against the wrong gdal.

Dependencies between packages and from source are a pain.

springmeyer commented 10 years ago

will mapnik-config always report the details for the mapnik install being used by tilemill?

It will report the details for the mapnik version that is on your PATH. If you are using TileMill from packages then it will be the packaged Mapnik via /usr/bin/mapnik-config. If you have source installed Mapnik then it will be wherever you installed mapnik.

This explains why tilemill can't display ecws, because mapnik is built against the wrong gdal.

Just remove the packaged TileMill and packaged Mapnik, install them both from source you'll be good to go.

laurentades commented 10 years ago

@springmeyer Sorry i might sound totally newbie (that's the case...), but can you confirm i got the above right ?

Using GDAL raster formats such as ECW is possible but it will require to build/compile tilemill from source whilst making sure that GDAL_DRIVER_PATH (God knows where that one is, but I guess i'll find it....) points to a GDAL install i have prepared on my system to cater for ECW (for example). To this extent, i first need to make sure that mapnik (which as far as i understand i s the rendering engine of tilemill, right?) is built against the same GDAL instance, thus meaning i also need to build my own brew of mapnik. Correct ?

So far i have been using the www.mapbox.com/tilemill/ downloadable app (os x). Reading from the tilemill project homepage and installation instruction from source, I won't be getting a .app as such but rather a web app accessible from a browser on port 20009 which should behave (hopefully) the same as the downloadable desktop app, right ?

It kinda makes sense for me, is that all correct before i get deep into all this ?

Thanks

springmeyer commented 10 years ago

Yes, you've got it basically right. It is a big effort. Can you explain your need more? What format are you trying to read? And why is it impracticable to convert to geotiff?

On Jun 13, 2014, at 3:33 AM, laurentades notifications@github.com wrote:

@springmeyer Sorry i might sound totally newbie (that's the case...), but can you confirm i got the above right ?

Using GDAL raster formats such as ECW is possible but it will require to build/compile tilemill from source whilst making sure that GDAL_DRIVER_PATH (God knows where that one is, but I guess i'll find it....) points to a GDAL install i have prepared on my system to cater for ECW (for example). To this extent, i first need to make sure that mapnik (which as far as i understand i s the rendering engine of tilemill, right?) is built against the same GDAL instance, thus meaning i also need to build my own brew of mapnik. Correct ?

So far i have been using the www.mapbox.com/tilemill/ downloadable app (os x). Reading from the tilemill project homepage and installation instruction from source, I won't be getting a .app as such but rather a web app accessible from a browser on port 20009 which should behave (hopefully) the same as the downloadable desktop app, right ?

It kinda makes sense for me, is that all correct before i get deep into all this ?

Thanks

— Reply to this email directly or view it on GitHub.

laurentades commented 10 years ago

I need to include a set of ECW files in map to make a aerial photography layer in Tilemill. . GeoTiFF sounds fine except it's the first time i'm getting to this step. I guess the challenge is really about getting all those dependencies, libraries and what not all lined up so that the thing work smoothly in production.

Happy if you can point me to the right directions:

GDAL incl ECW : i guess this one should be ok as i've been playing around with kyngchaos 'packages for Mac OS X (the boxes i'm using)
Mapnik: not too sure where to tell this bad boy whow to point to ly local GDAL brew rather than the one originally packaged on git
Building TileMill should be ok as long as i can find the dreaded GDAL_DRIVER_PATH
Running Tilemill from a web interface should not be very different from running it as a packaged app...

So.... If you can help with those bits and bops above....

Many thanks

pnorman commented 10 years ago

And why is it impracticable to convert to geotiff?

In my case, because it means some combination of slower, larger, or lower quality, depending on the exact settings used. I haven't actually needed tilemill support lately, largely because I haven't needed any raster support in tilemill.

laurentades commented 10 years ago

@springmeyer sorry to bug you. Just checking if you think you can point me in the right direction from my last post. Many thanks

laurentades commented 10 years ago

Sometimes it is a good thing not to receive answers : gets you going to find solutions on your own :-)

I eventually came to the conclusion that as my ecw source does not change much, it would be easier to use gdal from the CLI in a bash script to process all thousands of ecw files i have and output geotiff.

Now the following question needs help: i'm talking 'bout thousands of ecw's. What is the way to avoid having to create one layer per ecw>tif file ? Is it possible with gdal to create some massive single geotiff file ? or rather point a tilemill layer to a unique geotiff directory ? or should i try and combine everything in a single kHz using gdal_translate ?

Thanks

pnorman commented 10 years ago

or rather point a tilemill layer to a unique geotiff directory

You can create a virtual file which gdal will read. I don't know much about them, using mapserver for my pure imagery needs, where I use gdaltindex. Does Mapnik work with gdaltindex?

I would not suggest merging to one file.

springmeyer commented 10 years ago

Use VRT files (http://blog.perrygeo.net/2010/02/18/lazy-raster-processing-with-gdal-vrts/). They can be used to reference multiple rasters when processing/converting to geotiff. And they also can be used to load many rasters into TileMill. Just make sure to use absolute paths inside the VRT to the files they reference otherwise TileMill will have trouble reading the referenced files.

laurentades commented 10 years ago

Ok, it all ads up now. Will get going with that. Thanks for your precious input!

laurentades commented 10 years ago

Almost there.... Got my 2,800+ tiff render with jpg compression and a nice vrt file as well. Now.... this is taking quite a bit of time to render in tilemill as each tile is.... 6,000x6,000. Obviously the activity monitor shows all cpu cores spinning at full throttle for quite long... and eventually display nothing.... Is there any rule of thumb to optimise raster mosaic image size ?

laurentades commented 10 years ago

Now, spent the week-end reading GDAL stuff incl. gdaladdo and experimenting various stuff. Each run of making a vrt and building overviews is quite lengthy... So i realise gdal raster processing is a science of its own. Any good recourse i can turn to helping me understand how to optimise all that ? GDAL user guide is well documented but assumes a lot that the user knows its way around fundamental concepts of raster processing....

laurentades commented 10 years ago

ok, after much trial and error, i managed to:

gdal_translate all those ecw to geotiff keeping the original 6000x6000 dimensions with jpeg compression. Final output of 2,800 files is 15GB
build vrt
build overviews (2 > 16) over the vrt file. The .ovr file is about 4.8GB. I had to use the BIGTIFF_OVERVIEW option with gdalddo and COMPRESS_OVERVIEW JPEG as well.

Added this vrt as a TM layer... and.... i have huge black square box... ! When i build a much smaller vrt (5x4 tiff = 30,000x24,000) + overview with same options i get a nice aerial image as expected... So it would look like it has something to do with final dimension of the vrt...

?

laurentades commented 10 years ago

ok, sorry i'm bombing this thread with posts but it looks like this issue (https://github.com/mapbox/tilemill/issues/2104) is exactly what i'm after... Any news on these tutos ?

laurentades commented 10 years ago

I found out what i was doing wrong:

from @springmeyer

Just make sure to use absolute paths inside the VRT to the files they reference otherwise TileMill will have trouble reading the referenced files.

My demo vrt is working fine as it is built on a list of tif files in an input text file (-input_file_list option for gdalbuildvrt) as i wanted to only select a subset of those tifs, but when building the big vrt i was only specifying *.tif...

somebodyslapme!

rcoup commented 10 years ago

FWIW, a newer GDAL also solves this issue - Tilemill symlinks the VRT inside your project, and GDAL used to treat paths in the VRT as relative to the symlink location not the actual VRT location.

laurentades commented 10 years ago

Now this is weird: I have 2,816 files in my vrt. When i try and display it, it just won't or will display pieces of it, leaving other parts blank. It works with smaller vrt's. With trial and error i worked out that it breaks at 2,633 files (2,632 displays fine)! My vrt is built on file list in a text file (-input_file_list option for gdalbuildvrt) I first thought the issue was with some specific file breaking the thing but the result is the same whether i trim the number of files down from the beginning or the end of the list. Is that a GDAL issue ? Did i break something ?

rcoup commented 10 years ago

If you run "export CPL_DEBUG=ON" before running tilemill do you get any informative messages?

Otherwise maybe the number of open files is hitting a limit? http://stackoverflow.com/questions/34588/how-do-i-change-the-number-of-open-files-limit-in-linux has mac/Linux instructions

On Monday, 14 July 2014, Laurent Adès notifications@github.com wrote:

Now this is weird: I have 2,816 files in my vat. When i try and display it, it just won't or will display pieces of it, leaving other parts blank. It works with smaller vrt's. With trial and error i worked out that it breaks at 2,633 files (2,632 displays fine)! My vrt is built on file list in a text file (-input_file_list option for gdalbuildvrt) I first thought the issue was with some specific file breaking the thing but the result is the same whether i trim the number of files down from the beginning or the end of the list. Is that a GDAL issue ? Did i break something ?

— Reply to this email directly or view it on GitHub https://github.com/mapbox/tilemill/issues/1403#issuecomment-48871932.

laurentades commented 10 years ago

"export CPL_DEBUG=ON" did not produce anything helpful, but i have a feeling this is because i'm working off the tilemill binary....

ulimit -a gives:

core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 256 pipe size (512 bytes, -p) 1 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 709 virtual memory (kbytes, -v) unlimited

Which does not seem helpful either : if the open files limit is set to 256, then i would have hit this limit long ago since the issue is happening between 2,632 and 2,633 files...

laurentades commented 10 years ago

For what it is worth: loading the full vrt in QGis works fine. Displays as it should. Is something broken in TM ?

winsento commented 7 years ago

And what about the situation in 2017? I can build mapnik with support for ECW (Ubuntu), provided that GDAL + ECW already installed.

tilemill-project / tilemill

Support proprietary GDAL raster formats #1403

somebodyslapme!