Gaps In Sectional Database

[ssokol]

I'm using chartmaker to build just the sectionals database. The node app runs cleanly (no error messages) but I seem to be missing data at certain zoom levels - please see the images below.

Zoomed Out:

Zoomed In:

Please let me know if you need any additional information, want a copy of the mbtiles file, or any of the intermediate files.

[N129BZ]

A couple questions:

1. What OS are you using
2. Can you post the contents of your settings.json file

Thank you N129BZ

[ssokol]

Sure! Debian 12.5 ARM 64 (running as a VM under Parallels on my M2 Mac) and...

{
    "attribution": "Aviation charts <a href='https://github.com/n129bz/chartmaker'>github.com/n129bz/chartmaker</a>",
    "vfrdownloadtemplate": "https://aeronav.faa.gov/visual/<chartdate>/All_Files/<charttype>.zip",
    "ifrdownloadtemplate": "https://aeronav.faa.gov/enroute/<chartdate>/<charttype>.zip",

    "renameworkarea": false,
    "logtofile": false,
    "cleanprocessfolders": false,

    "tileimagequality" : 100,
    "zoomrange" : "0-11",

    "chartprocessindexes": [4],
    "faachartnames": [
        ["Grand_Canyon", "vfr", ""],
        ["Helicopter", "vfr", ""],
        ["Caribbean", "vfr", ""],
        ["Terminal", "vfr", ""],
        ["Sectional", "vfr", ""],
        ["DDECUS", "ifr", "Enroute_Low"],
        ["DDECUS", "ifr", "Enroute_High"]
    ],

    "layertypeindex": 1,
    "layertypes": [
        "baselayer",
        "overlay"
    ],

    "tiledriverindex": 0,
    "tiledrivers": [
        "png",
        "jpg",
        "webp"
    ]
}

[N129BZ]

Please try these settings changes: Set zoomrange to 0-12 Set tileimagequality to 80

I just ran the app with those settings and I can't duplicate the broken images you are seeing, so I think it may be a zoomlevel issue. What application are you using to view the map?

[ssokol]

I tried to run the process with those changes but ran out of disk space. Rebuilding with more space. Please stand by...

[ssokol]

Sorry for the delay. I'm getting ready for the big show at Oshkosh...

I rebuilt my build system (VM), re-ran the install, and re-ran chartmaker. I'm still getting gaps:

Here's a link to a video I recorded showing how this looks as I zoom in / out.

https://youtu.be/CScCqY96AxE

[ssokol]

For whatever it's worth, I've tried using both OpenLayers3 and Leaflet - same results. I've also tried manually querying the missing tiles and I get 404 Not Found. I can use either library to load the same locations from an OSM mbtiles file.

I'm starting to wonder if there's a version difference in the GDAL (or some other library) that's installed with the latest Debian.

[N129BZ]

Steven, I really appreciate your patience with this... I'm going to audit my machine for version #'s and all other components and I'll get back. In the meantime, do you have a Google drive or somewhere accessible for me to get your mbtiles database? I'm using a specialized app I wrote to display the map and I would like to test your dB with that app.

Thanks Brian Manlove

On Mon, Jul 1, 2024, 8:50 AM Steven Sokol @.***> wrote:

For whatever it's worth, I've tried using both OpenLayers3 and Leaflet - same results. I've also tried manually querying the missing tiles and I get 404 Not Found. I can use either library to load the same locations from an OSM mbtiles file.

I'm starting to wonder if there's a version difference in the GDAL (or some other library) that's installed with the latest Debian.

— Reply to this email directly, view it on GitHub https://github.com/N129BZ/chartmaker/issues/3#issuecomment-2200213707, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALK77SCSF73XQ7KOZTMZTXDZKFNCLAVCNFSM6AAAAABJYVFXESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBQGIYTGNZQG4 . You are receiving this because you were assigned.Message ID: @.***>

[N129BZ]

Steven - Results of audit:

Chartmaker Required Components:

gdal 3.6.2 unZip, version 6.00 pngquant, version 2.17.0 python 3.11 (minimum) perl 5.36.0

Perl is used for converging images. There are thousands of redundant images at high zoom levels because there is overlap between all of the individual sectional charts at their borders. I think this may be part of the problem. The following perl functions/utilities must be installed, they are all within perlsetup.sh:

cpan strict cpan warnings cpan autodie cpan Carp cpan Modern::Perl cpan Params::Validate cpan File::Slurp cpan File::Copy

If not already done, make perlsetup.sh executable and run it, and these dependencies should then be functional.

[jamez70]

I tried this about 6 months ago and had the same issue. I think I rolled back to overcome it I can't remember. I'll check later to see what I did.

Jim

On Mon, Jul 1, 2024, 5:12 PM Brian Manlove @.***> wrote:

Steven, I really appreciate your patience with this... I'm going to audit my machine for version #'s and all other components and I'll get back. In the meantime, do you have a Google drive or somewhere accessible for me to get your mbtiles database? I'm using a specialized app I wrote to display the map and I would like to test your dB with that app.

Thanks Brian Manlove

On Mon, Jul 1, 2024, 8:50 AM Steven Sokol @.***> wrote:

For whatever it's worth, I've tried using both OpenLayers3 and Leaflet - same results. I've also tried manually querying the missing tiles and I get 404 Not Found. I can use either library to load the same locations from an OSM mbtiles file.

I'm starting to wonder if there's a version difference in the GDAL (or some other library) that's installed with the latest Debian.

— Reply to this email directly, view it on GitHub https://github.com/N129BZ/chartmaker/issues/3#issuecomment-2200213707,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/ALK77SCSF73XQ7KOZTMZTXDZKFNCLAVCNFSM6AAAAABJYVFXESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBQGIYTGNZQG4>

. You are receiving this because you were assigned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/N129BZ/chartmaker/issues/3#issuecomment-2201165400, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIVBUYLG23TKGANDFBZCVTZKHH3TAVCNFSM6AAAAABJYVFXESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBRGE3DKNBQGA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[N129BZ]

Also if not already done, please enable logging by setting logtofile=true in settings.json

P.S. hello jamez70!!

[ssokol]

Working on getting the database uploaded. I'll send you a Google drive link shortly.

Here's the my audit of installed software:

GDAL 3.6.2, released 2023/01/02 UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP. pngquant --version 2.17.0 (September 2021) Python 3.11.2 This is perl 5, version 36, subversion 0 (v5.36.0) built for aarch64-linux-gnu-thread-multi

I ran the perlsetup.sh file included with the chartmaker project. It seemed to build as expected. Is there any way to check the versions of those cpan libs?

[ssokol]

I re-ran the perlsetup.sh and here's the output:

root@charts2:/home/ssokol/chartmaker# ./perlsetup.sh
Loading internal logger. Log::Log4perl recommended for better logging
Reading '/root/.cpan/Metadata'
  Database was generated on Wed, 26 Jun 2024 02:17:01 GMT
Fetching with HTTP::Tiny:
https://cpan.org/authors/01mailrc.txt.gz
Reading '/root/.cpan/sources/authors/01mailrc.txt.gz'
............................................................................DONE
Fetching with HTTP::Tiny:
https://cpan.org/modules/02packages.details.txt.gz
Reading '/root/.cpan/sources/modules/02packages.details.txt.gz'
  Database was generated on Mon, 01 Jul 2024 20:52:43 GMT
..............
  New CPAN.pm version (v2.36) available.
  [Currently running version is v2.33]
  You might want to try
    install CPAN
    reload cpan
  to both upgrade CPAN.pm and run the new version without leaving
  the current session.

..............................................................DONE
Fetching with HTTP::Tiny:
https://cpan.org/modules/03modlist.data.gz
Reading '/root/.cpan/sources/modules/03modlist.data.gz'
DONE
Writing /root/.cpan/Metadata
Running install for module 'strict'
The most recent version "1.13" of the module "strict"
is part of the perl-5.40.0 distribution. To install that, you need to run
  force install strict   --or--
  install HAARG/perl-5.40.0.tar.gz
Loading internal logger. Log::Log4perl recommended for better logging
Reading '/root/.cpan/Metadata'
  Database was generated on Mon, 01 Jul 2024 20:52:43 GMT
Running install for module 'warnings'
The most recent version "1.70" of the module "warnings"
is part of the perl-5.40.0 distribution. To install that, you need to run
  force install warnings   --or--
  install HAARG/perl-5.40.0.tar.gz
Loading internal logger. Log::Log4perl recommended for better logging
Reading '/root/.cpan/Metadata'
  Database was generated on Mon, 01 Jul 2024 20:52:43 GMT
autodie is up to date (2.37).
Loading internal logger. Log::Log4perl recommended for better logging
Reading '/root/.cpan/Metadata'
  Database was generated on Mon, 01 Jul 2024 20:52:43 GMT
Carp is up to date (1.52).
Loading internal logger. Log::Log4perl recommended for better logging
Reading '/root/.cpan/Metadata'
  Database was generated on Mon, 01 Jul 2024 20:52:43 GMT
Modern::Perl is up to date (1.20240115).
Loading internal logger. Log::Log4perl recommended for better logging
Reading '/root/.cpan/Metadata'
  Database was generated on Mon, 01 Jul 2024 20:52:43 GMT
Params::Validate is up to date (1.31).
Loading internal logger. Log::Log4perl recommended for better logging
Reading '/root/.cpan/Metadata'
  Database was generated on Mon, 01 Jul 2024 20:52:43 GMT
File::Slurp is up to date (9999.32).
Loading internal logger. Log::Log4perl recommended for better logging
Reading '/root/.cpan/Metadata'
  Database was generated on Mon, 01 Jul 2024 20:52:43 GMT
Running install for module 'File::Copy'
The most recent version "2.41" of the module "File::Copy"
is part of the perl-5.40.0 distribution. To install that, you need to run
  force install File::Copy   --or--
  install HAARG/perl-5.40.0.tar.gz

[ssokol]

Here's the folder with the Sectionals database:

https://drive.google.com/drive/folders/1BEvgG9f3kHzwgRg0-V4CalsLmRRxbFlY?usp=sharing

Please let me know when you have this downloaded so I can disable the share.

[N129BZ]

What version of sqlite3 is on your machine? Mine is 3.40.1 2022-12-28. I missed citing this as another dependency in the readme... now updated.

[jamez70]

We should create either an lxc or docker image of everything and build it using that instead of trying to install all the obtuse versions needed in order to make it work perfectly.

Jim

On Tue, Jul 2, 2024 at 11:49 PM Brian Manlove @.***> wrote:

What version of sqlite3 is on your machine?

— Reply to this email directly, view it on GitHub https://github.com/N129BZ/chartmaker/issues/3#issuecomment-2205094256, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIVBU47FURHIR2VFZVYKS3ZKN7EJAVCNFSM6AAAAABJYVFXESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBVGA4TIMRVGY . You are receiving this because you commented.Message ID: @.***>

[N129BZ]

ssokol, I downloaded your sectional.db and I get this error when I try to load it in my map viewing application:

Error: SQLITE_NOTADB: file is not a database {errno: 26, code: 'SQLITE_NOTADB', stack: 'Error: SQLITE_NOTADB: file is not a database', message: 'SQLITE_NOTADB: file is not a database'}

[N129BZ]

jamez70, I agree... I will be doing this in the near future.

[N129BZ]

Now testing a fully self-contained Docker image, will upload results when done processing the full US sectional chart...

[jamez70]

Great idea then all of us can do it with no confusion

Jim

On Wed, Jul 3, 2024, 10:18 AM Brian Manlove @.***> wrote:

Now testing a fully self-contained Docker image, will upload results when done processing the full US sectional chart...

— Reply to this email directly, view it on GitHub https://github.com/N129BZ/chartmaker/issues/3#issuecomment-2206511905, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIVBUYCWDHIJ2T5RRKUPJTZKQI3RAVCNFSM6AAAAABJYVFXESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBWGUYTCOJQGU . You are receiving this because you commented.Message ID: @.***>

[ssokol]

Apologies. There's something that seems to happen to large, complex files when uploaded to / downloaded from Google Docs. I've had the same thing happen with complex .STEP file CAD models.

I can try to upload it to one of my web servers if it will still be of use.

[N129BZ]

No problem, I may have duplicated your issue. Working on it!

[ssokol]

👍

Thank you very much - both for working on the bug and also for putting this project together in the first place.

[N129BZ]

I believe I have figured out what is going on. For whatever reason, the Perl objects were not actually being installed by the perlsetup.sh script. I manually ran each cpan install line and several objects were not successfully installed. I then installed cpanminus and did a forced install using cpanm -i -f for each line and then ran the app and I got a solid map. I have created a Docker image with everything and tested that and also got a solid map. I will upload that to my google drive for you to test and send you an email when it's ready, probably around noon.

[N129BZ]

Download the n129bz.chartmaker.v1.09.tar Docker image at:

https://drive.google.com/file/d/1UQyeotVQAOPMaq6bO5i1TjLMJhjgJ-k7/view?usp=drive_link

Load the docker image with command: docker load -i n129bz.chartmaker.v1.09.tar view Docker images via command: docker images copy the id for the image and in a terminal, run: docker run -it [image id]

This will run the image in a terminal. At the prompt enter : cd /home/chartmaker node make

[ssokol]

Thanks! Running the docker image now. First time I've ever used docker (directly - I'm sure it's on the back end of all kinds of thing I use regularly).

[N129BZ]

I just noticed you're in Cedar Park -- I'm in Burnet and my hangar is in Lampasas!!

[N129BZ]

Also - to copy the db file from the docker image to your local filesystem:

docker cp [imageid]:/home/chartmaker/workarea/Sectional.db [local path to copy to]

[jamez70]

I'm going to give it a try shortly I'll let you know how it goes. Thank you for preparing this.. a much better idea than trying to get different machines running it

On Sun, Jul 7, 2024, 10:50 PM Brian Manlove @.***> wrote:

Also - to copy the db file from the docker image to your local filesystem:

docker cp [imageid]:/home/chartmaker/workarea/Sectional.db [local path to copy to]

— Reply to this email directly, view it on GitHub https://github.com/N129BZ/chartmaker/issues/3#issuecomment-2212949402, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIVBUZQPJG34W33OFV7D2TZLIEALAVCNFSM6AAAAABJYVFXESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJSHE2DSNBQGI . You are receiving this because you commented.Message ID: @.***>

[ssokol]

I think I ran out of space on the docker drive:

Command failed: cp -f /home/chartmaker/workarea/Sectional/5_merged/12/501/2908.png /home/chartmaker/workarea/Sectional/6_quantized/12/501/2908.png
  error: failed writing image to /home/chartmaker/workarea/Sectional/6_quantized/12/501/2909.png (16)
Command failed: cp -f /home/chartmaker/workarea/Sectional/5_merged/12/501/2909.png /home/chartmaker/workarea/Sectional/6_quantized/12/501/2909.png
  error: failed writing image to /home/chartmaker/workarea/Sectional/6_quantized/12/501/2910.png (16)
Command failed: cp -f /home/chartmaker/workarea/Sectional/5_merged/12/501/2910.png /home/chartmaker/workarea/Sectional/6_quantized/12/501/2910.png
  error: failed writing image to /home/chartmaker/workarea/Sectional/6_quantized/12/501/2911.png (16)
Command failed: cp -f /home/chartmaker/workarea/Sectional/5_merged/12/501/2911.png /home/chartmaker/workarea/Sectional/6_quantized/12/501/2911.png
  error: failed writing image to /home/chartmaker/workarea/Sectional/6_quantized/12/501/2912.png (16)
Command failed: cp -f /home/chartmaker/workarea/Sectional/5_merged/12/501/2912.png /home/chartmaker/workarea/Sectional/6_quantized/12/501/2912.png
Command failed: cp -f /home/chartmaker/workarea/Sectional/5_merged/12/501/3003.png /home/chartmaker/workarea/Sectional/6_quantized/12/501/3003.png
  error: failed writing image to /home/chartmaker/workarea/Sectional/6_quantized/12/501/3004.png (16)
^C
root@8f832e16334f:/home/chartmaker#
root@8f832e16334f:/home/chartmaker# ls
LICENSE    chartInformation  chartdates.json  helpfiles  mbutil         node_modules       package.json  settings.json
README.md  chartcache        clipshapes       make.js    mergetiles.pl  package-lock.json  perlsetup.sh  workarea
root@8f832e16334f:/home/chartmaker# cd workarea/
root@8f832e16334f:/home/chartmaker/workarea# ls
Sectional
root@8f832e16334f:/home/chartmaker/workarea# cd Sectional/
root@8f832e16334f:/home/chartmaker/workarea/Sectional# ls
1_unzipped  2_expanded  3_clipped  4_tiled  5_merged  6_quantized
root@8f832e16334f:/home/chartmaker/workarea/Sectional# df
Filesystem     1K-blocks     Used Available Use% Mounted on
overlay         61202244 61185860         0 100% /
tmpfs              65536        0     65536   0% /dev
shm                65536        0     65536   0% /dev/shm
/dev/vda1       61202244 61185860         0 100% /etc/hosts
tmpfs            4014624        0   4014624   0% /sys/firmware
root@8f832e16334f:/home/chartmaker/workarea/Sectional#

How do I expand the filesystem? The underlying machine has plenty of storage.

[ssokol]

Well then 'howdy, neighbor!' Yes - I'm at Breakaway air park just off of Parmer Lane in Cedar Park.

[N129BZ]

Docker doesn't have a filesystem size limit other than the machine its physically running on or some limit in its configuration json file. I hate to say it but "it works on my machine..." which is WSL (ubuntu) on windows 10. My next test of the docker image will be on a full ubuntu mate machine, will report those results later.

Also - assuming you get it to work, DO NOT exit the docker container terminal, or everything will be lost. To copy the database you have to use the docker cp command from a separate terminal not connected to the docker container. In my case the id of the running container is fb2159899a89 so in terminal 2 it's docker cp fb2159899a89:/home/chartmaker/workarea/Sectional.db ./

[N129BZ]

If you do want to save the state of the container "as-is," before exiting it, a command in separate terminal 2 would be (using the actual container id, not mine below)

docker commit fb2159899a89 myid/mycontainername:version

[N129BZ]

I found this Quora article, may be some help:

[N129BZ]

Also you can get a lot of info on your docker installation with command docker info

[ssokol]

I think I found what I needed to adjust. I'm using the Docker Desktop (lame, I know, but I got out of corporate tech and into avionics just as Docker and other containerization tools were starting to become popular) and it set a 64 GB limit on virtual disk space. I've upped it to 256 and and am trying again.

[ssokol]

Still running. 5 hours in now. At the quantizing PNG images stage. Appears to be taking about 1 minute per 1000 images, and apparently the total count of images is 742,584 so it looks like it will take about 12.36 hours to get through the whole batch. Docker is running 110% so apparently this phase isn't parallelizable (the instance has 4 CPUs and was running at 400% during the earlier stages).

I think I may want to go through the process to get it running natively on the Mac. Not sure if that will be significantly faster but it's worth a shot.

[N129BZ]

I just finished running the docker image on my dedicated ubuntu machine and it completed successfully in 6 hours. Your image count sounds exactly correct, mine reported 742584 png images, so I think you'll be good. I also have the docker image running right now on my 8gb Mac Mini and it has been chugging away for 5 hours now. I don't expect that to be done anytime soon, it's a bit of a dinosaur but it works.

FWIW, I have never even tried to get it to run natively on Windows. I've haven't had the best luck with node apps running on Windows, especially with OS shell commands, and had to add Windows specific code to get things to work correctly, whereas linux shell support in node is pretty robust in is a relative "piece of cake."

I also have never tried it natively on the Mac, I just assumed it would be a nightmare to get everything to work, but if you're game for that and you get it to work I'll buy you a beer sometime!

[N129BZ]

P.S., I thinking about adding a Wiki to the project with a lot more detailed info on the exact requirements to run it outside of docker, as well as how to create a docker image with everything needed...

[N129BZ]

Issue closed. Added new dependency list in readme and will add detailed instructions on proper installation.

[N129BZ]

Added wiki page at https://github.com/n129bz/chartmaker/wiki

[ssokol]

My build finished yesterday - all looks good. Thank you very much for the help!

[N129BZ]

You're welcome. Thanks for using the app and reporting the issue!!

Brian Manlove N129BZ

On Wed, Jul 10, 2024, 9:26 AM Steven Sokol @.***> wrote:

My build finished yesterday - all looks good. Thank you very much for the help!

— Reply to this email directly, view it on GitHub https://github.com/N129BZ/chartmaker/issues/3#issuecomment-2220661615, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALK77SC3VEO34KDLYG2KVPLZLVAAZAVCNFSM6AAAAABJYVFXESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRQGY3DCNRRGU . You are receiving this because you modified the open/close state.Message ID: @.***>

[N129BZ]

I just ran the application natively (no docker) on my 8gb Mac Mini and it took 27 hours to do the Sectional.

[N129BZ]

Just a post-script note: Processing the full sectional at 60% quality using WEBP image format took under 2 hours, and quality is extremely good: