search

wdantuma / s57-tiler

S57-tiler creates vectortiles from S57 ENC's which can be used with freeboard-sk
3 stars 0 forks source link

Question: s57-tiler on macOS #3

Closed godind closed 1 month ago

godind commented 2 months ago

Hi,

Can s57-tiler be used on macOS without a Signal K installation? Just to do a format conversion? I am trying to convert NOAA charts for Freeboard SK usage.

Thanks for the help!

wdantuma commented 2 months ago

Hi David,

Using docker it should be possible, i don't own a Mac so i cannot test it for you.

Let me know if you find some bugs using s57-tiler on a Mac, or the converted NOAA charts in Freeboard sk.

Kind regards,

Willem

godind commented 2 months ago

Hi @wdantuma,

I have the same issues as found in issue #2 with docker.

Any chance you can help?

wdantuma commented 2 months ago

Hi David,

Sorry for the delayed response ( i am sailing right now ), can you execute the command

docker build --no-cache --progress=plain . &> build.log instead of docker build . -t s57-tiler and send me the content of the build.log file ?

godind commented 1 month ago

No problem. I'm sailing myself. Will test this week and report back.

Have you looked at converting any of the NOAA charts? They offer MBTiles charts but they appear to be only very high level charts. Maybe the charts already have everything and I just don't know how make the zoom levels work with FreeboardSK...

wdantuma commented 1 month ago

The NOAA offered mbtiles are raster tiles ( .png) and can be dropped into the signalk charts folder below are two screenshots

one of the NOAA offered .mbtiles (https://distribution.charts.noaa.gov/ncds/mbtiles/ncds_01a.mbtiles) image

and a s57-tiler converted .enc ( https://charts.noaa.gov/ENCs/US5MA1VH.zip) image

The difference is the .mbtiles are prerendered and the s57 enc vectortiles are rendered at runtime in freeboard-sk using the configured depth settings ( reflecting how openCPN handles ENC's but the implementation in freeboard-sk is not complete yet and as you can see still seems buggy the channel is rendered with a wrong color, actually this was the first time i tried to convert a NOAA chart )

godind commented 1 month ago

Thanks for the reply.

I'm trying to get Lake Champlain charts. It's the large lake south of Montreal in Canada. I have tried this same ncds_01a mbtiles chart you have and I could only display the very high level chart view.

Sorry for the question but you seem to imply that NOAA's mbtile includes all the details offered in their s-57 version. Have I got this right?

How did you configure the depth settings in FreeboardSK to get this level of details?

Maybe that chart contains what I need, if I can find how to play with depth levels.

wdantuma commented 1 month ago

The "charts" settings in freeboard-sk do not have any effect on displaying NOAA .mbtiles charts because these are raster charts. The "charts" settings in freeboard-sk only change the rendering of with s57-tiler converted enc charts.

Lake Champlain only starts to show detail from zoom level 12.5 and higher

lake champlain

godind commented 1 month ago

Ok. So i do need to convert the NOAA s-57 charts.

I was thinking the NOAA mbtile files included raster images for each different zoom levels.

wdantuma commented 1 month ago

actually the converted s57 enc's (vector tiles) only show from zoom level 14 and above ( for performance reasons) so no gain there. Also as you can see there are some issues with the NOAA supplied enc's in combination with the rendering in freeboard-sk. I will try to give the s57-tiler and renderer in freeboard-sh some love in the comming weeks

godind commented 1 month ago

Great! I'm a bit lost in chart terminology and chart sourcing!

I have RNC charts but that's of no use.

wdantuma commented 1 month ago

I've made some fixes to s57-tiler and put up a "test" tile set of lake champlain at 

http://dobs.nl/s57/US4NY1NS/

Configure it like this in the signalk-charts plugin:

image

Can you also please try if the pre-build docker image works for you ( see the README ) ?

I've also made some fixes to freeboard-sk (in the https://github.com/wdantuma/freeboard-sk/tree/S57-fixes branch) but these haven't been merged yet

godind commented 1 month ago

I've ran the docker successfully. My enc folder contained a bunch of s-57 NOAA charts for lake Champlain and they all got generated.

image

I tested your chart and looks like it's working. Thank you!

image

Questions

  1. Do I need to wait for the Freeboard-SK PR to be committed?
  2. In the configuration, can I use a "Chart paths" instead of "Online chart providers" for my Charts? I don't see the same options for both types and wondering if it will work. I will need to use local charts without internet.
  3. Some charts have different scales. Do I need to do something different to generate those?

Thanks for the help!

godind commented 1 month ago

While looking at the cart details, I just noticed a few things in the converted versions:

1- Walls are not generate. Here are 2 examples but I can provide a few more:

image

image

image

image

2- Some rocks don't show

image

image

3- I can only see one depth lines (bottom typographical lines) in the display. Is this a setting in Freeboard I need to change to see all depth lines?

Thank you

wdantuma commented 1 month ago
  1. Do I need to wait for the Freeboard-SK PR to be committed?

Depends if you want to use a released version, you can also use the version with the fixes

  1. In the configuration, can I use a "Chart paths" instead of "Online chart providers" for my Charts? I don't see the same options for both types and wondering if it will work. I will need to use local charts without internet.

You can also use the charts path

  1. Some charts have different scales. Do I need to do something different to generate those?
  • US5VT1ED and US5VT1DD at 1:10,000
  • US2EC04M at 1:675,000
  • all others, such as the one you created are at 1:40,000

For now s57-tiler default generates only tiles for zoomlevel 14 ( 1:35000 ) but the SCAMIn and SCAMAX properties in the ENC are currently ignored so all features are rendered on all zoomlevels ( TODO )

1- Walls are not generate. Here are 2 examples but I can provide a few more: 2- Some rocks don't show

I will try to fix these in the comming weeks

3- I can only see one depth lines (bottom typographical lines) in the display. Is this a setting in Freeboard I need to change to see all depth lines?

For now you cant see all depth lines but those you see can be configured in freeboard-sk

wdantuma commented 1 month ago

Fixed the rendering of rocks and walls, shown here compared tot the rendering in openCPN with the same settings

Object Freeboard-sk openCPN
Rock image image
Wall image image
Wall image image
godind commented 1 month ago

Great! I'm re-rendering the charts with your fix.

Questions

  1. Back to a previous question: What fix is in the Freeboard-SK PR? I'm not sure if the fixes are in s-57 tiller (the charts) or in the FSK app (viewer).
  2. When rendering multiple ENC charts, should I do one chart at a time or can I drop all the extracted zip files to the enc folder? I'm asking because all zipped charts contain a files named CATALOG.031. They have the same name but different content. If I do more than one chart I have to overwrite that file each time.
wdantuma commented 1 month ago
  1. Back to a previous question: What fix is in the Freeboard-SK PR? I'm not sure if the fixes are in s-57 tiller (the charts) or in the FSK app (viewer).

The changes where made in freeboard-sk:S57-fixes, so not need to re convert but you need that version until merged in master

2. When rendering multiple ENC charts, should I do one chart at a time or can I drop all the extracted zip files to the enc folder? I'm asking because all zipped charts contain a files named CATALOG.031. They have the same name but different content. If I do more than one chart I have to overwrite that file each time.

They should not overwrite if every catalog.031 is in its own folder which should be the case if the folder containing the catalog031 is zipped

godind commented 1 month ago

Got it for the FSK update. Thanks

They should not overwrite if every catalog.031 is in its own folder which should be the case if the folder containing the catalog031 is zipped

This is what I get from the extracted zip: image

Should I drop all ENC_ROOT XXX folders directly in signal-charts/enc and convers all at once?

wdantuma commented 1 month ago

The structure should be:

 charts/
       US4NY1KS/
             ENC_ROOT/
                  CATALOG.031
                  ....
       US4NY1LS/
             ENC_ROOT/
                  CATALOG.031
                  ....
       US4NY1??/
             ENC_ROOT/
                  CATALOG.031
                  ....
godind commented 1 month ago

Ok. Here is my confusion. This was the enc folder setup and the charts output after I ran the docker. Is this right? I don't see US4NY1LS folder in the charts output: image

wdantuma commented 1 month ago

actually its my mistake, the input is correct but all input files are combined in one output set, tiles are not over written but features from multiple non overlapping sets are combined in one output tile set.

If you want a separate tileset per input ENC you have to call s57-tiler multiple times

so if you put the adjacent sets

US4NY1PS US4NY1PT US4NY1OS US4NY1OT US4NY1NS US4NY1NT US4NY1MS US4NY1LS US4NY1KS

in the enc folder one tileset is generated

overlapping sets

US5VT1ED and US5VT1DD can together be combined in another set

So they have to be combined per scale

sorry for the confusion

godind commented 1 month ago

Ok good. TO make sure I get it right. If I use the following charts subfolders in the enc input: US4NY1PS US4NY1PT US4NY1OS US4NY1OT US4NY1NS US4NY1NT US4NY1MS US4NY1LS US4NY1KS

It will use the first enc input chart folder name found, and combine all enc input charts in one single charts output folder, right?

Is there a naming convention to use for the combined output folder? Or can I rename to LAKE_CHAMPLAIN_14?

godind commented 1 month ago

I just tried the above and it appears to only process the first enc subfolder. I only have one output folder and it took a very short time compared to when I do charts one by one.

wdantuma commented 1 month ago

its time for weekend i gues

The NOAA charts are structured a bit different then my original test charts ( waddenzee the netherlands ) so for now the solution for you is to call s57-tiler once per NOAA chart

docker run -v  ./signalk-charts:/app/workdir wdantuma/s57-tiler  /app/s57-tiler --in workdir/enc/US4NY1PS --out workdir/charts
docker run -v  ./signalk-charts:/app/workdir wdantuma/s57-tiler  /app/s57-tiler --in workdir/enc/US4NY1PT --out workdir/charts
docker run -v  ./signalk-charts:/app/workdir wdantuma/s57-tiler  /app/s57-tiler --in workdir/enc/US4NY1OS --out workdir/charts
...

and put these folders as seprrate charts paths in the SK configuration

godind commented 1 month ago

I generated the charts individually and moved each folder to the SK server and created a Charts Plugin folder path for each. No charts found. There is the debug log. The full path looks right to me:

Jul 12 15:47:43 2024-07-12T19:47:43.946Z @signalk/charts-plugin ** loaded config: { chartPaths: [ './enc_charts/US4NY1KS', './enc_charts/US4NY1LS' ], onlineChartProviders: [] }
Jul 12 15:47:43 2024-07-12T19:47:43.947Z @signalk/charts-plugin Start charts plugin. Chart paths: /home/parallels/.signalk/enc_charts/US4NY1KS, /home/parallels/.signalk/enc_charts/US4NY1LS, online charts: 0
Jul 12 15:47:43 2024-07-12T19:47:43.948Z @signalk/charts-plugin **urlBase** http://localhost:3000
Jul 12 15:47:43 ::ffff:10.211.55.2 - - [12/Jul/2024:19:47:43 +0000] "POST /skServer/plugins/charts/config HTTP/1.1" 200 39 "http://10.211.55.4:3000/admin/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15"
Jul 12 15:47:43 2024-07-12T19:47:43.960Z @signalk/charts-plugin Chart plugin: Found 0 charts from /home/parallels/.signalk/enc_charts/US4NY1KS, /home/parallels/.signalk/enc_charts/US4NY1LS.
image

Each path folder is like this:

image
wdantuma commented 1 month ago

I've tested it with signalk-server and actually it is not needed tot add separate paths, just put the US4* folders in ( in you're case) the folder /home/parallels/.signalk/charts

so

/home/parallels/.signalk/charts/US4NY1KS /home/parallels/.signalk/charts/US4NY1LS etc.

in the log you get the message

Jul 13 04:55:56 2024-07-13T04:55:56.419Z @signalk/charts-plugin Chart plugin: Found 2 charts from /home/node/.signalk/charts.
godind commented 1 month ago

Fantastic! @signalk/charts-plugin Chart plugin: Found 9 charts from /home/parallels/.signalk/enc_charts.

I found one issue I think: there are gaps between the charts.

image

Looking at the chart files, I see that some charts have overlapping parts. Such as: US4NY1KS\14\4846/5970.pbf (folder contains 5970 to 5989) US4NY1LS\14\4846/5970.pbf (folder contains 5951 to 5970)

It appears the issue could be with either, the conversion of the charts when overlap is present, or the way they are overlapped in the viewer.

Thanks for all the help. I know this is a lot (I do KIP myself). Sorry.

wdantuma commented 1 month ago

My guess is that it is a view problem, but I shall investigate it somewhere in the coming days.

Are you using a released version of FSK or my branch ?

godind commented 1 month ago

I'm using the official version.

wdantuma commented 1 month ago

The issue with the gaps has been fixed now, for it all to work correctly you have to re convert the enc charts with the latest s57-tiler and use the fixes in FSK ( merge pending)

s57-tiler now default generates tile levels 9 to 14 and adds bounds to the metadata.json file

Also extra contours are rendered, this will be a option in FSK in the future

image

image

godind commented 1 month ago

Fantastic! Thank you so much. I'll reconvert the charts and let you know how it goes.

wdantuma commented 1 month ago

@godind i close this issue now please create a new issue for new requests / bugs

wdantuma commented 1 month ago

@godind FSK 2.10.0 with the s57 fixes has been released today

godind commented 1 month ago

Just updated. I can see charts now which is fantastic.