search

treee111 / wahooMapsCreator

Create maps for Wahoo device based on latest OSM maps
247 stars 25 forks source link

Output folder name is too long #227

Closed 7FM closed 9 months ago

7FM commented 10 months ago

Expected Behavior

Create an output folder and copy the map.lzma files to it.

Current Behavior

The output folder name gets too long and the program aborts. As far as I know the typical name length limit is 255 characters.

PS: please tell me there is a way to execute the remaining steps without going through the previous ones again :'D

Steps to Reproduce the Issue

If you have too much time and 540GB of free storage:

python -m wahoomc cli --country albania,alps,andorra,austria,azores,belarus,belgium,bosnia-herzegovina,britain-and-ireland,bulgaria,croatia,cyprus,czech-republic,dach,denmark,estonia,faroe-islands,finland,france,georgia,germany,great-britain,greece,guernsey-jersey,hungary,iceland,ireland-and-northern-ireland,isle-of-man,italy,kosovo,latvia,liechtenstein,lithuania,luxembourg,macedonia,malta,moldova,monaco,montenegro,netherlands,norway,poland,portugal,romania,serbia,slovakia,slovenia,spain,sweden,switzerland,turkey,ukraine --contour --cruiser

Context

Log Output / Stack Trace

INFO:--------------------------------------------------------------------------------
INFO:# Create: .map.lzma files
INFO:+ Country: albania_alps_andorra_austria_azores_belarus_belgium_bosnia-herzegovina_britain-and-ireland_bulgaria_croatia_cyprus_czech-republic_dach_denmark_estonia_faroe-islands_finland_france_georgia_germany_great-britain_greece_guernsey-jersey_hungary_iceland_ireland-and-northern-ireland_isle-of-man_italy_kosovo_latvia_liechtenstein_lithuania_luxembourg_macedonia_malta_moldova_monaco_montenegro_netherlands_norway_poland_portugal_romania_serbia_slovakia_slovenia_spain_sweden_switzerland_turkey_ukraine
INFO:+ Copying .map.lzma tiles to output folders
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "wahooMapsCreator/wahoomc/__main__.py", line 7, in <module>
    main.run("run")
  File "wahooMapsCreator/wahoomc/main.py", line 100, in run
    o_osm_maps.make_and_zip_files('.map.lzma', o_input_data.zip_folder)
  File "wahooMapsCreator/wahoomc/osm_maps_functions.py", line 855, in make_and_zip_files
    self.copy_to_dst(extension, src, dst)
  File "wahooMapsCreator/wahoomc/osm_maps_functions.py", line 901, in copy_to_dst
    os.makedirs(outdir, exist_ok=True)
  File "<frozen os>", line 215, in makedirs
  File "<frozen os>", line 225, in makedirs
OSError: [Errno 36] File name too long: 'wahooMapsCreatorData/albania_alps_andorra_austria_azores_belarus_belgium_bosnia-herzegovina_britain-and-ireland_bulgaria_croatia_cyprus_czech-republic_dach_denmark_estonia_faroe-islands_finland_france_georgia_germany_great-britain_greece_guernsey-jersey_hungary_iceland_ireland-and-northern-ireland_isle-of-man_italy_kosovo_latvia_liechtenstein_lithuania_luxembourg_macedonia_malta_moldova_monaco_montenegro_netherlands_norway_poland_portugal_romania_serbia_slovakia_slovenia_spain_sweden_switzerland_turkey_ukraine'
treee111 commented 10 months ago

Hi @7FM, thanks for using wahooMapsCreator and the interesting issue! I have not catched on that before. You are generating a lot of countries, should be enought for TCR ;-)

Ok, enough jokes.

I have created a fix that cuts down the foldername to around 50 characters. With that, there should be enough space for the folder name in addition to the directory in the full path (full path is sometimes also limited).

Already created files should be reused without problem. If you have enough space you could - to be safe - backup the intermediate files.

Steps I would advice you to do:

  1. backup the directory of the intermediate files: <user_dir>/wahooMapsCreatorData/_tiles (if enough space)
  2. enter the Anaconda environment
  3. install the alpha version into your Anaconda environment pip install wahoomc==4.1.1a1
  4. run your call identically as before: python -m wahoomc cli --country albania,alps,andorra,austria,azores,belarus,belgium,bosnia-herzegovina,britain-and-ireland,bulgaria,croatia,cyprus,czech-republic,dach,denmark,estonia,faroe-islands,finland,france,georgia,germany,great-britain,greece,guernsey-jersey,hungary,iceland,ireland-and-northern-ireland,isle-of-man,italy,kosovo,latvia,liechtenstein,lithuania,luxembourg,macedonia,malta,moldova,monaco,montenegro,netherlands,norway,poland,portugal,romania,serbia,slovakia,slovenia,spain,sweden,switzerland,turkey,ukraine --contour --cruiser

That should skip processing of intermediate files if they are there and config files have not been changed (tags-to-keep.json and tag-wahoo.xml and are not older than 24 days (if they are older you can give -md 100 for max. 100 days).

Happy to hear from you if that worked!

7FM commented 9 months ago

Thanks for the fast reply, sadly it seems to regenerate all files starting from Generate contour lines for each coordinate... so this might take a while xD

I probably should have started with a smaller set of countries to test this project^^

treee111 commented 9 months ago

you are welcome! With a smaller set of countries you might not have run into this issue ;-)

Oh shame on me! You are right - because we had to increate the version, the tool generates all new. I forgot that reason which also triggers the geneation from scratch. Because a version change generally has effect on content of the map I implemented that a while ago.

could you please upload the log, the first lines are interesting to validate if my assumption is correct.

Might have to think of only starting from scratch if minor or major version bumps... If I have to supply you with another version I will make sure to uncomment this section or handle it somehow! Again: Sorry! :-/

7FM commented 9 months ago

Oh shame on me! You are right - because we had to increate the version, the tool generates all new. I forgot that reason which also triggers the geneation from scratch. Because a version change generally has effect on content of the map I implemented that a while ago.

Actually, I just cherry-picked https://github.com/treee111/wahooMapsCreator/commit/7faa95df616b3ac1990db4e07e49150ca9372530 and https://github.com/treee111/wahooMapsCreator/commit/539894fdf9c5e4c8c557232519fefb08da0317c8. So there is no version bump involved in my case :)

Anyways, here are the first lines:

INFO:--------------------------------------------------------------------------------
INFO:--------------------------------------------------------------------------------
INFO:# check geofabrik.json file
INFO:# Input country: albania,alps,andorra,austria,azores,belarus,belgium,bosnia-herzegovina,britain-and-ireland,bulgaria,croatia,cyprus,czech-republic,dach,denmark,estonia,faroe-islands,finland,france,georgia,germany,great-britain,greece,guernsey-jersey,hungary,iceland,ireland-and-northern-ireland,isle-of-man,italy,kosovo,latvia,liechtenstein,lithuania,luxembourg,macedonia,malta,moldova,monaco,montenegro,netherlands,norway,poland,portugal,romania,serbia,slovakia,slovenia,spain,sweden,switzerland,turkey,ukraine.
INFO:(+ tile 50 of 64) Find needed countries
INFO:(+ tile 50 of 57) Find needed countries
INFO:(+ tile 50 of 225) Find needed countries
INFO:(+ tile 100 of 225) Find needed countries
INFO:(+ tile 150 of 225) Find needed countries
INFO:(+ tile 200 of 225) Find needed countries
INFO:(+ tile 50 of 109) Find needed countries
INFO:(+ tile 100 of 109) Find needed countries
INFO:(+ tile 50 of 191) Find needed countries
INFO:(+ tile 100 of 191) Find needed countries
INFO:(+ tile 150 of 191) Find needed countries
INFO:(+ tile 50 of 144) Find needed countries
INFO:(+ tile 100 of 144) Find needed countries
INFO:(+ tile 50 of 71) Find needed countries
INFO:(+ tile 50 of 209) Find needed countries
INFO:(+ tile 100 of 209) Find needed countries
INFO:(+ tile 150 of 209) Find needed countries
INFO:(+ tile 200 of 209) Find needed countries
INFO:(+ tile 50 of 64) Find needed countries
INFO:(+ tile 50 of 100) Find needed countries
INFO:(+ tile 50 of 81) Find needed countries
INFO:(+ tile 50 of 131) Find needed countries
INFO:(+ tile 100 of 131) Find needed countries
INFO:(+ tile 50 of 1891) Find needed countries
INFO:(+ tile 100 of 1891) Find needed countries
INFO:(+ tile 150 of 1891) Find needed countries
INFO:(+ tile 200 of 1891) Find needed countries
INFO:(+ tile 250 of 1891) Find needed countries
INFO:(+ tile 300 of 1891) Find needed countries
INFO:(+ tile 350 of 1891) Find needed countries
INFO:(+ tile 400 of 1891) Find needed countries
INFO:(+ tile 450 of 1891) Find needed countries
INFO:(+ tile 500 of 1891) Find needed countries
INFO:(+ tile 550 of 1891) Find needed countries
INFO:(+ tile 600 of 1891) Find needed countries
INFO:(+ tile 650 of 1891) Find needed countries
INFO:(+ tile 700 of 1891) Find needed countries
INFO:(+ tile 750 of 1891) Find needed countries
INFO:(+ tile 800 of 1891) Find needed countries
INFO:(+ tile 850 of 1891) Find needed countries
INFO:(+ tile 900 of 1891) Find needed countries
INFO:(+ tile 950 of 1891) Find needed countries
INFO:(+ tile 1000 of 1891) Find needed countries
INFO:(+ tile 1050 of 1891) Find needed countries
INFO:(+ tile 1100 of 1891) Find needed countries
INFO:(+ tile 1150 of 1891) Find needed countries
INFO:(+ tile 1200 of 1891) Find needed countries
INFO:(+ tile 1250 of 1891) Find needed countries
INFO:(+ tile 1300 of 1891) Find needed countries
INFO:(+ tile 1350 of 1891) Find needed countries
INFO:(+ tile 1400 of 1891) Find needed countries
INFO:(+ tile 1450 of 1891) Find needed countries
INFO:(+ tile 1500 of 1891) Find needed countries
INFO:(+ tile 1550 of 1891) Find needed countries
INFO:(+ tile 1600 of 1891) Find needed countries
INFO:(+ tile 1650 of 1891) Find needed countries
INFO:(+ tile 1700 of 1891) Find needed countries
INFO:(+ tile 1750 of 1891) Find needed countries
INFO:(+ tile 1800 of 1891) Find needed countries
INFO:(+ tile 1850 of 1891) Find needed countries
INFO:(+ tile 50 of 73) Find needed countries
INFO:(+ tile 50 of 191) Find needed countries
INFO:(+ tile 100 of 191) Find needed countries
INFO:(+ tile 150 of 191) Find needed countries
INFO:(+ tile 50 of 111) Find needed countries
INFO:(+ tile 100 of 111) Find needed countries
INFO:(+ tile 50 of 243) Find needed countries
INFO:(+ tile 100 of 243) Find needed countries
INFO:(+ tile 150 of 243) Find needed countries
INFO:(+ tile 200 of 243) Find needed countries
INFO:(+ tile 50 of 99) Find needed countries
INFO:(+ tile 50 of 141) Find needed countries
INFO:(+ tile 100 of 141) Find needed countries
INFO:--------------------------------------------------------------------------------
INFO:# Determine involved/border countries
INFO:+ Involved country: albania
INFO:+ Involved country: bosnia-herzegovina
INFO:+ Involved country: croatia
INFO:+ Involved country: montenegro
INFO:+ Involved country: italy
INFO:+ Involved country: greece
INFO:+ Involved country: kosovo
INFO:+ Involved country: macedonia
INFO:+ Involved country: serbia
INFO:+ Involved country: alps
INFO:+ Involved country: france
INFO:+ Involved country: spain
INFO:+ Involved country: switzerland
INFO:+ Involved country: germany
INFO:+ Involved country: austria
INFO:+ Involved country: czech-republic
INFO:+ Involved country: slovenia
INFO:+ Involved country: slovakia
INFO:+ Involved country: hungary
INFO:+ Involved country: andorra
INFO:+ Involved country: liechtenstein
INFO:+ Involved country: azores
INFO:+ Involved country: portugal
INFO:+ Involved country: belarus
INFO:+ Involved country: lithuania
INFO:+ Involved country: poland
INFO:+ Involved country: ukraine
INFO:+ Involved country: latvia
INFO:+ Involved country: northwestern-fed-district
INFO:+ Involved country: central-fed-district
INFO:+ Involved country: belgium
INFO:+ Involved country: great-britain
INFO:+ Involved country: netherlands
INFO:+ Involved country: luxembourg
INFO:+ Involved country: britain-and-ireland
INFO:+ Involved country: ireland-and-northern-ireland
INFO:+ Involved country: norway
INFO:+ Involved country: bulgaria
INFO:+ Involved country: romania
INFO:+ Involved country: turkey
INFO:+ Involved country: cyprus
INFO:+ Involved country: dach
INFO:+ Involved country: denmark
INFO:+ Involved country: sweden
INFO:+ Involved country: estonia
INFO:+ Involved country: finland
INFO:+ Involved country: faroe-islands
INFO:+ Involved country: guernsey-jersey
INFO:+ Involved country: georgia
INFO:+ Involved country: north-caucasus-fed-district
INFO:+ Involved country: south-fed-district
INFO:+ Involved country: armenia
INFO:+ Involved country: azerbaijan
INFO:+ Involved country: isle-of-man
INFO:+ Involved country: greenland
INFO:+ Involved country: iceland
INFO:+ Involved country: monaco
INFO:+ Involved country: algeria
INFO:+ Involved country: tunisia
INFO:+ Involved country: malta
INFO:+ Involved country: kaliningrad
INFO:+ Involved country: moldova
INFO:+ Involved country: morocco
INFO:+ Involved country: syria
INFO:+ Involved country: iraq
INFO:+ Involved country: iran
INFO:+ Involved country: crimean-fed-district
INFO:+ Border countries will be processed
INFO:--------------------------------------------------------------------------------
INFO:# check land_polygons.shp file
INFO:--------------------------------------------------------------------------------
INFO:# check countries .osm.pbf files
INFO:+ Checking for old maps and remove them
INFO:+ mapfile for albania: up-to-date.
INFO:+ mapfile for bosnia-herzegovina: up-to-date.
INFO:+ mapfile for croatia: up-to-date.
INFO:+ mapfile for montenegro: up-to-date.
INFO:+ mapfile for italy: up-to-date.
INFO:+ mapfile for greece: up-to-date.
INFO:+ mapfile for kosovo: up-to-date.
INFO:+ mapfile for macedonia: up-to-date.
INFO:+ mapfile for serbia: up-to-date.
INFO:+ mapfile for alps: up-to-date.
INFO:+ mapfile for france: up-to-date.
INFO:+ mapfile for spain: up-to-date.
INFO:+ mapfile for switzerland: up-to-date.
INFO:+ mapfile for germany: up-to-date.
INFO:+ mapfile for austria: up-to-date.
INFO:+ mapfile for czech-republic: up-to-date.
INFO:+ mapfile for slovenia: up-to-date.
INFO:+ mapfile for slovakia: up-to-date.
INFO:+ mapfile for hungary: up-to-date.
INFO:+ mapfile for andorra: up-to-date.
INFO:+ mapfile for liechtenstein: up-to-date.
INFO:+ mapfile for azores: up-to-date.
INFO:+ mapfile for portugal: up-to-date.
INFO:+ mapfile for belarus: up-to-date.
INFO:+ mapfile for lithuania: up-to-date.
INFO:+ mapfile for poland: up-to-date.
INFO:+ mapfile for ukraine: up-to-date.
INFO:+ mapfile for latvia: up-to-date.
INFO:+ mapfile for northwestern-fed-district: up-to-date.
INFO:+ mapfile for central-fed-district: up-to-date.
INFO:+ mapfile for belgium: up-to-date.
INFO:+ mapfile for great-britain: up-to-date.
INFO:+ mapfile for netherlands: up-to-date.
INFO:+ mapfile for luxembourg: up-to-date.
INFO:+ mapfile for britain-and-ireland: up-to-date.
INFO:+ mapfile for ireland-and-northern-ireland: up-to-date.
INFO:+ mapfile for norway: up-to-date.
INFO:+ mapfile for bulgaria: up-to-date.
INFO:+ mapfile for romania: up-to-date.
INFO:+ mapfile for turkey: up-to-date.
INFO:+ mapfile for cyprus: up-to-date.
INFO:+ mapfile for dach: up-to-date.
INFO:+ mapfile for denmark: up-to-date.
INFO:+ mapfile for sweden: up-to-date.
INFO:+ mapfile for estonia: up-to-date.
INFO:+ mapfile for finland: up-to-date.
INFO:+ mapfile for faroe-islands: up-to-date.
INFO:+ mapfile for guernsey-jersey: up-to-date.
INFO:+ mapfile for georgia: up-to-date.
INFO:+ mapfile for north-caucasus-fed-district: up-to-date.
INFO:+ mapfile for south-fed-district: up-to-date.
INFO:+ mapfile for armenia: up-to-date.
INFO:+ mapfile for azerbaijan: up-to-date.
INFO:+ mapfile for isle-of-man: up-to-date.
INFO:+ mapfile for greenland: up-to-date.
INFO:+ mapfile for iceland: up-to-date.
INFO:+ mapfile for monaco: up-to-date.
INFO:+ mapfile for algeria: up-to-date.
INFO:+ mapfile for tunisia: up-to-date.
INFO:+ mapfile for malta: up-to-date.
INFO:+ mapfile for kaliningrad: up-to-date.
INFO:+ mapfile for moldova: up-to-date.
INFO:+ mapfile for morocco: up-to-date.
INFO:+ mapfile for syria: up-to-date.
INFO:+ mapfile for iraq: up-to-date.
INFO:+ mapfile for iran: up-to-date.
INFO:+ mapfile for crimean-fed-district: up-to-date.
INFO:--------------------------------------------------------------------------------
INFO:# Filter tags from country osm.pbf files
INFO:+ Filter tags from country osm.pbf files: OK
INFO:--------------------------------------------------------------------------------
INFO:# Generate land for each coordinate
INFO:+ Generate land for each coordinate: OK
INFO:--------------------------------------------------------------------------------
INFO:# Generate sea for each coordinate
INFO:+ Generate sea for each coordinate: OK
INFO:--------------------------------------------------------------------------------
INFO:# Generate contour lines for each coordinate
INFO:+ Coordinates: 141,94. (1 of 3155)
INFO:+ Coordinates: 141,95. (2 of 3155)
INFO:+ Coordinates: 141,96. (3 of 3155)
INFO:+ Coordinates: 142,94. (4 of 3155)
INFO:+ Coordinates: 142,95. (5 of 3155)
INFO:+ Coordinates: 142,96. (6 of 3155)
INFO:+ Coordinates: 142,97. (7 of 3155)
INFO:+ Coordinates: 105,96. (65 of 3155)
INFO:+ Coordinates: 105,97. (66 of 3155)
INFO:+ Coordinates: 105,98. (67 of 3155)
INFO:+ Coordinates: 106,96. (68 of 3155)
INFO:+ Coordinates: 106,97. (69 of 3155)
INFO:+ Coordinates: 106,98. (70 of 3155)
INFO:+ Coordinates: 106,99. (71 of 3155)
INFO:+ Coordinates: 107,96. (72 of 3155)
INFO:+ Coordinates: 107,97. (73 of 3155)
INFO:+ Coordinates: 107,98. (74 of 3155)
INFO:+ Coordinates: 107,99. (75 of 3155)
INFO:+ Coordinates: 108,96. (76 of 3155)
INFO:+ Coordinates: 108,97. (77 of 3155)
INFO:+ Coordinates: 108,98. (78 of 3155)
INFO:+ Coordinates: 108,99. (79 of 3155)
INFO:+ Coordinates: 109,96. (80 of 3155)
INFO:+ Coordinates: 109,97. (81 of 3155)
INFO:+ Coordinates: 109,98. (82 of 3155)
INFO:+ Coordinates: 109,99. (83 of 3155)
INFO:+ Coordinates: 109,100. (84 of 3155)
7FM commented 9 months ago

I could abort the current run and restart with some debugging options if that helps narrowing it down.

treee111 commented 9 months ago

nice, thanks for the fast response!

Good, that you cherry picked, the output is also not telling something about that there is a version bump and thus files are recreated.

And there is also no processing of all parts other than contour lines. As contour lines are the latest enhancement and I don't use them regulary the intermediate files might not be recognized correctly.

As you can see in your output, only contour lines are processed per tile

INFO:--------------------------------------------------------------------------------
INFO:# Filter tags from country osm.pbf files
INFO:+ Filter tags from country osm.pbf files: OK
INFO:--------------------------------------------------------------------------------
INFO:# Generate land for each coordinate
INFO:+ Generate land for each coordinate: OK
INFO:--------------------------------------------------------------------------------
INFO:# Generate sea for each coordinate
INFO:+ Generate sea for each coordinate: OK
INFO:--------------------------------------------------------------------------------
INFO:# Generate contour lines for each coordinate
INFO:+ Coordinates: 141,94. (1 of 3155)
INFO:+ Coordinates: 141,95. (2 of 3155)
INFO:+ Coordinates: 141,96. (3 of 3155)

If you run wahooMapsCreator for malta and it creates all intermediate files, it looks like that, so country files and tiles are (re)processed

INFO:--------------------------------------------------------------------------------
INFO:# Filter tags from country osm.pbf files
INFO:+ Filtering unwanted map objects out of map of malta
INFO:+ Filter tags from country osm.pbf files: OK
INFO:--------------------------------------------------------------------------------
INFO:# Generate land for each coordinate
INFO:+ (tile 1 of 2) Coordinates: 137,100
INFO:+ (tile 2 of 2) Coordinates: 138,100
INFO:+ Generate land for each coordinate: OK
INFO:--------------------------------------------------------------------------------
INFO:# Generate sea for each coordinate
INFO:+ (tile 1 of 2) Coordinates: 137,100
INFO:+ (tile 2 of 2) Coordinates: 138,100
INFO:+ Generate sea for each coordinate: OK
INFO:--------------------------------------------------------------------------------
INFO:# Generate contour lines for each coordinate
INFO:+ Coordinates: 137,100. (1 of 2)
INFO:+ Coordinates: 138,100. (2 of 2)
INFO:+ Generate contour lines for each coordinate: OK
INFO:--------------------------------------------------------------------------------
7FM commented 9 months ago

I can confirm that it completed successfully with the commits in https://github.com/treee111/wahooMapsCreator/pull/228 Thanks!

treee111 commented 9 months ago

I can confirm that it completed successfully with the commits in #228 Thanks!

thanks for confirming! 👍 I released v4.1.1 which you can install. I checked re-processing of the intermediate files. There is no reason of re-processing them if only the version bumps. Just when config changes (tags-to-keep etc.)