Timeout

[dyk74]

Hello, I've a problem generating the MOV file: my GPX file is quite big... 26170 Points and when I try to create it after 3-4 hours I receive it back TIMEOUT ERROR and the tilecache.ndbm.db is very big (2-3 GB). This is my timeout error:

Traceback (most recent call last): File "/home/tarcisio/venv/bin/gopro-dashboard.py", line 310, in draw_timer.time(lambda: buffer.draw(lambda frame: overlay.draw(dt, frame))) File "/home/tarcisio/venv/lib/python3.11/site-packages/gopro_overlay/timing.py", line 20, in time r = f() ^^^ File "/home/tarcisio/venv/bin/gopro-dashboard.py", line 310, in draw_timer.time(lambda: buffer.draw(lambda frame: overlay.draw(dt, frame))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tarcisio/venv/lib/python3.11/site-packages/gopro_overlay/buffering.py", line 27, in draw f(image) File "/home/tarcisio/venv/bin/gopro-dashboard.py", line 310, in draw_timer.time(lambda: buffer.draw(lambda frame: overlay.draw(dt, frame))) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tarcisio/venv/lib/python3.11/site-packages/gopro_overlay/layout.py", line 105, in draw return self.scene.draw(image) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/tarcisio/venv/lib/python3.11/site-packages/gopro_overlay/widgets/widgets.py", line 249, in draw w.draw(image, draw) File "/home/tarcisio/venv/lib/python3.11/site-packages/gopro_overlay/widgets/widgets.py", line 32, in draw w.draw(image, draw) File "/home/tarcisio/venv/lib/python3.11/site-packages/gopro_overlay/widgets/widgets.py", line 150, in draw self.widget.draw(ivp, dvp) File "/home/tarcisio/venv/lib/python3.11/site-packages/gopro_overlay/widgets/widgets.py", line 211, in draw self.child.draw(rect, rect_draw) File "/home/tarcisio/venv/lib/python3.11/site-packages/gopro_overlay/widgets/widgets.py", line 32, in draw w.draw(image, draw) File "/home/tarcisio/venv/lib/python3.11/site-packages/gopro_overlay/widgets/map.py", line 256, in draw self.cached_map, self.cached_map_image = self._redraw() ^^^^^^^^^^^^^^ File "/home/tarcisio/venv/lib/python3.11/site-packages/gopro_overlay/widgets/map.py", line 240, in _redraw map_image = self.renderer(map) ^^^^^^^^^^^^^^^^^^ File "/home/tarcisio/venv/lib/python3.11/site-packages/gopro_overlay/geo.py", line 132, in render return my_render_map(map, tiles, downloader=dbm_downloader(dbm_file), **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tarcisio/venv/lib/python3.11/site-packages/gopro_overlay/geo_render.py", line 87, in my_render_map tiles = cache.populate(downloader, tiles, _error_image(provider.tile_width, provider.tile_height)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tarcisio/venv/lib/python3.11/site-packages/gopro_overlay/geo_render.py", line 52, in populate downloaded = self.do_download(downloader, have_not) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tarcisio/venv/lib/python3.11/site-packages/gopro_overlay/geo_render.py", line 32, in do_download return loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "/home/tarcisio/venv/lib/python3.11/site-packages/gopro_overlay/geo_render.py", line 25, in do_async_download async for g in gen: File "/home/tarcisio/venv/lib/python3.11/site-packages/geotiler/cache.py", line 87, in caching_downloader async for t in result: File "/home/tarcisio/venv/lib/python3.11/site-packages/geotiler/tile/io.py", line 98, in fetch_tiles tile = await task # no exception expected at this stage ^^^^^^^^^^ File "/usr/lib/python3.11/asyncio/tasks.py", line 605, in _wait_for_one return f.result() # May raise f.exception(). ^^^^^^^^^^ File "/home/tarcisio/venv/lib/python3.11/site-packages/geotiler/tile/io.py", line 63, in fetch_tile async with session.get(tile.url) as response: File "/home/tarcisio/venv/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in aenter self._resp = await self._coro ^^^^^^^^^^^^^^^^ File "/home/tarcisio/venv/lib/python3.11/site-packages/aiohttp/client.py", line 560, in _request await resp.start(conn) File "/home/tarcisio/venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 894, in start with self._timer: File "/home/tarcisio/venv/lib/python3.11/site-packages/aiohttp/helpers.py", line 721, in exit raise asyncio.TimeoutError from None

is there anyway to have a verbose mode BEFORE the FFMPEG procedure?... or why is it so slow? (how much time is needed?) Thank you

[time4tea]

Hi - thanks for your interest in the project! I've not seen this error before. That must be a bit annoying. It looks like there was a timeout while downloading a map tile. Map tiles are only downloaded on-demand, for most use cases, but are pretty small so this isn't usually a problem. What is your internet connection like? What map provider were you using? Is it possible that you were rate-limited? Where does your GPS trace go from and to? This is a pretty big ndbm file! Way bigger that I'd expect to be honest. What zoom level is your map set to? if it is very zoomed in, then the map rendering will need many tiles to render.

You could try to:

• View a trace of the journey in google.maps. You can do this, by running venv/bin/gopro-to-gpx --every 1 <inputfile> <outputfile> . then go to https://www.google.com/maps/d/u/0/ - "create map", "import" - and see how far the trace goes... does it travel a long distance?

  How long is your video (in minutes & GB)? - Without any performance tricks (see PERFORMANCE_GUIDE.MD it will generally render at 1.5 real-time speed. So a 1h movie will take 40 mins to render.

[dyk74]

If you want I can send you my gpx file and the command I'm using: The trip (is a test for a future long one) is 30 min long and the gpx file is 400 kB With small gpx file is working like a charm (30kB files). I've tried with different internet connection different os (ubuntu Linux and windows docker)

⁣Ottieni BlueMail per Android ​

Il giorno 29 Ago 2023, 17:18, alle ore 17:18, James Richardson @.***> ha scritto:

Hi - thanks for your interest in the project! I've not seen this error before. That must be a bit annoying. It looks like there was a timeout while downloading a map tile. Map tiles are only downloaded on-demand, for most use cases, but are pretty small so this isn't usually a problem. What is your internet connection like? What map provider were you using? Is it possible that you were rate-limited? Where does your GPS trace go from and to? This is a pretty big ndbm file! Way bigger that I'd expect to be honest. What zoom level is your map set to? if it is very zoomed in, then the map rendering will need many tiles to render.

You could try to:

• View a trace of the journey in google.maps. You can do this, by running venv/bin/gopro-to-gpx --every 1 <inputfile> <outputfile> . then go to https://www.google.com/maps/d/u/0/ - "create map", "import"
• and see how far the trace goes... does it travel a long distance?

How long is your video (in minutes & GB)? - Without any performance tricks (see PERFORMANCE_GUIDE.MD it will generally render at 1.5 real-time speed. So a 1h movie will take 40 mins to render.

-- Reply to this email directly or view it on GitHub: https://github.com/time4tea/gopro-dashboard-overlay/issues/149#issuecomment-1697651436 You are receiving this because you authored the thread.

Message ID: @.***>

[time4tea]

If you could sent it, that would be really useful. Please send to @.*** - and also please let me know the command line you used, and the version information printed when the program starts.

Thanks!

James

On 29 Aug 2023, 16:43, at 16:43, dyk74 @.***> wrote:

If you want I can send you my gpx file and the command I'm using: The trip (is a test for a future long one) is 30 min long and the gpx file is 400 kB With small gpx file is working like a charm (30kB files). I've tried with different internet connection different os (ubuntu Linux and windows docker)

⁣Ottieni BlueMail per Android ​

Il giorno 29 Ago 2023, 17:18, alle ore 17:18, James Richardson @.***> ha scritto:

Hi - thanks for your interest in the project! I've not seen this error before. That must be a bit annoying. It looks like there was a timeout while downloading a map tile. Map tiles are only downloaded on-demand, for most use cases, but are pretty small so this isn't usually a problem. What is your internet connection like? What map provider were you using? Is it possible that you were rate-limited? Where does your GPS trace go from and to? This is a pretty big ndbm file! Way bigger that I'd expect to be honest. What zoom level is your map set to? if it is very zoomed in, then the map rendering will need many tiles to render.

You could try to:

• View a trace of the journey in google.maps. You can do this, by running venv/bin/gopro-to-gpx --every 1 <inputfile> <outputfile> . then go to https://www.google.com/maps/d/u/0/ - "create map", "import"
• and see how far the trace goes... does it travel a long distance?

How long is your video (in minutes & GB)? - Without any performance tricks (see PERFORMANCE_GUIDE.MD it will generally render at 1.5 real-time speed. So a 1h movie will take 40 mins to render.

-- Reply to this email directly or view it on GitHub: https://github.com/time4tea/gopro-dashboard-overlay/issues/149#issuecomment-1697651436 You are receiving this because you authored the thread.

Message ID: @.***>

-- Reply to this email directly or view it on GitHub: https://github.com/time4tea/gopro-dashboard-overlay/issues/149#issuecomment-1697703704 You are receiving this because you commented.

Message ID: @.***>

[time4tea]

Ah, github mail redacted the email address! It's gopro-overlay@time4tea.net

[time4tea]

Thanks for sending me the information. The problem is in your GPX track. Mostly your trip was in Switzerland, but one of the entries: 2023xxxx_160027_N, is an entire track of 45 seconds, located in Taiwan. As you are using the "moving journey map", which is rendered up front for the bounding box of your trip it will try to render the whole map at zoom level 13 (from your layout file) - this means an enormous map! You can see this from the output: MovingJourneyMap Rendering backing map ((654959, 163547)) However its not very obvious from this output that this might be a problem - however to render a map this size, the software will need to download 6.5 million tiles, and you probably got rate limited.

Removing the dodgy track from your gpx file, gives a more normal size for the backing map: MovingJourneyMap Rendering backing map ((857, 966)) Which completes in a fraction of a second.

I don't think this is a defect as such, but I'll think of a way of informing the user that the bounding box of the trip is suspiciously large.

[dyk74]

Thanks a lot about the solution, how can I find (in the future) this issue?

⁣Ottieni BlueMail per Android ​

Il giorno 2 Set 2023, 15:09, alle ore 15:09, James Richardson @.***> ha scritto:

Thanks for sending me the information. The problem is in your GPX track. Mostly your trip was in Switzerland, but one of the entries: 2023xxxx_160027_N, is an entire track of 45 seconds, located in Taiwan. As you are using the "moving journey map", which is rendered up front for the bounding box of your trip it will try to render the whole map at zoom level 13 (from your layout file) - this means an enormous map! You can see this from the output: MovingJourneyMap Rendering backing map ((654959, 163547)) However its not very obvious from this output that this might be a problem - however to render a map this size, the software will need to download 6.5 million tiles, and you probably got rate limited.

Removing the dodgy track from your gpx file, gives a more normal size for the backing map: MovingJourneyMap Rendering backing map ((857, 966)) Which completes in a fraction of a second.

I don't think this is a defect as such, but I'll think of a way of informing the user that the bounding box of the trip is suspiciously large.

-- Reply to this email directly or view it on GitHub: https://github.com/time4tea/gopro-dashboard-overlay/issues/149#issuecomment-1703829686 You are receiving this because you authored the thread.

Message ID: @.***>

[time4tea]

I've not seen this before. I'm not sure how you generated the GPX file? It seems to have come from a website, not a device. In general the "setup" part of the program only takes a few seconds, and things that take a long time (like the rendering) have a progress bar.

[jas0nc]

Hi, I have a similar issue. I have just ridden the Paris-Brest-Paris 1200km event by cycling. It is around 90 hours of fit file in total. since it is crossing a wide range of maps too, I got time-out issue as this topic owner. Would you happen to have any suggestion on how to get an overlay from a fit file with a moving map across 1200km?

[time4tea]

One easy thing to do would be to change the zoom level - changing it to a smaller number will require many fewer tiles. You might have to experiment a bit to get the best number - you don't need to wait for the whole movie to render, just a few seconds to see if its zoomed in "enough" for you. The tiles are cached, so you'll only have to download them once. I suggest upgrading to the latest version too, as the caching has changed a bit (although it won't make much speed difference, the format has been changed, so if you're going to download many tiles upgrade first)

[time4tea]

@jas0nc - did the zoom make a difference? What are you expecting to do with a 90h movie? What features or capabilities would be useful for a ride of that length?

[jas0nc]

@time4tea Thanks for your reply. The video I made is to record the whole journey with immersive feeling. so ppl can use it as training video indoor while knowing more about the route. I think changing zoom level is not a good approach for me. as I want viewer can see where it is. I wonder, if it is possible I generate the overlay by parts while keeping the total distance accumulate to 1200km? if so, I think I can keep all the feature I need, while keeping the cache mimimal.

[time4tea]

OK. Let's see if we can find a solution. Would you be willing to share your gpx file with me? I think in this way I could understand a bit more about the file sizes and timings to create a video. If not, of course absolutely no problem. If you are OK to share, please could you send to gopro-overlay@time4tea.net and I'll take a look at what we might do to create this 90h video.... Thanks! James

[time4tea]

@jas0nc - thanks for the file! Its pretty big! - I'm not quite sure how to make the software quick enough to process... its about 100x larger than most. What is the bit thats failing for you, or is it just that its very slow?

[jas0nc]

Hi James,

I still have no success with this fit file. I am not sure the timeout is from OSM, or I am just running out of ram (I have only 8GB on my M1 MacBook Air). Starting gopro-dashboard version 0.102.0 ffmpeg version is 6.0 Using Python version 3.11.5 (main, Aug 24 2023, 15:09:45) [Clang 14.0.3 (clang-1403.0.22.14.1)] GPX/FIT file: 2023-08-20T18:13:04+00:00 -> 2023-08-24T01:01:51+00:00 Timer(loading timeseries - Called: 1, Total: 343.20675, Avg: 343.20675, Rate: 0.00) Generating overlay at Dimension(x=1920, y=1080) Timeseries has 2837271 data points Processing.... Timer(processing - Called: 1, Total: 589.05235, Avg: 589.05235, Rate: 0.00) FFMPEG Output is in /var/folders/c5/2jycsshj3390g6rjh88c8_ym0000gn/T/tmptumm44ou Timelapse Factor = 1.000 Layout -> Include component 'power-box' = True Layout -> Include component 'power' = True Layout -> Include component 'cadence' = True Layout -> Include component 'heartbeat' = True Layout -> Include component 'top-box' = True Layout -> Include component 'speed' = True Layout -> Include component 'altitude' = True Layout -> Include component 'odo' = True Layout -> Include component 'temperature' = True Layout -> Include component 'date_and_time' = True Layout -> Include component 'channel-name' = True Layout -> Include component 'map-box' = True Layout -> Include component 'moving_map' = True Layout -> Include component 'gradient' = True Executing 'ffmpeg -hide_banner -y -hide_banner -loglevel info -f rawvideo -framerate 10.0 -s 1920x1080 -pix_fmt rgba -i - -r 30 -vcodec hevc_videotoolbox -q:v 65 -alpha_quality 0.75 -vtag hvc1 /Volumes/VI550_1TB/Overlay/pbp-overlay-NorthUP.mov' Render: 0 [ 0%] [ 0.0/s] | | ETA: --:--:--MovingJourneyMap Rendering backing map ((294284, 47760)) (can be slow) /Users/jason/Documents/gopro-dashboard-overlay/FIT2Overlay-NorthUP.sh: line 7: 3057 Killed: 9 venv/bin/gopro-dashboard.py --units-speed kph --units-distance km --profile overlay-mac-hevc-alpha --overlay-size 1920x1080 --bg 0,0,0,0 --layout-xml config/layout-NorthUP.xml --use-fit-only --fit $input /Volumes/VI550_1TB/Overlay/$filename-overlay-NorthUP.mov

Saving session... ...copying shared history... ...saving history...truncating history files... ...completed. Deleting expired sessions... 12 completed.

[Process completed] Do you know what option I can use if I want to make the overlay by parts? Ie. 1 hour long for each. Then I can have 90hours in 90 clips.

I found these 2 options, but not sure how to use it to make it generate overlay for only certain part of the fit file. [--video-time-start {file-created,file-modified,file-accessed}] [--video-time-end {file-created,file-modified,file-accessed}]

For your reference, here is the video https://youtu.be/8A_s5o5ZFWY?feature=shared&t=11552 I made with ultra-long-distance overlay. Hope you can understand what I am looking for.

And this is the ride I am working with now: https://www.strava.com/activities/9717601745

Thanks for your help again. Jason.

On 4 Sep 2023, at 4:34 AM, James Richardson @.***> wrote:

@jas0nc https://github.com/jas0nc - thanks for the file! Its pretty big! - I'm not quite sure how to make the software quick enough to process... its about 100x larger than most. What is the bit thats failing for you, or is it just that its very slow?

— Reply to this email directly, view it on GitHub https://github.com/time4tea/gopro-dashboard-overlay/issues/149#issuecomment-1704395446, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIBF5KHVETK2Q6PAEWY5KJTXYTSVJANCNFSM6AAAAAA4DCYXW4. You are receiving this because you were mentioned.

[jas0nc]

[Updates] I found out my issue was due to tilecache.ndbm.db occupying 137GB on my 256GB MacBook. I deleted it and mounted a cache directory to an external disk. now I can finally pass the "Rendering backing map" stage and reach the "render" stage. ie. Render: 10322 [ 0%] [ 24.7/s] I ETA: 1 day, 7:48:161 I will need to wait one more day till it is completed. but I believe this is my issue ansd it is solved now. Not sure if anyone will have the same issue as I am, but you may look into the /.gopro-graphics folder (default cache location) to see if you have big files occupying the space. 20230907: The rendering is completed now. i have a 60GB 1080p overlay. Everything is fine. :) Thanks.