search

mapillary / mapillary_tools

Command line tools for processing and uploading Mapillary imagery
BSD 2-Clause "Simplified" License
264 stars 137 forks source link

Version 0.10.0 fails to process large video files #600

Closed stefanbj closed 1 year ago

stefanbj commented 1 year ago

Basic information

Steps to reproduce behavior

  1. Process video: python3.exe -m mapillary_tools.commands video_process_and_upload --user_name bruno360 --overwrite_all_EXIF_tags --interpolate_directions --geotag_source camm R0011493.MP4 R0011493
  2. The ffmpeg image sample extraction fails, likley due to a commandline that way too long.

Expected behavior

6481 video samples uploaded ...

Actual behavior

No images upploaded.

ERROR - MapillaryFFmpegNotFoundError: The ffmpeg command "ffmpeg" not found

Corresponding data

...

Additional information

stefanb@Harrington /cygdrive/d/Bilder/2023-02-26
$  for i in R0011493.MP4; do python3.exe -m mapillary_tools.commands video_process_and_upload --user_name bruno360 --overwrite_all_EXIF_tags --interpolate_directions  --geotag_source camm ./$i ./$(basename $i .MP4) ; done ; exiftool -p 'C:\Program Files\exiftool\gpx.fmt' -api largefilesupport=1 -ee -ext MP4 -w %f/%f.gpx  *.MP4;  echo -e \\a\\a\\a\\a
2023-02-26 19:52:41,200 - INFO    - Extracting video information: ffprobe -print_format json -hide_banner -show_format -show_streams R0011493.MP4
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000242171f6680] UDTA parsing failed retrying raw
    Last message repeated 1 times
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000242171f6680] DTS discontinuity in stream 2: packet 226 with DTS 555979021, packet 227 with DTS 560942136
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'R0011493.MP4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2023-02-26T16:03:27.000000Z
    com.apple.quicktime.location.ISO6709: +58.9959+016.8167+0024/
    com.apple.quicktime.make: RICOH
    com.apple.quicktime.model: RICOH THETA X
    make            : RICOH
    model           : RICOH THETA X
  Duration: 00:25:01.62, start: 0.000000, bitrate: 120490 kb/s
  Stream #0:0[0x1](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, mono, fltp, 256 kb/s (default)
    Metadata:
      creation_time   : 2023-02-26T16:03:27.000000Z
      handler_name    : SoundHandle
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](eng): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuvj420p(pc, bt470bg/bt709/smpte170m, progressive), 7680x3840, 120058 kb/s, SAR 1:1 DAR 2:1, 10 fps, 10 tbr, 90k tbn (default)
    Metadata:
      creation_time   : 2023-02-26T16:03:27.000000Z
      handler_name    : VideoHandle
      vendor_id       : [0][0][0][0]
    Side data:
      spherical: equirectangular (0.000000/0.000000/0.000000)
  Stream #0:2[0x3](und): Data: none (camm / 0x6D6D6163), 575591 kb/s (default)
    Metadata:
      creation_time   : 2023-02-26T16:03:27.000000Z
      handler_name    : CameraMetadataMotionHandler
Unsupported codec with id 0 for input stream 2
2023-02-26 19:52:41,507 - INFO    - Extracting video metdata
2023-02-26 19:53:10,639 - INFO    - Found total 1496 GPS points
2023-02-26 19:53:10,639 - INFO    - Extracting video samples
2023-02-26 19:53:10,711 - INFO    - Extracting video samples
2023-02-26 19:53:10,855 - INFO    - Found total 15011 video samples
2023-02-26 19:53:10,856 - INFO    - Interpolating video samples in the time range from 0.555979021 to 1500.720445844
2023-02-26 19:53:11,205 - INFO    - Found total 15002 interpolated video samples
2023-02-26 19:53:11,265 - INFO    - Selected 6481 video samples by the minimal sample distance 3.0
2023-02-26 19:53:11,273 - INFO    - Extracting frames: ffmpeg -hide_banner -nostdin -i R0011493.MP4 -map 0:1 -vf select=eq(n\,6)+eq(n\,43)+eq(n\,55)+eq(n\,70)+eq(n\,88)+eq(n\,97)+eq(n\,104)+eq(n\,110)+eq(n\,116)+
eq(n\,122)+eq(n\,126)+eq(n\,130)+eq(n\,134)+eq(n\,137)+eq(n\,140)+eq(n\,143)+eq(n\,146)+eq(n\,149)+eq(n\,152)+eq(n\,155)+eq(n\,158)+eq(n\,161)+eq(n\,164)+eq(n\,167)+eq(n\,170)+eq(n\,173)+eq(n\,176)+eq(n\,179)+eq(
n\,182)+eq(n\,185)+eq(n\,188)+eq(n\,191)+eq(n\,194)+eq(n\,197)+eq(n\,199)+eq(n\,201)+eq(n\,203)+eq(n\,205)+eq(n\,207)+eq(n\,209)+eq(n\,211)+eq(n\,213)+eq(n\,215)+eq(n\,217)+eq(n\,219)+eq(n\,221)+eq(n\,223)+eq(n\,
225)+eq(n\,227)+eq(n\,229)+eq(n\,231)+eq(n\,233)+eq(n\,235)+eq(n\,237)+eq(n\,239)+eq(n\,241)+eq(n\,243)+eq(n\,245)+eq(n\,247)+eq(n\,249)+eq(n\,251)+eq(n\,253)+eq(n\,255)+eq(n\,257)+eq(n\,259)+eq(n\,261)+eq(n\,263
)+eq(n\,265)+eq(n\,267)+eq(n\,269)+eq(n\,271)+eq(n\,273)+eq(n\,276)+eq(n\,279)+eq(n\,282)+eq(n\,285)+eq(n\,288)+eq(n\,291)+eq(n\,294)+eq(n\,297)+eq(n\,300)+eq(n\,303)+eq(n\,307)+eq(n\,311)+eq(n\,315)+eq(n\,318)+e
q(n\,321)+eq(n\,324)+eq(n\,327)+eq(n\,330)+eq(n\,333)+eq(n\,336)+eq(n\,338)+eq(n\,340)+eq(n\,342)+eq(n\,345)+eq(n\,348)+eq(n\,351)+eq(n\,354)+eq(n\,357)+eq(n\,360)+eq(n\,363)+eq(n\,366)+eq(n\,369)+eq(n\,372)+eq(n
\,375)+eq(n\,377)+eq(n\,379)+eq(n\,381)+eq(n\,383)+eq(n\,386)+eq(n\,389)+eq(n\,392)+eq(n\,395)+eq(n\,397)+eq(n\,399)+eq(n\,401)+eq(n\,403)+eq(n\,405)+eq(n\,407)+eq(n\,409)+eq(n\,411)+eq(n\,413)+eq(n\,415)+eq(n\,4
17)+eq(n\,419)+eq(n\,421)+eq(n\,424)+eq(n\,427)+eq(n\,430)+eq(n\,433)+eq(n\,435)+eq(n\,437)+eq(n\,439)+eq(n\,441)+eq(n\,443)+eq(n\,445)+eq(n\,447)+eq(n\,449)+eq(n\,451)+eq(n\,453)+eq(n\,455)+eq(n\,457)+eq(n\,459)
+eq(n\,461)+eq(n\,463)+eq(n\,465)+eq(n\,467)+eq(n\,469)+eq(n\,471)+eq(n\,473)+eq(n\,475)+eq(n\,477)+eq(n\,479)+eq(n\,481)+eq(n\,483)+eq(n\,485)+eq(n\,487)+eq(n\,489)+eq(n\,491)+eq(n\,493)+eq(n\,496)+eq(n\,499)+eq
(n\,502)+eq(n\,505)+eq(n\,507)+eq(n\,509)+eq(n\,511)+eq(n\,513)+eq(n\,515)+eq(n\,517)+eq(n\,519)+eq(n\,521)+eq(n\,523)+eq(n\,525)+eq(n\,527)+eq(n\,529)+eq(n\,531)+eq(n\,534)+eq(n\,537)+eq(n\,540)+eq(n\,543)+eq(n\
,546)+eq(n\,549)+eq(n\,552)+eq(n\,555)+eq(n\,558)+eq(n\,561)+eq(n\,565)+eq(n\,569)+eq(n\,573)+eq(n\,578)+eq(n\,583)+eq(n\,587)+eq(n\,591)+eq(n\,595)+eq(n\,599)+eq(n\,603)+eq(n\,608)+eq(n\,613)+eq(n\,617)+eq(n\,62
1)+eq(n\,625)+eq(n\,630)+eq(n\,635)+eq(n\,641)+eq(n\,647)+eq(n\,652)+eq(n\,657)+eq(n\,662)+eq(n\,667)+eq(n\,672)+eq(n\,676)+eq(n\,679)+eq(n\,682)+eq(n\,686)+eq(n\,690)+eq(n\,694)+eq(n\,698)+eq(n\,702)+eq(n\,706)+
eq(n\,710)+eq(n\,714)+eq(n\,718)+eq(n\,722)+eq(n\,726)+eq(n\,730)+eq(n\,734)+eq(n\,738)+eq(n\,742)+eq(n\,747)+eq(n\,752)+eq(n\,757)+eq(n\,762)+eq(n\,768)+eq(n\,774)+eq(n\,782)+eq(n\,789)+eq(n\,797)+eq(n\,805)+eq(
n\,813)+eq(n\,820)+eq(n\,830)+eq(n\,838)+eq(n\,844)+eq(n\,850)+eq(n\,856)+eq(n\,862)+ ....... +eq(n\,14991)+eq(n\,14994)+eq(n\,14997)+eq(n\,15000)+eq(n\,15003)+eq(n\,15006) -vsync 0 -frames:1 6481 -frame_pts 1 -qscale:1 2 D:\Bilder\2023-02-26\R0011493\.mly_ffmpeg_R0011493.MP4_1596_1677441191\R0011493_1_%06d.jpg
2023-02-26 19:53:11,315 - ERROR   - MapillaryFFmpegNotFoundError: The ffmpeg command "ffmpeg" not found
stefanbj commented 1 year ago

Also seems the check (in previous version) to prevent accidental re-uploads image sample is broken, or has been removed.

stefanbj commented 1 year ago

Not exactly the correct solution, but the approach seems to kind of work:

diff --git a/mapillary_tools/ffmpeg.py b/mapillary_tools/ffmpeg.py
index 7290e39..758a5be 100644
--- a/mapillary_tools/ffmpeg.py
+++ b/mapillary_tools/ffmpeg.py
@@ -8,6 +8,7 @@ import re
 import subprocess
 import sys
 import typing as T
+import tempfile^M
 from pathlib import Path

 if sys.version_info >= (3, 8):
@@ -227,6 +229,16 @@ class FFMPEG:
             ouput_template = f"{sample_prefix}_{NA_STREAM_IDX}_%06d{FRAME_EXT}"
             stream_specifier = "v"

+        # prepare video filter grapgh script -- this can become too large for the cmdline
+        vfiltergraph= f"select={eqs}"
+        LOG.info(f"filtergraph: {vfiltergraph}")
+        #vfg = tempfile.NamedTemporaryFile(prefix="ffmpeg_vf",
+        #                                  suffix=".tmp")
+        vfg = open(r"C:\Users\stefanb\AppData\Local\Temp\ffmpeg_vfoy1yg7ih.tmp", "w") 
+        LOG.info(f"filtergraph file is: {str(vfg.name)}")
+        #vfg.write(b"{str(vfiltergraph)}")
+        vfg.write(f"{str(vfiltergraph)}")
+
         cmd: T.List[str] = [
             # global options should be specified first
             *["-hide_banner", "-nostdin"],
@@ -236,7 +248,8 @@ class FFMPEG:
             *stream_selector,
             # filter videos
             *[
-                *["-vf", f"select={eqs}"],
+                #*["-vf", f"select={eqs}"],
+                *["-filter_script", vfg.name],
                 # Each frame is passed with its timestamp from the demuxer to the muxer
                 # vsync is deprecated but -fps_mode is not avaliable on some versions ;(
                 *["-vsync", "0"],
@@ -254,7 +267,9 @@ class FFMPEG:
             ouput_template,
         ]

+        LOG.info(f"extract_specified_frames: {' '.join(cmd)}")
         self._run_ffmpeg(cmd)
+        vfg .close()^M

 class Probe:

The resulting printout is:


PS C:\Users\stefanb\OUTDIR\mt\mapillary_tools>  python3.exe -m mapillary_tools.commands video_process --overwrite_all_EXIF_tags --interpolate_directions  --geotag_source camm D:\Bilder\2023-02-26\R0011493.MP4 D:\Bilder\2023-02-26\R0011493
2023-02-26 23:13:24,717 - INFO    - Extracting video information: ffprobe -print_format json -hide_banner -show_format -show_streams D:\Bilder\2023-02-26\R0011493.MP4
[mov,mp4,m4a,3gp,3g2,mj2 @ 000002070a717c40] UDTA parsing failed retrying raw
    Last message repeated 1 times
[mov,mp4,m4a,3gp,3g2,mj2 @ 000002070a717c40] DTS discontinuity in stream 2: packet 226 with DTS 555979021, packet 227 with DTS 560942136
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'D:\Bilder\2023-02-26\R0011493.MP4':
....
Unsupported codec with id 0 for input stream 2
2023-02-26 23:13:25,023 - INFO    - Extracting video metdata
2023-02-26 23:13:54,396 - INFO    - Found total 1496 GPS points
2023-02-26 23:13:54,397 - INFO    - Extracting video samples
2023-02-26 23:13:54,472 - INFO    - Extracting video samples
2023-02-26 23:13:54,616 - INFO    - Found total 15011 video samples
2023-02-26 23:13:54,617 - INFO    - Interpolating video samples in the time range from 0.555979021 to 1500.720445844
2023-02-26 23:13:54,978 - INFO    - Found total 15002 interpolated video samples
2023-02-26 23:13:55,039 - INFO    - Selected 6481 video samples by the minimal sample distance 3.0
2023-02-26 23:13:55,047 - INFO    - filtergraph: select=eq(n\,6)+eq(n\,43)+eq(n\,55)+eq(n\,70)+eq(n\,88)+eq(n\,97)+eq(n\,104)+eq(n\,110)+eq(n\,116)+eq(n\,122)+eq(n\,126)+eq(n\,130)+eq(n\,134)+eq(n\,137)+eq(n\,140)+eq(n\,143)+eq(n\,146)+eq(n\,149)+eq(n\,152)+eq(n\,155)+eq(n\,158)+eq(n\,161)+eq(n\,164)+eq(n\,167)+eq(n\,170)+eq(n\,173)+eq(n\,176)+eq(n\,179)+eq(n\,182)+eq(n\,185..... 
2023-02-26 23:13:55,058 - INFO    - filtergraph file is: C:\Users\stefanb\AppData\Local\Temp\ffmpeg_vfoy1yg7ih.tmp
2023-02-26 23:13:55,059 - INFO    - extract_specified_frames: -hide_banner -nostdin -i D:\Bilder\2023-02-26\R0011493.MP4 -map 0:1 -filter_script C:\Users\stefanb\AppData\Local\Temp\ffmpeg_vfoy1yg7ih.tmp -vsync 0 -frames:1 6481 -frame_pts 1 -qscale:1 2 D:\Bilder\2023-02-26\R0011493\.mly_ffmpeg_R0011493.MP4_3976_1677449635\R0011493_1_%06d.jpg
2023-02-26 23:13:55,064 - INFO    - Extracting frames: ffmpeg -hide_banner -nostdin -i D:\Bilder\2023-02-26\R0011493.MP4 -map 0:1 -filter_script C:\Users\stefanb\AppData\Local\Temp\ffmpeg_vfoy1yg7ih.tmp -vsync 0 -frames:1 6481 -frame_pts 1 -qscale:1 2 D:\Bilder\2023-02-26\R0011493\.mly_ffmpeg_R0011493.MP4_3976_1677449635\R0011493_1_%06d.jpg
-vsync is deprecated. Use -fps_mode
Passing a number to -vsync is deprecated, use a string argument as described in the manual.
[mov,mp4,m4a,3gp,3g2,mj2 @ 000001e5cbfae780] UDTA parsing failed retrying raw
    Last message repeated 1 times
[mov,mp4,m4a,3gp,3g2,mj2 @ 000001e5cbfae780] DTS discontinuity in stream 2: packet 226 with DTS 555979021, packet 227 with DTS 560942136
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'D:\Bilder\2023-02-26\R0011493.MP4':
...
frame= 6481 fps=8.2 q=24.8 Lsize=N/A time=00:25:00.60 bitrate=N/A speed= 1.9x
video:2861306kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
2023-02-26 23:27:26,133 - WARNING - The specified frame index 13409 was not extracted from stream index 1
2023-02-26 23:27:26,133 - WARNING - The specified frame index 13413 was not extracted from stream index 1
2023-02-26 23:27:26,151 - WARNING - The specified frame index 13449 was not extracted from stream index 1
2
....
2023-02-26 23:27:26,271 - WARNING - The specified frame index 14993 was not extracted from stream index 1
2023-02-26 23:27:26,271 - WARNING - The specified frame index 14996 was not extracted from stream index 1
2023-02-26 23:27:26,271 - WARNING - The specified frame index 14999 was not extracted from stream index 1
2023-02-26 23:27:26,271 - WARNING - The specified frame index 15002 was not extracted from stream index 1
2023-02-26 23:27:26,271 - WARNING - The specified frame index 15005 was not extracted from stream index 1
2023-02-26 23:27:26,290 - WARNING - Force the option "filetypes" to be "image" to avoid processing and uploading both the video samples and the videos themselves
2023-02-26 23:27:26,290 - WARNING - Since sample geotags have been written in EXIFs when you sampled videos by distance (3.0 meter(s)), so we force the option "geotag_source" to be "exif" to avoid unnecessary geotagging from the videos again
Extracting geotags from images: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6481/6481 [00:27<00:00, 234.31images/s]
Overwriting EXIF: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5839/5839 [00:12<00:00, 450.74images/s]
Verifying image EXIF writing: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5839/5839 [00:40<00:00, 143.61images/s]
2023-02-26 23:29:24,091 - INFO    - Check the description file for details: D:\Bilder\2023-02-26\R0011493\mapillary_image_description.json
2023-02-26 23:29:24,101 - INFO    -     6481 image(s) read in total
2023-02-26 23:29:24,102 - INFO    -          5839 image(s) are ready to be uploaded
2023-02-26 23:29:24,102 - ERROR   -           642 image(s) failed due to MapillaryGeoTaggingError
2023-02-26 23:29:24,103 - ERROR   - MapillaryProcessError: Failed to process 642 files. To skip these errors, specify --skip_process_errors
PS C:\Users\stefanb\OUTDIR\mt\mapillary_tools>```
ptpt commented 1 year ago

Thanks for reporting the issue! Could you try out the latest main branch and see if it fixes? Reopen if it does not.

stefanbj commented 1 year ago

Just did a new upload using main/HEAD (8be59838ecb75a7aaa1a9c199ee13a47689d9b78), and can confirm that the issue is now resolved for me. (Sorry fr the late reply.)