trek-view / gopro2gsv

Processes .jpg images and .mp4/.360 videos shot on GoPro MAX or Fusion cameras and uploads to Google Street View.
Apache License 2.0
4 stars 0 forks source link

Add dual fisheye .jpg mode (Fusion) #31

Open himynamesdave opened 9 months ago

himynamesdave commented 9 months ago

Another video mode, this time will accept dual jpg images shot in fisheye.

GoPro Fusion cameras also produce a dual .jpg format (two fisheye images that can be stitched together to create a single equirectangular 360).

Gopro2gsv validates as follows:

We already have some old code to join

Tool: https://github.com/trek-view/fusion2sphere Example implementation in another tool: https://github.com/trek-view/gopro2frames/blob/main/docs/LOGIC.md#7b-fusion2sphere-flow-2-gopro-fisheye-frames

Note: resusing code is fine, but do not copy the whole codebase of this implementation.

After this, the GPS points are then embedded to the equirectangular images (same as https://github.com/trek-view/gopro2gsv/issues/26) and the final video is processed in the same way as the image->video pipeline including the new function; https://github.com/trek-view/gopro2gsv/issues/25

In this mode, a user should be able to set --keep_extracted_frames which will also store a copy of the extracted images used to create final video (with geotags). If not set, frames extracted will be removed on completion.

===== below example docs

dual fisheye timelapse frames -> fusion2sphere -> timelapse frames -> final video (for Fusion cameras)

Mode 5: dual fisheye timelapse frames -> fusion2sphere -> timelapse frames -> final video (for Fusion cameras)

Fusion cameras produce a pair of two .jpg fisheye images. These need to be converted into equirectangular format for use with most major viewing software (e.g. GSV).

Overview:

5a: User options on input

Same as 2a: User options on input

5b: Validation

Gopro2gsv validates as follows:

  1. GoPro2GSV checks the directory for .jpg files (case insensitive). It ignores any other files that might be present.
  2. Checks there are >=40 frames in the input (20 final images)
  3. GoPro2GSV checks the filename prefixes all match to a single timelapse recorded on the camera and that there is a pair of GPFR and GPBK with same 4 digit number, e.g. GPFR0001.JPG+GPBK0001.JPG, GPFR0002.JPG+GPBK0002.JPG ...
  4. GoPro2GSV checks the metadata of all valid GPFR images from previous step using exiftool, and images that are not valid are discarded;
    • file extension must be .jpg (case insensitive)
    • GPS:GPSLatitudeRef must be present
    • GPS:GPSLatitude must be present
    • GPS:GPSLongitudeRef must be present
    • GPS:GPSLongitude must be present
    • GPS:GPSAltitudeRef must be present
    • GPS:GPSAltitude must be present
    • GPS:GPSTimeStamp must be present
    • GPS:GPSDateStamp must be present
  5. GoPro2GSV checks all valid images from previous step have the same dimensions by ensuring all images have the same values for File:ImageWidth and File:ImageHeight properties
exiftool -ee -G3 -api LargeFileSupport=1 -ProjectionType -GPSLatitudeRef -GPSLatitude -GPSLongitudeRef -GPSLongitude -GPSAltitudeRef -GPSAltitude -GPSTimeStamp -GPSDateStamp -ImageWidth -ImageHeight GPFR<IMAGE>.jpg
  1. GoPro2GSV sorts all valid images from previous step by time (GPS:GPSDateStamp+GPS:GPSDateStamp), earliest to latest.
  2. GoPro2GSV checks that time between consecutive GPFR images (GPS:GPSDateStamp+GPS:GPSDateStamp) is not >20 seconds. If true then the whole directory is not considered and an ERROR log is reported
  3. GoPro2GSV checks that a directory with all valid images from previous step has >=40 valid images after the previous step. If <40 valid images then the whole directory is not considered and an ERROR log is reported
  4. GoPro2GSV renames the images files in format NNNNN.jpg, starting from 00001.jpg and ascends, e.g. 00001.jpg, 00002.jpg...
  5. GoPro2GSV tracks the data of each timelapse image, including those that failed validation, in the local database. The original and new filename is included so that it is clear what files were not considered in the final video (and why)

5c: Processing

Useful supporting information for this section;

Processing is broken up into four key parts;

  1. extract GPS from video
  2. extracting frames from .360 input
  3. turning extracted frames into equirectangular projections using max2sphere
  4. geotag extracted frames