Fix upload_photos_by_exif.py to work with Mapillary EXIF

[hbogner]

I have a lot of photos taken with Mapillary app and would like to upload them to OSV too.

When I start upload_photos_by_exif.py with full path i get this error:

:~/git/uploader/upload_photos_by_exif$ python3 upload_photos_by_exif.py -p /home/hbogner/git/uploader/upload_photos_by_exif/2016_08_01_sg2_1/
/home/hbogner/git/uploader/upload_photos_by_exif/osv/local/lib/python3.5/site-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data.  Expecting to read 3806836976 bytes but only got 0. Skipping tag 14286
  "Skipping tag %s" % (size, len(data), tag))
/home/hbogner/git/uploader/upload_photos_by_exif/osv/local/lib/python3.5/site-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data.  Expecting to read 24032256 bytes but only got 0. Skipping tag 4352
  "Skipping tag %s" % (size, len(data), tag))
/home/hbogner/git/uploader/upload_photos_by_exif/osv/local/lib/python3.5/site-packages/PIL/TiffImagePlugin.py:694: UserWarning: Corrupt EXIF data.  Expecting to read 12 bytes but only got 3.
  warnings.warn(str(msg))
Traceback (most recent call last):
  File "upload_photos_by_exif.py", line 411, in <module>
    main(sys.argv[1:])
  File "upload_photos_by_exif.py", line 302, in main
    latitude, longitude, compas = get_gps_lat_long_compass(path + photo_path)
TypeError: 'NoneType' object is not iterable

There is EXIF written in each photo, here is gdalinfo output:

:~/git/uploader/upload_photos_by_exif$ gdalinfo 2016_08_01_sg2_1/2016_08_01_11_22_48_868.jpg
Driver: JPEG/JPEG JFIF
Files: 2016_08_01_sg2_1/2016_08_01_11_22_48_868.jpg
Size is 3264, 2448
Coordinate System is `'
Metadata:
  EXIF_ApertureValue=(2.81)
  EXIF_BrightnessValue=(8.31)
  EXIF_ColorSpace=1
  EXIF_DateTime=2016:08:01 11:22:48
  EXIF_DateTimeDigitized=2016:08:01 11:22:48
  EXIF_DateTimeOriginal=2016:08:01 11:22:48
  EXIF_ExifVersion=0220
  EXIF_ExposureBiasValue=(8)
  EXIF_ExposureMode=0
  EXIF_ExposureProgram=3
  EXIF_ExposureTime=(0.00215054)
  EXIF_Flash=16
  EXIF_FNumber=(2.65)
  EXIF_FocalLength=(4.03)
  EXIF_GPSImgDirection=(164.53)
  EXIF_GPSLatitude=(45) (28) (52.0536)
  EXIF_GPSLatitudeRef=N
  EXIF_GPSLongitude=(15) (33) (13.8348)
  EXIF_GPSLongitudeRef=E
  EXIF_ImageDescription={"MAPDeviceMake":"Samsung","MAPCameraMode":3,"MAPCaptureTime":"2016_08_01_11_22_48_868","MAPSettingsEmail":"hbogner+mapillary@gmail.com","MAPAPI":"LEGACY","MAPLocalTimeZone":"+0200","MAPVersionString":"2.37-jaakkoh","MAPCompassHeading":{"TrueHeading":168.06877278855276,"MagneticHeading":164.52964400818777},"MAPDeviceModel":"GT-I9100","MAPPhotoUUID":"8c153ac0-1e39-4d64-8406-9d306c1d8fbc","MAPLatitude":"45.481126","MAPAutoFocusMode":"INFINITY","MAPGPSAccuracyMeters":"3.9","MAPSequenceUUID":"e11687bf-7867-4789-98d6-823535a2ff35","MAPAltitude":"117.1","MAPSettingsUploadHash":"328a579db1b98f19de4612378fb3a4281b760bcd2e004e3802db8abbecce0523","MAPAccelerometerVector":{"z":-1.372520089149475,"y":-0.07459858059883118,"x":9.458673477172852},"MAPLongitude":"15.553843","MAPPicRotation":0}
  EXIF_ISOSpeedRatings=32
  EXIF_Make=Samsung
  EXIF_MaxApertureValue=(2.81)
  EXIF_MeteringMode=2
  EXIF_Model=GT-I9100
  EXIF_Orientation=1
  EXIF_PixelXDimension=3264
  EXIF_PixelYDimension=2448
  EXIF_SceneCaptureType=0
  EXIF_ShutterSpeedValue=(8.86)
  EXIF_Software=KTU84Q
  EXIF_UserComment=User comments
  EXIF_WhiteBalance=0
  EXIF_YCbCrPositioning=1
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 2448.0)
Upper Right ( 3264.0,    0.0)
Lower Right ( 3264.0, 2448.0)
Center      ( 1632.0, 1224.0)
Band 1 Block=3264x1 Type=Byte, ColorInterp=Red
  Overviews: 1632x1224, 816x612, 408x306
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 2 Block=3264x1 Type=Byte, ColorInterp=Green
  Overviews: 1632x1224, 816x612, 408x306
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 3 Block=3264x1 Type=Byte, ColorInterp=Blue
  Overviews: 1632x1224, 816x612, 408x306
  Image Structure Metadata:
    COMPRESSION=JPEG

Please find a way to read those photos too.

[bogdan-racasan]

@hbogner Thanks for reporting :) can you gave me a link with some photos.

[hbogner]

Take this one:

[bogdan-racasan]

Thanks

[bogdan-racasan]

@hbogner i fixed the issue. Please try again and i will appreciate any feedback

[hbogner]

Upload went OK. Sequence shows ok on map, but when viewing sequence photos are not in order. Take a look at the sequence: http://openstreetview.org/map/@45.48046211462135,15.552319586277006,18z

[bogdan-racasan]

Can you send me this photos ?? i know what is the problem but i have to investigate and fix this problem in general.

[hbogner]

Here you go: https://dl.dropboxusercontent.com/u/3220458/2016_08_01-sg2.zip

[bogdan-racasan]

Ok, i fixed this issue by sorting different the photos because there was several images with the same timestamp. Can you please delete from your folder where you have the images: -sequence_file.txt -count_file.txt

The try again to upload those images and delete the sequence with the bug? Thanks. I will appreciate any feedback 👍

[hbogner]

Every photo has different time stamp. Deleted old sequence and those files, then repeated the upload, it looks the same.

[bogdan-racasan]

If i print for several images the timestamp: 2016_08_01-sg2_3/2016_08_01_11_22_48_868.jpg 2016-08-01 11:47:44 2016_08_01-sg2_3/2016_08_01_11_22_52_575.jpg 2016-08-01 11:47:44 2016_08_01-sg2_3/2016_08_01_11_22_53_740.jpg 2016-08-01 11:47:44 2016_08_01-sg2_3/2016_08_01_11_23_19_133.jpg 2016-08-01 11:47:44 2016_08_01-sg2_3/2016_08_01_11_23_20_369.jpg 2016-08-01 11:47:44 2016_08_01-sg2_3/2016_08_01_11_23_21_549.jpg 2016-08-01 11:47:44 2016_08_01-sg2_3/2016_08_01_11_23_22_774.jpg 2016-08-01 11:47:46

Also if i'm looking into the exif the timestamp is the same of the images...

[bogdan-racasan]

can you try again with the same steps? Thanks 👍

[hbogner]

You are looking at the wrong place for time info ;) 2016-08-01 11:47:44 is the time they were copied from phone to the computer.

gdalinfo 2016_08_01_11_22_48_868.jpg | grep Time
  EXIF_DateTime=2016:08:01 11:22:48
  EXIF_DateTimeDigitized=2016:08:01 11:22:48
  EXIF_DateTimeOriginal=2016:08:01 11:22:48
  EXIF_ExposureTime=(0.00215054)
  EXIF_ImageDescription={"MAPDeviceMake":"Samsung","MAPCameraMode":3,"MAPCaptureTime":"2016_08_01_11_22_48_868","MAPSettingsEmail":"hbogner+mapillary@gmail.com","MAPAPI":"LEGACY","MAPLocalTimeZone":"+0200","MAPVersionString":"2.37-jaakkoh","MAPCompassHeading":{"TrueHeading":168.06877278855276,"MagneticHeading":164.52964400818777},"MAPDeviceModel":"GT-I9100","MAPPhotoUUID":"8c153ac0-1e39-4d64-8406-9d306c1d8fbc","MAPLatitude":"45.481126","MAPAutoFocusMode":"INFINITY","MAPGPSAccuracyMeters":"3.9","MAPSequenceUUID":"e11687bf-7867-4789-98d6-823535a2ff35","MAPAltitude":"117.1","MAPSettingsUploadHash":"328a579db1b98f19de4612378fb3a4281b760bcd2e004e3802db8abbecce0523","MAPAccelerometerVector":{"z":-1.372520089149475,"y":-0.07459858059883118,"x":9.458673477172852},"MAPLongitude":"15.553843","MAPPicRotation":0}

gdalinfo 2016_08_01_11_23_29_838.jpg | grep Time
  EXIF_DateTime=2016:08:01 11:23:29
  EXIF_DateTimeDigitized=2016:08:01 11:23:29
  EXIF_DateTimeOriginal=2016:08:01 11:23:29
  EXIF_ExposureTime=(0.00102459)
  EXIF_ImageDescription={"MAPDeviceMake":"Samsung","MAPCameraMode":3,"MAPCaptureTime":"2016_08_01_11_23_29_838","MAPSettingsEmail":"hbogner+mapillary@gmail.com","MAPAPI":"LEGACY","MAPLocalTimeZone":"+0200","MAPVersionString":"2.37-jaakkoh","MAPCompassHeading":{"TrueHeading":266.5192448135241,"MagneticHeading":262.98080244024834},"MAPDeviceModel":"GT-I9100","MAPPhotoUUID":"db4c8358-d4ef-4e56-9292-6d480818e205","MAPLatitude":"45.479709","MAPAutoFocusMode":"INFINITY","MAPGPSAccuracyMeters":"4.5","MAPSequenceUUID":"ebf61469-5698-47cf-9e78-389d0600cf24","MAPAltitude":"116.1","MAPSettingsUploadHash":"c0952aded7470e92d575c1e39278aee3cfc358ec7898ca9b30be2746f1fb2150","MAPAccelerometerVector":{"z":-0.7468734383583069,"y":0.1515873521566391,"x":9.505647659301758},"MAPLongitude":"15.550836","MAPPicRotation":0}

[bogdan-racasan]

Thanks for info i will fix this tomorrow 👍

[bogdan-racasan]

Solve this issue by sorting from exif timestamp. Please test it again and if it's ok i will close this issue. Thanks @hbogner

[hbogner]

First two photos seem to be switched, I'll test on another larger sequence to see If it works fine. http://openstreetview.org/details/14475/0 Same sequence on mapillary: https://www.mapillary.com/app/?lat=45.481126&lng=15.553843&z=17&pKey=4M-BMjhQbQHsl2WHSwOITw&focus=photo

Another point:

Date of recording: Aug 25, 2016

Shouldn't that be read from exif, and not the time of the upload?

[hbogner]

It would also be great to read mapillary sequence uuid and to automaticaly split them into different OSV sequences. I have all my photos grouped by days, not individual sequences.

[bogdan-racasan]

We could create a script that will split the photos in several folders if the exif timestamp will differ with more than one minute. What would you say is this a good solution? And the date of recording is on progress :)

[bogdan-racasan]

@hbogner Can we still help you with sorting your photos and upload tool?

[hbogner]

@bogdan-telenav Yes, that kind of a script would help. Busy currently so more feedback later. Script could read Mapillary sequence UUID, or even timestamp as you said, and split them into several folder.

[bogdan-racasan]

@hbogner i just commit the script that separate the photos from a folder. You only need python 3 to run and have the same dependencies from upload_photos_by_exif. eg: python sort_by_timestamp /Full/path/to/images This will sort your photos based on timestamp if there are photos taken by 30s different time this will split in different folder with timestamp name for the folders.

[hbogner]

OK, give me some time to test it. Not much free time at the moment.