Error uploading photos by exif

[Xevib]

I got this message:

usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 524288 bytes but only got 6108. Skipping tag 1 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 9764864 bytes but only got 0. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 9830400 bytes but only got 7108. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 28311552 bytes but only got 7108. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 65536 bytes but only got 7108. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 243400704 bytes but only got 0. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 18481152 bytes but only got 0. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 19988480 bytes but only got 0. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 2271805440 bytes but only got 0. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 2359296 bytes but only got 0. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 786432 bytes but only got 0. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 134742016 bytes but only got 0. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:694: UserWarning: Corrupt EXIF data. Expecting to read 12 bytes but only got 6. warnings.warn(str(msg)) No sequence file existing Found 4 pictures to upload /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 524288 bytes but only got 6107. Skipping tag 1 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 9830400 bytes but only got 7107. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 28311552 bytes but only got 7107. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 65536 bytes but only got 7107. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 254541824 bytes but only got 0. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 25600 bytes but only got 7107. Skipping tag 52224 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:694: UserWarning: Corrupt EXIF data. Expecting to read 12 bytes but only got 5. warnings.warn(str(msg)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 524288 bytes but only got 6105. Skipping tag 1 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 9830400 bytes but only got 7105. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 28311552 bytes but only got 7105. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 65536 bytes but only got 7105. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 245760000 bytes but only got 0. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:694: UserWarning: Corrupt EXIF data. Expecting to read 12 bytes but only got 3. warnings.warn(str(msg)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 524288 bytes but only got 6104. Skipping tag 1 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 6375342080 bytes but only got 7104. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 9830400 bytes but only got 7104. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 28311552 bytes but only got 7104. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 65536 bytes but only got 7104. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 235798528 bytes but only got 0. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 262146 bytes but only got 0. Skipping tag 2 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 262151 bytes but only got 0. Skipping tag 56 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 524288 bytes but only got 0. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 5242880 bytes but only got 0. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:680: UserWarning: Possibly corrupt EXIF data. Expecting to read 875954176 bytes but only got 0. Skipping tag 0 "Skipping tag %s" % (size, len(data), tag)) /usr/local/lib/python3.5/dist-packages/PIL/TiffImagePlugin.py:694: UserWarning: Corrupt EXIF data. Expecting to read 12 bytes but only got 2. warnings.warn(str(msg)) processing 2016_07_31_13_50_25_903.jpg Uploaded - 1 of total :4, percentage: 25.0% processing 2016_07_31_13_50_22_794.jpg Uploaded - 2 of total :4, percentage: 50.0% processing 2016_07_31_13_50_24_389.jpg Uploaded - 3 of total :4, percentage: 75.0% processing 2016_07_31_13_50_21_415.jpg Uploaded - 4 of total :4, percentage: 100.0% Finish uploading form dir: /home/xevi/Baixades/hector/ with sequence id: 11023

The result is this sequence http://openstreetview.org/details/11023/0

[james2432]

you might not have gps exif data in certain images. Try viewing it with an exif viewer to see if you have the latitude and longitude under the gps section

[Xevib]

@james2432 the images has exif tags at least i can see it at genome image viewer. If you want I can check it with some tty app

[james2432]

Seems it's a pillow error(library they use) that fails to handle corrupted exif data: https://github.com/python-pillow/Pillow/issues/518

[Xevib]

@james2432 does it work with a specific version of pillow?

[bogdan-racasan]

@Xevib Can you send us some pictures by a dropbox link or something to download them?

[Xevib]

@bogdan-telenav https://www.dropbox.com/sh/aow8sglrzmewnkt/AADNV9hX8EZWcViKPHaf-po1a?dl=0

[bogdan-racasan]

@Xevib as i can see there is a problem with Pillow library but i use another one library because there are some images that work it Pillow and the other work with exifread. So there are only some warning's but moves forward and finish the upload with valid data. However i don't know why i can see the images on openstreetview.com

[drkludge]

@james2432 I downloaded @Xevib's pictures. They open just fine in JOSM. There's no corruption.

[drkludge]

@bogdan-telenav please provide more details on how to install the environment if you feel that there is a library issue.

[drkludge]

I am experiencing similar failures on two different python environments. On a QNAP nas with python3 and pytyhon 2.7 I used alias pip='/share/CACHEDEV1_DATA/.qpkg/Python3/python3/bin/pip' alias python='/share/CACHEDEV1_DATA/.qpkg/Python3/python3/bin/python3.5'

On Fedora 24 I used alias pip='/usr/bin/pip3.4' alias python='/usr/bin/python3'

then piped these packages.

pip install exifread pip install http pip install urllib5 pip install rauth pip install Image

Both old and new pictures have the same issue.

python ~/mapillary/upload_photos_by_exif.py -p /home/drkludge/mapillary/2015_08_02_17_17_53_999___Mapillary -r -t 8 /usr/lib64/python3.4/site-packages/PIL/TiffImagePlugin.py:555: UserWarning: Possibly corrupt EXIF data. Expecting to read 285413634 bytes but only got 0. Skipping tag 780 "Skipping tag %s" % (size, len(data), tag)) /usr/lib64/python3.4/site-packages/PIL/TiffImagePlugin.py:555: UserWarning: Possibly corrupt EXIF data. Expecting to read 9015296 bytes but only got 0. Skipping tag 26368 "Skipping tag %s" % (size, len(data), tag)) /usr/lib64/python3.4/site-packages/PIL/TiffImagePlugin.py:569: UserWarning: Corrupt EXIF data. Expecting to read 12 bytes but only got 11. warnings.warn(str(msg)) Traceback (most recent call last): File "/home/drkludge/mapillary/upload_photos_by_exif.py", line 411, in main(sys.argv[1:]) File "/home/drkludge/mapillary/upload_photos_by_exif.py", line 302, in main latitude, longitude, compas = get_gps_lat_long_compass(path + photo_path) TypeError: 'NoneType' object is not iterable

python ~/mapillary/upload_photos_by_exif.py -p /home/drkludge/mapillary/2016_07_30_15_12_01_739___Mapillary -r -t 8 /usr/lib64/python3.4/site-packages/PIL/TiffImagePlugin.py:555: UserWarning: Possibly corrupt EXIF data. Expecting to read 285413634 bytes but only got 0. Skipping tag 780 "Skipping tag %s" % (size, len(data), tag)) /usr/lib64/python3.4/site-packages/PIL/TiffImagePlugin.py:555: UserWarning: Possibly corrupt EXIF data. Expecting to read 25600 bytes but only got 0. Skipping tag 22272 "Skipping tag %s" % (size, len(data), tag)) /usr/lib64/python3.4/site-packages/PIL/TiffImagePlugin.py:555: UserWarning: Possibly corrupt EXIF data. Expecting to read 25600 bytes but only got 11727. Skipping tag 5120 "Skipping tag %s" % (size, len(data), tag)) /usr/lib64/python3.4/site-packages/PIL/TiffImagePlugin.py:555: UserWarning: Possibly corrupt EXIF data. Expecting to read 53248 bytes but only got 9423. Skipping tag 38144 "Skipping tag %s" % (size, len(data), tag)) /usr/lib64/python3.4/site-packages/PIL/TiffImagePlugin.py:569: UserWarning: Corrupt EXIF data. Expecting to read 12 bytes but only got 9. warnings.warn(str(msg)) Traceback (most recent call last): File "/home/drkludge/mapillary/upload_photos_by_exif.py", line 411, in main(sys.argv[1:]) File "/home/drkludge/mapillary/upload_photos_by_exif.py", line 302, in main latitude, longitude, compas = get_gps_lat_long_compass(path + photo_path) TypeError: 'NoneType' object is not iterable

[drkludge]

Here you go. All 315 images load in Josm with a File>Open operation as shown in the image. Here's the link to the images in Google Drive. https://drive.google.com/folderview?id=0Bw7WBSr9fv4tVUE5LWpNR1dNdTA&usp=sharing

[bogdan-racasan]

Thanks for reporting @drkludge we are working on it and try different type of exif readers and try to solve the problem.

[drkludge]

I was curious about this problem. Is the image library providing you with a GPSInfo key. I could not find that key via gdalinfo or a read of the file with exifread. exifread had no trouble locating the keys used in your calculations i.e. 'GPS GPSLatitude'.

def get_gps_lat_long_compass(path_image):
    image = Image.open(path_image)
...
        exif_data_gpsInfo = exif_data.get('GPSInfo')
        if not exif_data_gpsInfo:
            raise ValueError("No GPS metadata found.")

A hack of https://pypi.python.org/pypi/ExifRead page's example works on the same file that fails with an image read. So is image more trouble than it is worth compared to using exifread?

more pyexif.py
import exifread
# Open image file for reading (binary mode)
f = open('2016_07_30_15_19_07_580.jpg', 'rb')

# Return Exif tags
tags = exifread.process_file(f)
for tag in tags.keys():
    # if tag not in ('JPEGThumbnail', 'TIFFThumbnail', 'Filename', 'EXIF MakerNote'):
    if tag in ('GPS GPSImgDirection', 'GPS GPSLatitude', 'GPS GPSLatitudeRef', 'GPS GPSLongitude', 'GPS GPSLongitudeRef'):

        print ("Key: %s, value %s" % (tag, tags[tag]))

 python pyexif.py |sort
Key: GPS GPSImgDirection, value 3002679/10474
Key: GPS GPSLatitude, value [33, 26, 395176/7581]
Key: GPS GPSLatitudeRef, value N
Key: GPS GPSLongitude, value [112, 2, 167151/12500]
Key: GPS GPSLongitudeRef, value W

[bogdan-racasan]

@drkludge I managed to upload this photos using this type of reading the exif. However the route renders fine on the map but there are no photos...

[bogdan-racasan]

@drkludge @james2432 @Xevib i fixed this issue. I will appreciate if you test it and send me some feedback. Thanks again

[Xevib]

@bogdan-telenav I don't know if its related with the problem , but now I'm getting this error

Traceback (most recent call last): File "/home/xevi/gits/uploader/upload_photos_by_exif/upload_photos_by_exif.py", line 9, in import urllib.request, urllib.error, urllib.parse ImportError: No module named request

I'm using python 2.7.11+ and my pip freeze is:

apt-xapian-index==0.47 apturl==0.5.2 beautifulsoup4==4.4.1 blinker==1.3 Brlapi==0.6.4 cairocffi==0.7.2 certifi==2016.2.28 cffi==1.5.2 chardet==2.3.0 checkbox-support==0.22 click==6.6 command-not-found==0.3 configobj==5.0.6 contextlib2==0.5.3 coverage==4.2 cryptography==1.2.3 defer==1.0.6 devscripts===2.16.2ubuntu3 Django==1.9.8 ExifRead==2.1.2 expects==0.8.0 feedparser==5.1.3 Flask==0.11.1 future==0.15.2 geoip2==2.4.0 geojson==1.3.3 google-api-python-client==1.5.1 guacamole==0.9.2 html5lib==0.999 httplib2==0.9.2 idna==2.0 image==1.4.4 itsdangerous==0.24 Jinja2==2.8 language-selector==0.1 louis==2.6.4 lxml==3.5.0 Magic-file-extensions==0.2 Mako==1.0.3 mapnik==0.1 MarkupSafe==0.23 maxminddb==1.2.1 oauth2client==3.0.0 oauthlib==1.0.3 onboard==1.2.0 oneconf==0.3.9 osconf==0.1.3 osmapi==0.6.2 overpass==0.4.0 padme==1.1.1 pexpect==4.0.1 Pillow==3.1.1 piston-mini-client==0.7.5 plainbox==0.25 ply==3.7 psycopg2==2.6.2 ptyprocess==0.5 pyasn1==0.1.9 pyasn1-modules==0.0.8 pycparser==2.14 pycups==1.9.73 pycurl==7.43.0 pygobject==3.20.0 PyJWT==1.3.0 pynominatim==0.2.4 pyparsing==2.0.3 pystache==0.5.4 python-apt==1.1.0b1 python-debian==0.1.27 python-systemd==231 python-telegram-bot==5.0.0 pyxdg==0.25 rauth==0.7.2 raven==5.23.0 reportlab==3.3.0 requests==2.9.1 rsa==3.4.2 screen-resolution-extra==0.0.0 sessioninstaller==0.0.0 simplejson==3.8.2 six==1.10.0 software-center-aptd-plugins==0.0.0 ssh-import-id==5.5 system-service==0.3 ubuntu-drivers-common==0.0.0 ufw==0.35 unattended-upgrades==0.1 unity-scope-calculator==0.1 unity-scope-chromiumbookmarks==0.1 unity-scope-colourlovers==0.1 unity-scope-devhelp==0.1 unity-scope-gdrive==0.7 unity-scope-manpages==0.1 unity-scope-openclipart==0.1 unity-scope-texdoc==0.1 unity-scope-tomboy==0.1 unity-scope-virtualbox==0.1 unity-scope-yelp==0.1 unity-scope-zotero==0.1 uritemplate==0.6 urllib3==1.16 usb-creator==0.3.0 Werkzeug==0.11.10 xcffib==0.3.6 xdiagnose==3.8.4 xkit==0.0.0 XlsxWriter==0.7.3

[bogdan-racasan]

This tool is functional only with python 3.x, and i'm using python 3.5.1

[Xevib]

@bogdan-telenav ok , please update this https://github.com/openstreetview/uploader#requirements to avoid confusions

With python3.5 I'm geting this message Fail to create the sequence Please restart the script

I debugged the script and the variable id_sequence is empty on the line 321

[bogdan-racasan]

@Xevib Can you go to the folder and delete if there exist an sequence_file.txt and a count_file.txt form the folder where you have the images am the restart the script ?

[drkludge]

@bogdan-telenav Thanks. I will try it this evening.

[drkludge]

It looks like there's some more work to be done. I received a login error on a new use of the *py code.

1.I received the login error. I tried a few more times. I then logged into both osv and osm. With no luck. 2.I used touch id_file.txt 3.I used chmod ugo+rw id_file.txt 4.I added my osm_numeric_id;osm_user_text_id to the id_file.txt file. 5.The id_file.txt and the upload py file are at the same level. Please see the ls command.

Finally, I was given a good report on the QNAP NAS box with 315 files. That matches the JOSM count that I posted earlier.

python upload_photos_by_exif.py -p 2016_07_30_15_12_01_739___Mapillary
No sequence file existing
Found 315 pictures to upload
processing 2016_07_30_15_24_02_847.jpg
Uploaded - 1 of total :315, percentage: 0.32000000000000001%
...
processing 2016_07_30_15_23_53_287.jpg
Uploaded - 315 of total :315, percentage: 100.0%
Finish uploading form dir: /share/external/DEV3301_2/gis/data/osm_survey/mapillary/2016_07_30_15_12_01_739___Mapillary/ with sequence id: 14314

ls -ld id* upload* 2016_07_30_15_12_01_739___Mapillary
drwxrwxrwx    1 admin    administ     90112 Aug 24 21:43 2016_07_30_15_12_01_739___Mapillary/
-rw-rw-rw-    1 admin    administ        17 Aug 24 21:43 id_file.txt
-rwxrwxrwx    1 admin    administ     18322 Aug 24 21:10 upload_photos_by_exif.py*
-rwxrwxrwx    1 admin    administ     17753 Aug 13 06:33 upload_photos_by_exif2.py*
-rwxrwxrwx    1 admin    administ     17757 Aug  5 22:32 upload_photos_by_exif_old.py*
-rw-rw-rw-    1 admin    administ      8832 Aug  6 00:00 uploader_install.txt

... Login and grant acces then press ENTER

Traceback (most recent call last): File "upload_photos_by_exif.py", line 210, in main id_file = open("id_file.txt", "r+") FileNotFoundError: [Errno 2] No such file or directory: 'id_file.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 1240, in do_open h.request(req.get_method(), req.selector, req.data, headers) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/http/client.py", line 1083, in request self._send_request(method, url, body, headers) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/http/client.py", line 1128, in _send_request self.endheaders(body) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/http/client.py", line 1079, in endheaders self._send_output(message_body) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/http/client.py", line 911, in _send_output self.send(msg) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/http/client.py", line 854, in send self.connect() File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/http/client.py", line 1237, in connect server_hostname=server_hostname) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/ssl.py", line 376, in wrap_socket _context=self) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/ssl.py", line 747, in init self.do_handshake() File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/ssl.py", line 983, in do_handshake self._sslobj.do_handshake() File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/ssl.py", line 628, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:646)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "upload_photos_by_exif.py", line 424, in main(sys.argv[1:]) File "upload_photos_by_exif.py", line 259, in main opener.open(urllib.request.Request(authorize_url)) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 471, in open response = meth(req, response) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 581, in http_response 'http', request, response, code, msg, hdrs) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 503, in error result = self._call_chain(_args) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 443, in _call_chain result = func(_args) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 686, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 471, in open response = meth(req, response) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 581, in http_response 'http', request, response, code, msg, hdrs) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 503, in error result = self._call_chain(_args) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 443, in _call_chain result = func(_args) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 686, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 465, in open response = self._open(req, data) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 483, in _open '_open', req) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 443, in _call_chain result = func(*args) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 1283, in https_open context=self._context, check_hostname=self._check_hostname) File "/share/CACHEDEV1_DATA/.qpkg/Python3/python3/lib/python3.5/urllib/request.py", line 1242, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:646)> [/share/mapillary/gis/data/osm_survey/mapillary] #

[Xevib]

@bogdan-telenav happenns the same after removing the files

[bogdan-racasan]

@drkludge Can you retry the upload with the latest fixes on upload_by_exif because there was a bug by jumping the images between locations. And i fix the problem by sorting different. Please send some feedback with sequence number. And i see that you uploaded a lot of images 👍 Thanks

[bogdan-racasan]

@Xevib can you retry ... I added several logs and send me the error because i can't reproduce it.

[Xevib]

@bogdan-telenav With the last version I'm getting this error: Traceback (most recent call last): File "/home/xevi/gits/uploader/upload_photos_by_exif/upload_photos_by_exif.py", line 441, in main(sys.argv[1:]) File "/home/xevi/gits/uploader/upload_photos_by_exif/upload_photos_by_exif.py", line 323, in main time_stamp_list.append({"file": photo_path, "timestamp": get_exif(path + photo_path).values}) File "/home/xevi/gits/uploader/upload_photos_by_exif/upload_photos_by_exif.py", line 25, in get_exif dateTaken = tags["EXIF DateTimeOriginal"] KeyError: 'EXIF DateTimeOriginal'

[drkludge]

@bogdan-telenav Will a new upload replace the pictures that I have already uploaded? I ask because it says that I have 100+ tracks being processed. I do have one track uploading at the moment.

1.Can you remove all these tracks? 2.Do I need to remove the count.txt and sequence_file.txt in the image directory before uploading the files again?

I can try again tonight.

[bogdan-racasan]

@drkludge

1. I can remove the tracks but you can doit faster by going to your account and press the garbage icon, but we only can do this in the next week.
2. Yes you have to delete count.txt and sequence_file.txt manually.
3. The pictures will not be replaced ...
4. Can you try with one folder and send me and sequence id? Thanks Thanks again for helping 👍

[drkludge]

I removed all the tracks via the OSV interface. The track paths are still on the map. I uploaded the 2016_07_30_15_12_01_739___Mapillary directory of photos again via sequence id: 14860. Let me know if these photos are good.

[bogdan-racasan]

@drkludge This is good 👍 Thanks for the feedback 👍 You can go one.

[drkludge]

@bogdan-telenav Thanks for taking the time to listen to several reports and fixing the problems as they showed up. I will start the upload over the weekend.

1.I submitted a ticket concerning the removal of images on the website. 2.Did you fix the initial id_file.txt issue? I had to create the file before the program would work.

[bogdan-racasan]

@drkludge

1. I'm trying to reproduce but i can't find the pattern ... but i'm gonna try more 👍 and let you know when we solve this issue
2. We know about the issue when remove the deleted tracks and they are on the map we gonna solve this issue Thanks for reporting 👍
3. If you find any other problem/bug or something i will appreciate any report Thanks for helping and for the photos

[drkludge]

@bogdan-telenav The long story short is that you are not handling an empty id_file.txt on first use. Here's how to recreate the problem.

1.You must become mortal to see the problem. 2.I had about 16 OSV OAuth settings. I removed all of those links in OpenStreetMap. 3.I moved the id file out of the way mv id_file.txt old_stuff_to_break_things.txt. You need to run the program as if you pulled it down from github for the first time. 4.Run it python upload_photos_by_exif.py -p 2015_12_07_08_08_43_316___Mapillary.

5. See Error Report One. The program is trying to read a nonexistent id_file.txt and does not handle the exception well. 6.Run it again with no success python upload_photos_by_exif.py -p 2015_12_07_08_08_43_316___Mapillary.
6. See Error Report Two. The same error was produced and still no upload.
7. Move the id_file.txt from step three back into place. mv old_stuff_to_break_things.txt id_file.txt. 9.Run it again python upload_photos_by_exif.py -p 2015_12_07_08_08_43_316___Mapillary. 10.See Success Report One and sequence 15113.

Yes there were a number of exif issues to fix but the id_file.txt issue was a pain until I created one manually.

## Error Report One. For login go to this URL in your browser: http://www.openstreetmap.org/oauth/authorize?oauth_token=... Login and grant acces then press ENTER

Traceback (most recent call last): File "upload_photos_by_exif.py", line 218, in main id_file = open("id_file.txt", "r+") FileNotFoundError: [Errno 2] No such file or directory: 'id_file.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 1240, in do_open h.request(req.get_method(), req.selector, req.data, headers) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/http/client.py", line 1083, in request self._send_request(method, url, body, headers) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/http/client.py", line 1128, in _send_request self.endheaders(body) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/http/client.py", line 1079, in endheaders self._send_output(message_body) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/http/client.py", line 911, in _send_output self.send(msg) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/http/client.py", line 854, in send self.connect() File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/http/client.py", line 1237, in connect server_hostname=server_hostname) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/ssl.py", line 376, in wrap_socket _context=self) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/ssl.py", line 747, in init self.do_handshake() File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/ssl.py", line 983, in do_handshake self._sslobj.do_handshake() File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/ssl.py", line 628, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:646)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "upload_photos_by_exif.py", line 441, in main(sys.argv[1:]) File "upload_photos_by_exif.py", line 268, in main opener.open(urllib.request.Request(authorize_url)) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 471, in open response = meth(req, response) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 581, in http_response 'http', request, response, code, msg, hdrs) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 503, in error result = self._call_chain(_args) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 443, in _call_chain result = func(_args) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 686, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 471, in open response = meth(req, response) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 581, in http_response 'http', request, response, code, msg, hdrs) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 503, in error result = self._call_chain(_args) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 443, in _call_chain result = func(_args) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 686, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 465, in open response = self._open(req, data) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 483, in _open '_open', req) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 443, in _call_chain result = func(*args) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 1283, in https_open context=self._context, check_hostname=self._check_hostname) File "/share/CACHEDEV1_DATA/.qpkg/Python3/src/lib/python3.5/urllib/request.py", line 1242, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:646)> [/share/mapillary/gis/data/osm_survey/mapillary] #

## Error Report Two. python upload_photos_by_exif.py -p 2015_12_07_08_08_43_316___Mapillary

For login go to this URL in your browser: http://www.openstreetmap.org/oauth/authorize?oauth_token=... Login and grant acces then press ENTER

Traceback (most recent call last): File "upload_photos_by_exif.py", line 218, in main id_file = open("id_file.txt", "r+") FileNotFoundError: [Errno 2] No such file or directory: 'id_file.txt'

During handling of the above exception, another exception occurred: ...

## Success Report One.

python upload_photos_by_exif.py -p 2015_12_07_08_08_43_316_Mapillary No sequence file existing Found 403 pictures to upload processing 2015_12_07_08_08_51_017.jpg Uploaded - 1 of total :403, percentage: 0.25% processing 2015_12_07_08_08_43_836.jpg Uploaded - 2 of total :403, percentage: 0.5% processing 2015_12_07_08_08_48_550.jpg Uploaded - 3 of total :403, percentage: 0.73999999999999999% processing 2015_12_07_08_08_46_151.jpg Uploaded - 4 of total :403, percentage: 0.98999999999999999% ... Uploaded - 399 of total :403, percentage: 99.010000000000005% processing 2015_12_07_08_32_19_519.jpg Uploaded - 400 of total :403, percentage: 99.260000000000005% processing 2015_12_07_08_32_21_900.jpg Uploaded - 401 of total :403, percentage: 99.5% processing 2015_12_07_08_32_26_850.jpg Uploaded - 402 of total :403, percentage: 99.75% processing 2015_12_07_08_32_24_335.jpg Uploaded - 403 of total :403, percentage: 100.0% Finish uploading form dir: /share/external/DEV3301_2/gis/data/osm_survey/mapillary/2015_12_07_08_08_43316Mapillary/ with sequence id: 15113

[bogdan-racasan]

@Xevib Do you still have problems with this script?

[Xevib]

@bogdan-telenav with the last update I'm having this message: Fail code:'osv' Fail to create the sequence Please restart the script

[bogdan-racasan]

@Xevib I changed the login methods ... can you try now. Ty for reporting

[Xevib]

@bogdan-telenav with a clean virtualenv I'm getting this error: Traceback (most recent call last): File "upload_photos_by_exif.py", line 9, in import http.cookiejar ImportError: No module named http.cookiejar

I supose that the module is not on the requirements.txt

[bogdan-racasan]

@Xevib i'm looking for this issue.

• Can you tell me what python.
• pip freeze in the terminal when you enter in virtualenv so you se the modules that you have instaled in virtuaenv

[Xevib]

@bogdan-telenav Pip freeze: Django==1.10.2 ExifRead==2.1.2 image==1.4.4 Pillow==3.4.1 rauth==0.7.2 requests==2.9.1 six==1.10.0

Python version: Python 2.7.12

[bogdan-racasan]

@Xevib So the problem is your version for Python. This script is only for Python 3.x i recomand Python 3.5, this is what i use for development. The script that upload images based on exif works only with python 3.x. The script that upload images created with OSV works only with python 2.x.

[Xevib]

Ok, i will try with Python 3

[bogdan-racasan]

Thanks for helping 👍 Try again and if you didn't do yet put the python script where all users have privileges (I think that can be an issue for creating some files that are used for cashing data). If theres a problem let me know.

[Xevib]

I tried with python3 and I'm having a problem with the version of Pillow on the requierements.txt that have a problem when installs

[bogdan-racasan]

@Xevib there could be a problem when you want to install on the system. I recommend to create an virtualenv. I see that problem reproduced when you try to install on the system. If the problem persist try to install the latest

[bogdans-telenav]

We made major changes on our upload scripts and we consider this issue addressed so i will close it. If you find any problem with the new version please create an issue and we will address it as soon as possible.