search

kartaview / upload-scripts

Uploader tools for KartaView
MIT License
61 stars 30 forks source link

Upload fails when over 2048 pictures to upload #1

Closed james2432 closed 8 years ago

james2432 commented 8 years ago

I have over 2700 photos in one sequence to upload and it fails as there are too many files open. Here are the logs. I'm running on windows with the upload_photos_by_exif3.py upload script.

[Errno 24] Too many open files: 'C:\pyscripts\openstreetview\photos2\1\virb0114-2702.jpg' [Errno 24] Too many open files: 'C:\pyscripts\openstreetview\photos2\1\virb0114-2703.jpg' [Errno 24] Too many open files: 'C:\pyscripts\openstreetview\photos2\1\virb0114-2704.jpg' [Errno 24] Too many open files: 'C:\pyscripts\openstreetview\photos2\1\virb0114-2705.jpg' [Errno 24] Too many open files: 'C:\pyscripts\openstreetview\photos2\1\virb0114-2706.jpg' [Errno 24] Too many open files: 'C:\pyscripts\openstreetview\photos2\1\virb0114-2707.jpg' [Errno 24] Too many open files: 'C:\pyscripts\openstreetview\photos2\1\virb0114-2708.jpg' [Errno 24] Too many open files: 'C:\pyscripts\openstreetview\photos2\1\virb0114-2709.jpg' [Errno 24] Too many open files: 'C:\pyscripts\openstreetview\photos2\1\virb0114-2710.jpg' [Errno 24] Too many open files: 'C:\pyscripts\openstreetview\photos2\1\virb0114-2711.jpg' [Errno 24] Too many open files: 'C:\pyscripts\openstreetview\photos2\1\virb0114-2712.jpg' Traceback (most recent call last): File "upload_photos_by_exif3.py", line 349, in main File "C:\Python34\lib\concurrent\futures_base.py", line 395, in result File "C:\Python34\lib\concurrent\futures_base.py", line 354, in __get_result File "C:\Python34\lib\concurrent\futures\thread.py", line 54, in run File "upload_photos_by_exif3.py", line 109, in upload_photos File "C:\Python34\lib\site-packages\requests\api.py", line 107, in post File "C:\Python34\lib\site-packages\requests\api.py", line 53, in request File "C:\Python34\lib\site-packages\requests\sessions.py", line 454, in request File "C:\Python34\lib\site-packages\requests\sessions.py", line 375, in prepare_request File "C:\Python34\lib\site-packages\requests\utils.py", line 95, in get_netrc_auth File "", line 2237, in _find_and_load File "", line 2226, in _find_and_load_unlocked File "", line 1200, in _load_unlocked File "", line 1129, in _exec File "", line 1467, in exec_module File "", line 1571, in get_code File "", line 1623, in get_data OSError: [Errno 24] Too many open files: 'C:\Python34\lib\netrc.py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "upload_photos_by_exif3.py", line 384, in File "upload_photos_by_exif3.py", line 369, in main TypeError: a float is required

bogdans-telenav commented 8 years ago

@james2432 thanks for reporting this. We managed yesterday to fix this and we will make everything available on this repo later this week.

james2432 commented 8 years ago

I found a work around in the mean time: to split the folder in to 2 and upload couple hundred photos(until the resulting remainder is ~2000) kill the python script, update the count file for the a bit less than the image count on the website, copy the remainder back into the folder and restart the script, obviously this is a lot of work, but I'm glad you fixed it

bogdan-racasan commented 8 years ago

@james2432 We solve the problem in a different and clean way so the script run flawless. I hope we can make it available in this week.

bogdan-racasan commented 8 years ago

@james2432 I fixed the bug that you reported. You can find the upload script in this repo. The path to file is : upload_photos_by_exif/upload_photo_by_exif.py