search

kartaview / openstreetcam.org

The openstreetcam.org web site
MIT License
75 stars 10 forks source link

You are not allowed to add a duplicate entry (sequenceIndex) #157

Closed Samhamsam closed 5 years ago

Samhamsam commented 6 years ago

I get this error a lot. It would be awesome if it get fixed.

Found 119 pictures to upload
 30%|████████████████████▎                                               36/119 remaining:38:42  elapsed:05:02multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/mypc/Dokumente/Programmieren/osc_mapillary/upload-scripts/upload_photos_by_exif/photo.py", line 79, in __call__
    self.make_upload(photo, data_photo)
  File "/home/mypc/Dokumente/Programmieren/osc_mapillary/upload-scripts/upload_photos_by_exif/photo.py", line 60, in make_upload
    if conn.json()['status']['apiMessage'] == ' You are not allowed to add a duplicate entry (sequenceIndex)':
  File "/home/mypc/Dokumente/Programmieren/osc_mapillary/upload-scripts/lib/python3.6/site-packages/requests/models.py", line 808, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "upload_exif.py", line 67, in <module>
    main()
  File "upload_exif.py", line 58, in main
    do_upload(thread, UploadPhoto(path, access_token, id_sequence, count_list, url_photo), photos_path)
  File "/home/mypc/Dokumente/Programmieren/osc_mapillary/upload-scripts/osc/utils.py", line 67, in do_upload
    list(tqdm.tqdm(p.imap(generator, payload), total=len(payload), bar_format='{l_bar}{bar} {n_fmt}/{total_fmt} remaining:{remaining}  elapsed:{elapsed}'))
  File "/home/mypc/Dokumente/Programmieren/osc_mapillary/upload-scripts/lib/python3.6/site-packages/tqdm/_tqdm.py", line 872, in __iter__
    for obj in iterable:
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
ToeBee commented 6 years ago

Probably the same thing as openstreetcam/upload-scripts#50. And really, this is probably the correct place to report it. This does seem to be a server side error, not an error in the upload script.

ToeBee commented 6 years ago

Ok, managed to get the real error happening here. It is a 502 "Bad Gateway" error being returned by Apache. It looks like you are using Apache to proxy requests to an application server. The application server is throwing an error and Apache doesn't know what to do with it so it is returning the default Apache 502 response. This is what the script is trying to parse into a JSON object:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a href="/1.0/photo/">POST&nbsp;/1.0/photo/</a></em>.<p>
Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>
ToeBee commented 6 years ago

Oh and the subject of this issue is kind of misleading. While the string You are not allowed to add a duplicate entry (sequenceIndex) is indeed shown in the middle of the traceback, it actually has nothing to do with the error itself. It is just printing the line of code where the error is happening which is a string comparison, looking for this string in the server response.

alexandrui-telenav commented 6 years ago

You're right about the proxy and the 502, @ToeBee. It's not entirely clear for us what exactly is happening on the server but we'll try to reproduce and investigate further. One thing that should probably be done in the scripts is a retry mechanism so that upload doesn't stop when an error occurs. The mobile apps are pretty solid in that department, the script not so much, it seems.

moenk commented 6 years ago

Thank you for pointing out the reason for this issue, I was wondering why the same setup used to work and now it does not any more. The only thing I changed in the meantime was switching to a faster internet connection. As it is a server problem a workaround may be to catch the exception, wait a bit and retry.

moenk commented 6 years ago

I did a dirty hack in the upload script, it's so bad I won't file a push for that, but it solves the problem for me. Replaced the do_upload line by this: success=False while not(success): try: success=True do_upload(thread, UploadPhoto(path, access_token, id_sequence, count_list, url_photo), photos_path) except: success=False pass

ToeBee commented 6 years ago

Ping on this issue. I am still receiving an HTTP 502 error every couple hundred pictures when I upload. The way the script recovers from failure also causes it to drop one or two images and sometimes mess up the ordering every time there is a failure and I have to restart the upload. I guess I should try @moenk's hack. I was concerned it might lead to re-introducing the infinite loop problem the script had before but looking at it again, it seems this is in a different location and should avoid that particular loop.

As the bicycle riding season heats up and I take more pictures with my GoPro, this is going to become a showstopper for contributing to OSC for me. I don't have time to babysit and restart uploads dozens of times because of constant server errors.

ToeBee commented 6 years ago

@moenk It looks like your hack causes the script to restart the upload from the first image every time there is an error. Although it doesn't create a new track. I started a 4,100 picture upload last night. It still isn't finished but the website now reports that there are over 7,000 pictures in the track after it experienced 11 upload errors. And for bonus points, it actually prevents me from CTRL-Cing the upload. I had to kill -9 the thing. :)

Samhamsam commented 6 years ago

Yes, it is like @ToeBee says. The code hack does not work.

danbjoseph commented 6 years ago

missed the existing issues and also logged this https://github.com/openstreetcam/upload-scripts/issues/51

@kudarisenmon also noted (on that issue) running into the same error

ToeBee commented 6 years ago

Sooo... this has been an ongoing production server problem for 2 months now and I have probably 50,000 images in my upload backlog. Any news? You talk about not being able to reproduce it. It happens to me every time I upload, usually less than 100 pictures into an upload. Put me in direct contact with someone who can access production server logs and I will ping them as soon as I have a failure.

alexandrui-telenav commented 6 years ago

@adriann-telenav just finished re-implementing the upload/processing/file access components of OSC ( we all knew about this big problem but was almost impossible to pin-point/fix, most likely was due to a third party framework ) Currently deployed on testing server and things look good, 0 error rate so far. Testing some more this week then production follows. Hopefully we can put this mess behind us very soon.

danbjoseph commented 6 years ago

@alexandrui-telenav @adriann-telenav can you please post an update here when the re-implementation is live?

ToeBee commented 6 years ago

According to Slack, the back end rework is now live. However this upload problem is still present. I had 3 server crashes while uploading a 1,437 image track this morning.

ToeBee commented 6 years ago

However... I will say that the problem is slightly better now. It was crashing every 50-100 photos. Now I am getting up to 500 or 600 at a time. Having to restart the upload 3-4 times instead of 10+ times for my tracks is bearable so I am now working through some of my upload backlog

adriann-telenav commented 6 years ago

Hello @ToeBee,

We are working now on both processing and API, but the migration is step by step, since we get huge amount of photos every day, we need to be sure that we process correctly everything which is new. We also started to reprocess old data in small chunks, and we hope to reprocess everything this summer.

The servers are at full load every minute now, and will be the same for the next 4 days until we finish current queues.

So there might be requests with bad response from our servers on upload, mostly because the small timeout setup on request. Actually the files are uploading correctly even if you might receive bad response. Bad response might come also from apache/nginx services(because they cannot manage so many requests at a time).

You can check if the file was uploaded properly by trying to upload it again with the same sequence_index. If you receive "duplicate" error, then everything went well, so you can go to the next file.

DB entry will not be created if there was a problem with the file upload.

Uploading scripts needs to be reviewed and re-implemented. They were made for legacy APIv1 upload(which had many issues). Now we are moving step by step to APIv2. Upload has a V1 wrapper for V2 code base. You will still receive V1 compatible response, but the logic in the backend is the new one, which logs every found problem.

From what I know upload scripts are using multiple parallel uploads, which might not work 100% ok when bandwidth is full. They need to have limited threads or to be converted to sequential upload.

We are still working on APIv2. When we will fully release it we will convert all tools to use it. Final V2 will be a new standalone project and will have better error management features. We cannot make too many critical changes to V1 because we might break legacy tools.

Best regards, Adrian

ToeBee commented 6 years ago

There are some photos being dropped when these errors happen. For example the sequence I just finished uploading has 1,441 images on my hard drive but the completed track on the OSC website says 1,438. But I don't know if that is the server or the script's fault. I will try the next upload with a single thread and see how that affects things. Hopefully I can limp along until this v2 rollout.

danbjoseph commented 6 years ago

@adriann-telenav, when you release v2 and the updated tools, will there be a way to split sequences and remove duplicates (of geotagged images taken with an action camera)?

james2432 commented 6 years ago

@danbjoseph I've already implemented duplicate elimination in my GUI tool if you dont mind preprocessing them: https://github.com/osmottawa/OSVUploadr/releases

ToeBee commented 6 years ago

Yeah... looks like the server crashes are load dependent. Over the past day I've had over 1,000 go through without an error or as few as 50. The good news is that dropping the thread count down to 1 does indeed fix the problem with photos being dropped when errors happen. There must be a threading problem in the script where it writes out to the count_file.txt from a thread after another thread has crashed out or something like that.

rwelty1889 commented 6 years ago

what is the current status of this issue? i just saw this error come up in my second "successful" use of the script, and am uncertain how to proceed.

rwelty1889 commented 6 years ago

does the existing client side script return proper posix return codes for success and failure? if so, i have a script that might band aid this problem for now.

rwelty1889 commented 6 years ago

script works for the simple failure but fails when the infinite retry loop occurs.

bogdans-telenav commented 5 years ago

@rwelty1889 , @ToeBee , @Samhamsam, @danbjoseph , @moenk we updated the upload scripts and this issue should no longer block you uploads. The bug on our server most likely is still there but we now work around it properly on the scripts side. I will close this issue if you feel like it is still reproducing with the new scripts please create an issue on the upload_scripts repo.