search

mapillary / mapillary_tools

Command line tools for processing and uploading Mapillary imagery
BSD 2-Clause "Simplified" License
267 stars 138 forks source link

[BUG] Interrupted uploads do not resume #569

Open JakeSmarter opened 2 years ago

JakeSmarter commented 2 years ago

Basic information

Steps to reproduce behavior

  1. Start uploading a bunch of images, preferably like gigabytes per sequence.
  2. Wait for the server to choke or deliberately throttle your network bandwidth, potentially down to 0. Pressing Ctrl+C and then resuming (while the upload session is still alive on the server) with exactly the same command should have the same effect.
  3. Observe that either the server or the client resumes at 'offset': 0.

It is unclear whether this is a client or server bug. My gut feeling tells me that it is a server bug but I may be wrong. The server may change the chunk size during upload depending on load. The client should adapt to this. The server should also always respond with the correct offset, independent of the current chunk size.

Expected behavior

Per sequence uploads should resume on the offset of the first incomplete chunk.

Actual behavior

2022-10-13 14:11:23,512 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 0, 'offset': 0, 'retries': 0, 'upload_last_restart_time': 1665663083.5125833, 'upload_first_offset': 0}
2022-10-13 14:11:37,891 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 0, 'offset': 3792867, 'retries': 0, 'upload_last_restart_time': 1665663083.5125833, 'upload_first_offset': 0, 'chunk_size': 3792867}
2022-10-13 14:11:53,101 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 0, 'offset': 7585734, 'retries': 0, 'upload_last_restart_time': 1665663083.5125833, 'upload_first_offset': 0, 'chunk_size': 3792867}
2022-10-13 14:12:08,341 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 0, 'offset': 11378601, 'retries': 0, 'upload_last_restart_time': 1665663083.5125833, 'upload_first_offset': 0, 'chunk_size': 3792867}
2022-10-13 14:12:22,579 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 0, 'offset': 15171468, 'retries': 0, 'upload_last_restart_time': 1665663083.5125833, 'upload_first_offset': 0, 'chunk_size': 3792867}
2022-10-13 14:12:25,712 - WARNING - Error uploading chunk_size 3792867 at offset 0: HTTPError: 412 Client Error: Precondition Failed for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_c043f21e7f5d2f4ab25c1ba5099bf5a7.zip
2022-10-13 14:12:25,712 - INFO    - Retrying in 2 seconds (1/200)
2022-10-13 14:12:28,367 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 62.19941592216492, 'offset': 0, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665663148.3675609}

Note that 'offset': 0, while it should have been 'offset': 15171468 because this is the offset of the first incomplete chunk.

All of the above means that it is currently extremely difficult to upload large sequences on low bandwidth connections because as soon as the load on the server changes (which is very likely on long uploads), the chunk size changes and the client has to restart uploading from offset 0. This is a huge waste of resources!

JakeSmarter commented 2 years ago

:man_facepalming: I have just spent over 4 hours uploading a sequence of 163 images worth 618,237,384 bytes. mapillary_tools is still not done because it is constantly resetting to either 0 or an arbitrary number. This is something that should have been completed in about 5 minutes. And, before you ask: no, my connection is stable.

You see, this is exactly what I mean by “Mapillary being dysfunctional”; every upload tool from Mapillary is completely useless and unusable. Or, a horror to use and nobody seems to care. It is almost offending if you look at the amount of hoops I have to jump through in order to just contribute some meager amount of images voluntarily. Just because I am geek I have the patience and skill to deal with it. But, how do you expect any non‑technical people to be willing to contribute via this mess?

ptpt commented 2 years ago

Thanks for reporting and sorry for the inconvenience.

This looks suspicious indeed. It seems the server returned offset=0 regardless it had data chunk uploaded there.

Could you check if it work without a proxy?

I can't reproduce it locally:

2022-10-13 10:43:20,649 - DEBUG   - Sending upload_start via IPC: {'total_sequence_count': 4, 'sequence_idx': 0, 'file_type': 'gopro', 'entity_size': 3307582003, 'import_path': 'GoProMax/100GOPRO/GH013770.MP4', 'md5sum': '1e13989bbf76d8cab1eed147c7193fe3', 'upload_start_time': 1665683000.649157, 'upload_total_time': 0}
2022-10-13 10:43:21,049 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 4, 'sequence_idx': 0, 'file_type': 'gopro', 'entity_size': 3307582003, 'import_path': 'GoProMax/100GOPRO/GH013770.MP4', 'md5sum': '1e13989bbf76d8cab1eed147c7193fe3', 'upload_start_time': 1665683000.649157, 'upload_total_time': 0, 'offset': 20340672, 'retries': 0, 'upload_last_restart_time': 1665683001.049864, 'upload_first_offset': 20340672}
2022-10-13 10:43:28,298 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 4, 'sequence_idx': 0, 'file_type': 'gopro', 'entity_size': 3307582003, 'import_path': 'GoProMax/100GOPRO/GH013770.MP4', 'md5sum': '1e13989bbf76d8cab1eed147c7193fe3', 'upload_start_time': 1665683000.649157, 'upload_total_time': 0, 'offset': 37117888, 'retries': 0, 'upload_last_restart_time': 1665683001.049864, 'upload_first_offset': 20340672, 'chunk_size': 16777216}
2022-10-13 10:43:44,106 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 4, 'sequence_idx': 0, 'file_type': 'gopro', 'entity_size': 3307582003, 'import_path': 'GoProMax/100GOPRO/GH013770.MP4', 'md5sum': '1e13989bbf76d8cab1eed147c7193fe3', 'upload_start_time': 1665683000.649157, 'upload_total_time': 0, 'offset': 53895104, 'retries': 0, 'upload_last_restart_time': 1665683001.049864, 'upload_first_offset': 20340672, 'chunk_size': 16777216}

2022-10-13 10:44:05,663 - WARNING - Error uploading chunk_size 16777216 at offset 20340672: ConnectionError: ('Connection aborted.', OSError(65, 'No route to host'))
2022-10-13 10:44:05,664 - INFO    - Retrying in 2 seconds (1/200)
2022-10-13 10:44:07,682 - WARNING - Error uploading chunk_size 16777216 at offset 20340672: ConnectionError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_1e13989bbf76d8cab1eed147c7193fe3.mp4 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10af3f550>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
2022-10-13 10:44:07,682 - INFO    - Retrying in 4 seconds (2/200)
2022-10-13 10:44:11,691 - WARNING - Error uploading chunk_size 16777216 at offset 20340672: ConnectionError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_1e13989bbf76d8cab1eed147c7193fe3.mp4 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10ac28b50>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
2022-10-13 10:44:11,691 - INFO    - Retrying in 8 seconds (3/200)
2022-10-13 10:44:19,704 - WARNING - Error uploading chunk_size 16777216 at offset 20340672: ConnectionError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_1e13989bbf76d8cab1eed147c7193fe3.mp4 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10ac28b80>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
2022-10-13 10:44:19,704 - INFO    - Retrying in 16 seconds (4/200)
2022-10-13 10:44:36,045 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 4, 'sequence_idx': 0, 'file_type': 'gopro', 'entity_size': 3307582003, 'import_path': 'GoProMax/100GOPRO/GH013770.MP4', 'md5sum': '1e13989bbf76d8cab1eed147c7193fe3', 'upload_start_time': 1665683000.649157, 'upload_total_time': 44.612921953201294, 'offset': 58990528, 'retries': 4, 'upload_first_offset': 20340672, 'chunk_size': 16777216, 'upload_last_restart_time': 1665683076.0457752}
2022-10-13 10:44:43,132 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 4, 'sequence_idx': 0, 'file_type': 'gopro', 'entity_size': 3307582003, 'import_path': 'GoProMax/100GOPRO/GH013770.MP4', 'md5sum': '1e13989bbf76d8cab1eed147c7193fe3', 'upload_start_time': 1665683000.649157, 'upload_total_time': 44.612921953201294, 'offset': 75767744, 'retries': 4, 'upload_first_offset': 20340672, 'chunk_size': 16777216, 'upload_last_restart_time': 1665683076.0457752}
bob3bob3 commented 2 years ago

If either of you need a reference point (debug logs) to work with I have been using the tools to upload BlackVue videos 24/7 for 2 weeks now. Something like 870GBytes, 180,000 mp4's so far that seems to be working okay. I have not been watching the logs closely, but only have maybe 20 of the ConnectionError similar to above.

No doubt that I am uploading mp4's rather than jpg's has a lesser chance of failure,

Drop me a note if you would like any analysis or dump,

JakeSmarter commented 2 years ago

More logs…

2022-10-13 20:31:10,269 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22126.903530597687, 'offset': 428593971, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685613.4696445}
2022-10-13 20:31:11,338 - WARNING - Error uploading chunk_size 3792867 at offset 360322365: ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2022-10-13 20:31:11,339 - INFO    - Retrying in 2 seconds (1/200)
2022-10-13 20:31:16,334 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22384.77271795273, 'offset': 371700966, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685876.3342628}
2022-10-13 20:31:31,591 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22384.77271795273, 'offset': 375493833, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685876.3342628}
2022-10-13 20:31:46,339 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22384.77271795273, 'offset': 379286700, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685876.3342628}
2022-10-13 20:32:01,102 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22384.77271795273, 'offset': 383079567, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685876.3342628}
2022-10-13 20:32:26,992 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22384.77271795273, 'offset': 386872434, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685876.3342628}
2022-10-13 20:32:37,634 - WARNING - Error uploading chunk_size 3792867 at offset 371700966: HTTPError: 412 Client Error: Precondition Failed for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_c043f21e7f5d2f4ab25c1ba5099bf5a7.zip
2022-10-13 20:32:37,634 - INFO    - Retrying in 2 seconds (1/200)
2022-10-13 20:32:44,441 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 375493833, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:33:11,953 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 379286700, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:33:28,227 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 383079567, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:33:46,208 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 386872434, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:34:03,132 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 390665301, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:34:17,073 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 394458168, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:34:30,741 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 398251035, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:34:45,439 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 402043902, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:34:59,471 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 405836769, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:35:14,861 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 409629636, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:35:31,725 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 413422503, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:35:46,345 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 417215370, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:35:59,818 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 421008237, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:36:18,185 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 424801104, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:36:32,820 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 428593971, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:36:47,286 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 432386838, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:37:04,628 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 436179705, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:37:26,269 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 439972572, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:37:47,220 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 443765439, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:38:11,899 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 447558306, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:38:31,960 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 451351173, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:38:47,581 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 455144040, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:39:03,287 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 458936907, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:39:26,043 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 462729774, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:39:44,205 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 466522641, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:39:57,682 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 470315508, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:40:11,354 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 474108375, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:40:24,670 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 477901242, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:40:39,238 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 481694109, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:40:40,743 - WARNING - Error uploading chunk_size 3792867 at offset 375493833: HTTPError: 412 Client Error: Precondition Failed for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_c043f21e7f5d2f4ab25c1ba5099bf5a7.zip
2022-10-13 20:40:40,743 - INFO    - Retrying in 2 seconds (1/200)
2022-10-13 20:40:44,878 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22942.374705791473, 'offset': 364115232, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665686444.8789496}
2022-10-13 20:41:17,740 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22942.374705791473, 'offset': 367908099, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665686444.8789496}
2022-10-13 20:42:09,556 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22942.374705791473, 'offset': 371700966, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665686444.8789496}

Could you check if it work without a proxy?

Sure, this may be proxy related. However, I highly doubt it. Too me, the server looks like the most probable culprit because the offset is computed by the server. Unfortunately, I cannot bypass the proxy. Anyhow, proxy or not it should work because I cannot imagine a scenario where the server should respond a wrong offset over a proxy or the client to modify the offset. It is the server that keeps track of the upload session state. Btw, I have never had any issues with other online services accepting large (interrupted) uploads.

I am not sure how you could reproduce this. But, it seems like these recurring messages are usually causing trouble:

WARNING - Error uploading chunk_size 3792867 at offset 360322365: ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
WARNING - Error uploading chunk_size 3792867 at offset 371700966: HTTPError: 412 Client Error: Precondition Failed for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_c043f21e7f5d2f4ab25c1ba5099bf5a7.zip
ptpt commented 2 years ago

@GITNE I suspect it's related to HTTP status 412 error, curious to see the payload.

I'm trying to add a check on retires: if the offset fetched from the server does not move as expected, exit with full HTTP response printed out.

I will make a new alpha release and let you know soon.

JakeSmarter commented 2 years ago

I will make a new alpha release and let you know soon.

Great, thank you! :smile:

Just FYI: I have observed this behavior ever since I have started using version 0.8.0.

ptpt commented 2 years ago

Just a quick check @GITNE: were you uploading this sequence in multiple processes/machines simultaneously? That might cause the offset to be inconsistent AFICS.

JakeSmarter commented 2 years ago

Just a quick check @GITNE: were you uploading this sequence in multiple processes/machines simultaneously?

No, single instance on one machine only.

ptpt commented 2 years ago

The fix is released here https://github.com/mapillary/mapillary_tools/releases/tag/v0.9.5a1

BTW it would be great if you could setup a local env here https://github.com/mapillary/mapillary_tools#development so we can test on branches without releasing binaries (faster iteration).

JakeSmarter commented 2 years ago

This is what I got on my first attempt with v0.9.5a1:

2022-10-14 12:25:44,228 - DEBUG   - CLI param: verbose: True
2022-10-14 12:25:44,228 - DEBUG   - CLI param: import_path: [PosixPath('mapillary_public_uploads')]
2022-10-14 12:25:44,228 - DEBUG   - CLI param: dry_run: False
2022-10-14 12:25:44,272 - INFO    - File mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip has been uploaded already. Check the upload history at /home/GITNE/.local/share/mapillary_tools/upload_history/c0/43f21e7f5d2f4ab25c1ba5099bf5a7.json
2022-10-14 12:25:44,273 - DEBUG   - Uploaded to cluster: None
2022-10-14 12:25:44,307 - DEBUG   - Sending upload_start via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0}
2022-10-14 12:25:46,299 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 39330460, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460}
2022-10-14 12:26:05,397 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 42355880, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:26:24,231 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 45381300, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:26:35,169 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 48406720, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:26:46,149 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 51432140, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:26:57,397 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 54457560, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:27:07,733 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 57482980, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:27:28,181 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 60508400, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:27:47,437 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 63533820, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:28:06,483 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 66559240, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:28:24,166 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 69584660, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:28:41,148 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 72610080, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:29:00,671 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 75635500, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:29:11,399 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 78660920, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:29:22,306 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 81686340, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:29:32,839 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 84711760, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:29:45,819 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 87737180, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:30:01,497 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 90762600, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:30:12,107 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 93788020, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:30:22,642 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 96813440, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:30:32,824 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 99838860, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:30:42,983 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 102864280, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:30:53,270 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 105889700, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:31:04,135 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 108915120, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:31:14,425 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 111940540, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:31:26,219 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 114965960, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:31:37,200 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 117991380, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:31:48,100 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 121016800, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:31:58,424 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 124042220, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:32:09,483 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 127067640, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:32:20,301 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 130093060, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:32:30,732 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 133118480, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:32:41,610 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 136143900, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:32:52,026 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 139169320, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:33:03,374 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 142194740, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:33:14,322 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 145220160, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:33:30,316 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 148245580, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:33:45,847 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 151271000, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:33:56,210 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 154296420, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:34:06,474 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 157321840, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:34:18,293 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 160347260, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:34:30,022 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 163372680, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:34:40,938 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 166398100, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:34:51,999 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 169423520, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:35:02,358 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 172448940, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:35:15,365 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 175474360, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:35:26,466 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 178499780, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:35:38,630 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 181525200, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:35:55,036 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 184550620, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:36:02,285 - CRITICAL - Upload server offset 69584660 does not match expected offset 184550620
2022-10-14 12:36:02,286 - DEBUG   - API Logging for action upload_failed_upload: {'images': 0, 'sequences': 0, 'size': 0.0, 'uploaded_size': 0.0, 'speed': 0, 'time': 0}
Traceback (most recent call last):
  File "mapillary_tools/uploader.py", line 488, in _upload_fp
  File "mapillary_tools/upload_api_v4.py", line 128, in upload
  File "requests/models.py", line 1021, in raise_for_status
requests.exceptions.HTTPError: 412 Client Error: Precondition Failed for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_3a59c68825c203b5bbf22154e4802238.zip

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "mapillary_tools/upload.py", line 785, in _upload_zipfiles
  File "mapillary_tools/uploader.py", line 140, in upload_zipfile
  File "mapillary_tools/uploader.py", line 362, in _upload_zipfile_fp
  File "mapillary_tools/uploader.py", line 511, in _upload_fp
mapillary_tools.upload_api_v4.UploadHTTPError: POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_3a59c68825c203b5bbf22154e4802238.zip
> HTTP Status: 412
{"debug_info":{"retriable":true,"type":"OffsetInvalidError","message":"Request starting offset is invalid"}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "main.py", line 8, in <module>
  File "mapillary_tools/commands/__main__.py", line 141, in main
  File "mapillary_tools/commands/upload_zip.py", line 32, in run
  File "mapillary_tools/upload.py", line 623, in upload
  File "mapillary_tools/upload.py", line 618, in upload
  File "mapillary_tools/upload.py", line 789, in _upload_zipfiles
mapillary_tools.upload.UploadError: POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_3a59c68825c203b5bbf22154e4802238.zip
> HTTP Status: 412
{"debug_info":{"retriable":true,"type":"OffsetInvalidError","message":"Request starting offset is invalid"}}
[837964] Failed to execute script 'main' due to unhandled exception!

Next, I am simply going to retry and see how far it will go.

BTW it would be great if you could setup a local env here https://github.com/mapillary/mapillary_tools#development so we can test on branches without releasing binaries (faster iteration).

I will see what I can do. My best bet is to build a Flatpak because it provides a defined set of dependencies. This is going to help get reproducible results. :crossed_fingers:

JakeSmarter commented 2 years ago

After retry I got this:

2022-10-14 12:48:07,229 - DEBUG   - CLI param: verbose: True
2022-10-14 12:48:07,229 - DEBUG   - CLI param: import_path: [PosixPath('mapillary_public_uploads')]
2022-10-14 12:48:07,229 - DEBUG   - CLI param: dry_run: False
2022-10-14 12:48:07,263 - INFO    - File mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip has been uploaded already. Check the upload history at /home/GITNE/.local/share/mapillary_tools/upload_history/c0/43f21e7f5d2f4ab25c1ba5099bf5a7.json
2022-10-14 12:48:07,264 - DEBUG   - Uploaded to cluster: None
2022-10-14 12:48:07,291 - DEBUG   - Sending upload_start via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0}
2022-10-14 12:48:10,780 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 18152520, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520}
2022-10-14 12:49:30,090 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 21177940, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:50:18,054 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 24203360, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:50:48,944 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 27228780, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:51:20,380 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 30254200, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:51:34,717 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 33279620, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:51:45,598 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 36305040, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:52:00,109 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 39330460, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:52:15,236 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 42355880, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:52:28,474 - CRITICAL - Upload server offset 39330460 does not match expected offset 42355880
2022-10-14 12:52:28,474 - DEBUG   - API Logging for action upload_failed_upload: {'images': 0, 'sequences': 0, 'size': 0.0, 'uploaded_size': 0.0, 'speed': 0, 'time': 0}
Traceback (most recent call last):
  File "mapillary_tools/uploader.py", line 488, in _upload_fp
  File "mapillary_tools/upload_api_v4.py", line 128, in upload
  File "requests/models.py", line 1021, in raise_for_status
requests.exceptions.HTTPError: 412 Client Error: Precondition Failed for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_3a59c68825c203b5bbf22154e4802238.zip

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "mapillary_tools/upload.py", line 785, in _upload_zipfiles
  File "mapillary_tools/uploader.py", line 140, in upload_zipfile
  File "mapillary_tools/uploader.py", line 362, in _upload_zipfile_fp
  File "mapillary_tools/uploader.py", line 511, in _upload_fp
mapillary_tools.upload_api_v4.UploadHTTPError: POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_3a59c68825c203b5bbf22154e4802238.zip
> HTTP Status: 412
{"debug_info":{"retriable":true,"type":"OffsetInvalidError","message":"Request starting offset is invalid"}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "main.py", line 8, in <module>
  File "mapillary_tools/commands/__main__.py", line 141, in main
  File "mapillary_tools/commands/upload_zip.py", line 32, in run
  File "mapillary_tools/upload.py", line 623, in upload
  File "mapillary_tools/upload.py", line 618, in upload
  File "mapillary_tools/upload.py", line 789, in _upload_zipfiles
mapillary_tools.upload.UploadError: POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_3a59c68825c203b5bbf22154e4802238.zip
> HTTP Status: 412
{"debug_info":{"retriable":true,"type":"OffsetInvalidError","message":"Request starting offset is invalid"}}
[838207] Failed to execute script 'main' due to unhandled exception!

The offset is apparently still incorrect on retry. The last offset was 184,550,620 and on retry it was 18,152,520. :confused: That’s like a magnitude lower! :astonished:

ptpt commented 2 years ago

It seems that you were uploading zipfiles from mapillary_public_uploads which is the default location for the dry run upload testing. I think it's fine.

What looks really strange though, is that the embedded md5sum in the zip filename mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip is not same as the calculated md5sum 3a59c68825c203b5bbf22154e4802238 in the log. This makes me suspect if there is anything wrong with the md5sum calculation, so multiple zipfiles (with different content) got the same md5sum and uploaded to the same endpoint, therefore the HTTP 412 error returned.

I can't reproduce this issue neither unfortunately:

# generate zipfiles in mapillary_public_uploads
python3 -m mapillary_tools.commands --verbose process_and_upload ~/ImageTestData/myimages/ --dry_run

# upload these zipfiles
python3 -m mapillary_tools.commands --verbose upload_zip ./mapillary_public_uploads

# Here is one of the logs:
# as you can see the md5sum in the filename and the log match
2022-10-14 10:30:25,780 - DEBUG   - Sending upload_end via IPC: {'total_sequence_count': 3, 'sequence_idx': 2, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_6bfe61d03070c89f5d6fccdced76f0c7.zip', 'sequence_image_count': 2, 'entity_size': 6746255, 'md5sum': '6bfe61d03070c89f5d6fccdced76f0c7', 'upload_start_time': 1665768624.169763, 'upload_total_time': 1.478506088256836, 'offset': 6746255, 'retries': 0, 'upload_last_restart_time': 1665768624.301506, 'upload_first_offset': 0, 'chunk_size': 0, 'upload_end_time': 1665768625.7800121}

@GITNE would you mind share us one of these zipfile for debugging via support _at_ mapillary.zendesk.com?

ptpt commented 2 years ago

A bit context of how the md5sum is calculated for zipfiles (all images will be zipped before being uploaded). Assume we are uploading N images as a sequence:

1. calculate md5sum(image_content) for each image file as the `image_md5sum` (before updating image EXIF)
2. add images to the zip in the capture time order
3. name the image files in the zip as `{image_md5sum}.jpg`
4. concatenate all image md5sums, and calculate md5sum(concatenated_image_md5sums) as the zipfile_md5sum

Once we got the zipfile_md5sum, the zipfile will be uploaded to rupload.facebook.com/mly_tools_{zipfile_md5sum}.zip.

JakeSmarter commented 2 years ago

I think, I may have solved the mystery behind those resetting offsets. And, it definitely IS server related/caused.

It turns out that our ISP uses dynamic public IP addresses. Whenever the ISP shifts to a new public IP address the upload server counts this as a new upload session. Naturally, our proxy sits on the local network and thus is also affected by the public IP address change. So, apparently the upload server’s upload session is tied to a specific client (public) IP address. Well, lets just say that this is a bit sub‑optimal. Creating and assigning an upload session token free of any IP address per upload session is probably going to fix this issue and other similar scenarios where the upload client has no control over its public IP address. Migrating to an upload session token will not pose a security threat either because not only is this a widely adopted and commonly used method for authenticating an upload session but also a very secure method because the token is (only) a session secret with a very limited lifetime. The only thing the server has to be weary of is creating too many tokens per IP address and per unit of time. Fortunately, because of the intrinsic nature of a lifetime limited token the upload server does not need to store or keep track of issued tokens. Thus, you can implement load balancing and flooding protection in one go.

I have tested this hypothesis by rerouting network traffic over a static public IP address node and I did not have a single interruption over GBs of data. So after all, I guess my gut feeling was right.

ptpt commented 2 years ago

Very interesting observation. Thanks for sharing.

Whenever the ISP shifts to a new public IP address the upload server counts this as a new upload session.

The upload session is decided by the authorized user and the session_key in the url. I don't think IP changes upload sessions. If you upload a single file from IP A, then interrupt it, and then resume from IP B, it will result in the single file uploaded in the same bucket, instead of two buckets.

Changing IP during uploading will interrupt and close the underlying TCP connection, which result in the data chunk partially uploaded. In this case the HTTP client should raise a network error, and retry with the new offset fetched from the server. However, what we observed from the logs is that the HTTP didn't get interrupted and the server even returned success (2xx), then the client continued uploading a few more chunks based on the calculated (expected) offset, until the HTTP 412 error responded.

Could you run the test cli with/without static public IP address, and see what happens (especially the HTTP responses)?

# Generate 2GB file
dd if=/dev/zero of=TEST_BIG_FILE count=4024000
# Upload it
# NOTE: The test cli does upload `TEST_BIG_FILE` to the server, but they won't be shown on your profile, and it will be deleted after a few days.
python3 -m tests.cli.upload_api_v4 --verbose TEST_BIG_FILE TEST_SESSION --user_name=YOUR_ACCOUNT
JakeSmarter commented 2 years ago

Sorry for the long delay. Unfortunately, I was :sneezing_face: incapacitated for the last week (no COVID). Anyway, I have run the test you have suggested, as usual over our proxy, no rerouting:

2022-10-24 19:40:39,782 - INFO    - Session key: mly_tools_test_a981130cf2b7e09f4686dc273cf7187e
2022-10-24 19:40:39,783 - INFO    - Entity size: 2147483648
2022-10-24 19:40:39,783 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_test_a981130cf2b7e09f4686dc273cf7187e
2022-10-24 19:40:41,312 - DEBUG   - HTTP response 200: b'{"dc":"lla2c16","offset":10076160}'
2022-10-24 19:40:41,313 - INFO    - Initial offset: 10076160
2022-10-24 19:40:41,313 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_test_a981130cf2b7e09f4686dc273cf7187e
2022-10-24 19:40:43,019 - DEBUG   - HTTP response 200: b'{"dc":"lla2c16","offset":10076160}'
2022-10-24 19:40:43,027 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_test_a981130cf2b7e09f4686dc273cf7187e
2022-10-24 19:40:44,269 - DEBUG   - HTTP response 200: b'{"dc":"lla2c16","offset":10076160}'
2022-10-24 19:40:44,294 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_test_a981130cf2b7e09f4686dc273cf7187e HEADERS {"Offset": "10076160", "X-Entity-Length": "2147483648", "X-Entity-Name": "mly_tools_test_a981130cf2b7e09f4686dc273cf7187e", "X-Entity-Type": "application/zip"}
2022-10-24 19:42:23,177 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-24 19:42:23,178 - DEBUG   - The next offset will be: 26853376
2022-10-24 19:42:23,203 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_test_a981130cf2b7e09f4686dc273cf7187e HEADERS {"Offset": "26853376", "X-Entity-Length": "2147483648", "X-Entity-Name": "mly_tools_test_a981130cf2b7e09f4686dc273cf7187e", "X-Entity-Type": "application/zip"}
2022-10-24 19:43:42,860 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-24 19:43:42,860 - DEBUG   - The next offset will be: 43630592
2022-10-24 19:43:42,870 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_test_a981130cf2b7e09f4686dc273cf7187e HEADERS {"Offset": "43630592", "X-Entity-Length": "2147483648", "X-Entity-Name": "mly_tools_test_a981130cf2b7e09f4686dc273cf7187e", "X-Entity-Type": "application/zip"}
2022-10-24 19:45:11,429 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-24 19:45:11,429 - DEBUG   - The next offset will be: 60407808
2022-10-24 19:45:11,438 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_test_a981130cf2b7e09f4686dc273cf7187e HEADERS {"Offset": "60407808", "X-Entity-Length": "2147483648", "X-Entity-Name": "mly_tools_test_a981130cf2b7e09f4686dc273cf7187e", "X-Entity-Type": "application/zip"}
2022-10-24 19:47:05,818 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-24 19:47:05,819 - DEBUG   - The next offset will be: 77185024
2022-10-24 19:47:05,829 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_test_a981130cf2b7e09f4686dc273cf7187e HEADERS {"Offset": "77185024", "X-Entity-Length": "2147483648", "X-Entity-Name": "mly_tools_test_a981130cf2b7e09f4686dc273cf7187e", "X-Entity-Type": "application/zip"}
2022-10-24 19:48:21,719 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-24 19:48:21,720 - DEBUG   - The next offset will be: 93962240
2022-10-24 19:48:21,740 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_test_a981130cf2b7e09f4686dc273cf7187e HEADERS {"Offset": "93962240", "X-Entity-Length": "2147483648", "X-Entity-Name": "mly_tools_test_a981130cf2b7e09f4686dc273cf7187e", "X-Entity-Type": "application/zip"}
2022-10-24 19:50:23,689 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-24 19:50:23,689 - DEBUG   - The next offset will be: 110739456
2022-10-24 19:50:23,699 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_test_a981130cf2b7e09f4686dc273cf7187e HEADERS {"Offset": "110739456", "X-Entity-Length": "2147483648", "X-Entity-Name": "mly_tools_test_a981130cf2b7e09f4686dc273cf7187e", "X-Entity-Type": "application/zip"}
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1073, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.9/http/client.py", line 995, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.9/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.9/ssl.py", line 1173, in send
    return self._sslobj.write(data)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.9/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3.9/site-packages/urllib3/packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1073, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.9/http/client.py", line 995, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.9/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.9/ssl.py", line 1173, in send
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 98, in <module>
    main()
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 90, in main
    file_handle = service.upload(fp)
  File "/home/GITNE/src/GitHub/Mapillary tools/mapillary_tools/upload_api_v4.py", line 150, in upload
    resp = requests.post(
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Well, and then it crashed. I am not sure this is intended? Anyway, I hope this output is going to help you figure out what you wanted to know.

JakeSmarter commented 2 years ago

So, I have run the test with rerouting. Btw, whenever you see a traceback in the log it means that I have manually resumed uploading, just so that it gets finished.

2022-10-25 20:13:30,932 - INFO    - Session key: TEST_SESSION
2022-10-25 20:13:30,932 - INFO    - Entity size: 2147483648
2022-10-25 20:13:30,932 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-10-25 20:13:32,042 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":0}'
2022-10-25 20:13:32,043 - INFO    - Initial offset: 0
2022-10-25 20:13:32,043 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-10-25 20:13:32,855 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":0}'
2022-10-25 20:13:32,865 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-10-25 20:13:33,945 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":0}'
2022-10-25 20:13:33,975 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "0", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:14:54,573 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:14:54,574 - DEBUG   - The next offset will be: 16777216
2022-10-25 20:14:54,600 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "16777216", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:18:25,833 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:18:25,833 - DEBUG   - The next offset will be: 33554432
2022-10-25 20:18:25,843 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "33554432", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:20:05,879 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:20:05,879 - DEBUG   - The next offset will be: 50331648
2022-10-25 20:20:05,889 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "50331648", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:21:34,866 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:21:34,867 - DEBUG   - The next offset will be: 67108864
2022-10-25 20:21:34,877 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "67108864", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:22:30,900 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:22:30,901 - DEBUG   - The next offset will be: 83886080
2022-10-25 20:22:30,910 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "83886080", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:23:45,400 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:23:45,400 - DEBUG   - The next offset will be: 100663296
2022-10-25 20:23:45,410 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "100663296", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:24:40,982 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:24:40,983 - DEBUG   - The next offset will be: 117440512
2022-10-25 20:24:40,992 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "117440512", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:26:00,286 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:26:00,287 - DEBUG   - The next offset will be: 134217728
2022-10-25 20:26:00,296 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "134217728", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:27:16,182 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:27:16,182 - DEBUG   - The next offset will be: 150994944
2022-10-25 20:27:16,192 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "150994944", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:28:12,256 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:28:12,257 - DEBUG   - The next offset will be: 167772160
2022-10-25 20:28:12,266 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "167772160", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:29:17,003 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:29:17,003 - DEBUG   - The next offset will be: 184549376
2022-10-25 20:29:17,012 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "184549376", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:30:26,010 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:30:26,011 - DEBUG   - The next offset will be: 201326592
2022-10-25 20:30:26,020 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "201326592", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:31:22,760 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:31:22,760 - DEBUG   - The next offset will be: 218103808
2022-10-25 20:31:22,770 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "218103808", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:33:13,422 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:33:13,423 - DEBUG   - The next offset will be: 234881024
2022-10-25 20:33:13,432 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "234881024", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:34:25,641 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:34:25,641 - DEBUG   - The next offset will be: 251658240
2022-10-25 20:34:25,651 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "251658240", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1073, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.9/http/client.py", line 995, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.9/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.9/ssl.py", line 1173, in send
    return self._sslobj.write(data)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.9/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3.9/site-packages/urllib3/packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1073, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.9/http/client.py", line 995, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.9/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.9/ssl.py", line 1173, in send
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 98, in <module>
    main()
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 90, in main
    file_handle = service.upload(fp)
  File "/home/GITNE/src/GitHub/Mapillary tools/mapillary_tools/upload_api_v4.py", line 150, in upload
    resp = requests.post(
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2022-10-25 20:35:30,278 - INFO    - Session key: TEST_SESSION
2022-10-25 20:35:30,278 - INFO    - Entity size: 2147483648
2022-10-25 20:35:30,278 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-10-25 20:35:30,800 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":259883008}'
2022-10-25 20:35:30,801 - INFO    - Initial offset: 259883008
2022-10-25 20:35:30,801 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-10-25 20:35:31,501 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":259883008}'
2022-10-25 20:35:31,510 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-10-25 20:35:32,668 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":259883008}'
2022-10-25 20:35:32,698 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "259883008", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:36:28,646 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:36:28,646 - DEBUG   - The next offset will be: 276660224
2022-10-25 20:36:28,672 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "276660224", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
^CTraceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 98, in <module>
    main()
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 90, in main
    file_handle = service.upload(fp)
  File "/home/GITNE/src/GitHub/Mapillary tools/mapillary_tools/upload_api_v4.py", line 150, in upload
    resp = requests.post(
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1073, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.9/http/client.py", line 995, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.9/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.9/ssl.py", line 1173, in send
    return self._sslobj.write(data)
KeyboardInterrupt

I have interrupted here because I have noticed that the offset has been reset to 0 again. This log is actually my second attempt.

2022-10-25 20:37:40,292 - INFO    - Session key: TEST_SESSION
2022-10-25 20:37:40,292 - INFO    - Entity size: 2147483648
2022-10-25 20:37:40,292 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-10-25 20:37:41,125 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":283901952}'
2022-10-25 20:37:41,126 - INFO    - Initial offset: 283901952
2022-10-25 20:37:41,126 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-10-25 20:37:41,851 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":283901952}'
2022-10-25 20:37:41,860 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-10-25 20:37:42,728 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":283901952}'
2022-10-25 20:37:42,755 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "283901952", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:38:39,019 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:38:39,019 - DEBUG   - The next offset will be: 300679168
2022-10-25 20:38:39,045 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "300679168", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:39:34,987 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:39:34,987 - DEBUG   - The next offset will be: 317456384
2022-10-25 20:39:34,997 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "317456384", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:40:31,012 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:40:31,012 - DEBUG   - The next offset will be: 334233600
2022-10-25 20:40:31,021 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "334233600", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:42:07,984 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:42:07,985 - DEBUG   - The next offset will be: 351010816
2022-10-25 20:42:07,994 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "351010816", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:43:04,344 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:43:04,344 - DEBUG   - The next offset will be: 367788032
2022-10-25 20:43:04,354 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "367788032", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:44:00,216 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:44:00,216 - DEBUG   - The next offset will be: 384565248
2022-10-25 20:44:00,225 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "384565248", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:44:56,734 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:44:56,734 - DEBUG   - The next offset will be: 401342464
2022-10-25 20:44:56,743 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "401342464", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:45:52,413 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:45:52,413 - DEBUG   - The next offset will be: 418119680
2022-10-25 20:45:52,422 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "418119680", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:46:48,371 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:46:48,371 - DEBUG   - The next offset will be: 434896896
2022-10-25 20:46:48,380 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "434896896", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:47:44,558 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:47:44,558 - DEBUG   - The next offset will be: 451674112
2022-10-25 20:47:44,567 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "451674112", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:48:40,239 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:48:40,239 - DEBUG   - The next offset will be: 468451328
2022-10-25 20:48:40,248 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "468451328", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:49:35,860 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:49:35,860 - DEBUG   - The next offset will be: 485228544
2022-10-25 20:49:35,869 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "485228544", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:50:31,903 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:50:31,903 - DEBUG   - The next offset will be: 502005760
2022-10-25 20:50:31,912 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "502005760", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:51:28,351 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:51:28,352 - DEBUG   - The next offset will be: 518782976
2022-10-25 20:51:28,361 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "518782976", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:52:24,000 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:52:24,000 - DEBUG   - The next offset will be: 535560192
2022-10-25 20:52:24,010 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "535560192", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:53:20,297 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:53:20,297 - DEBUG   - The next offset will be: 552337408
2022-10-25 20:53:20,306 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "552337408", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:54:20,273 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:54:20,273 - DEBUG   - The next offset will be: 569114624
2022-10-25 20:54:20,282 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "569114624", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:55:22,828 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:55:22,829 - DEBUG   - The next offset will be: 585891840
2022-10-25 20:55:22,838 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "585891840", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:56:19,885 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:56:19,885 - DEBUG   - The next offset will be: 602669056
2022-10-25 20:56:19,895 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "602669056", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 20:59:08,108 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 20:59:08,109 - DEBUG   - The next offset will be: 619446272
2022-10-25 20:59:08,117 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "619446272", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:01:21,108 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:01:21,108 - DEBUG   - The next offset will be: 636223488
2022-10-25 21:01:21,117 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "636223488", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:03:50,355 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:03:50,355 - DEBUG   - The next offset will be: 653000704
2022-10-25 21:03:50,364 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "653000704", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:05:13,361 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:05:13,365 - DEBUG   - The next offset will be: 669777920
2022-10-25 21:05:13,379 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "669777920", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:06:25,875 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:06:25,875 - DEBUG   - The next offset will be: 686555136
2022-10-25 21:06:25,885 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "686555136", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:07:23,834 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:07:23,834 - DEBUG   - The next offset will be: 703332352
2022-10-25 21:07:23,845 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "703332352", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:08:22,528 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:08:22,528 - DEBUG   - The next offset will be: 720109568
2022-10-25 21:08:22,537 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "720109568", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:10:44,439 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:10:44,439 - DEBUG   - The next offset will be: 736886784
2022-10-25 21:10:44,449 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "736886784", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:11:40,718 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:11:40,718 - DEBUG   - The next offset will be: 753664000
2022-10-25 21:11:40,727 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "753664000", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:12:38,518 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:12:38,518 - DEBUG   - The next offset will be: 770441216
2022-10-25 21:12:38,528 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "770441216", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:13:36,781 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:13:36,781 - DEBUG   - The next offset will be: 787218432
2022-10-25 21:13:36,790 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "787218432", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:14:37,655 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:14:37,656 - DEBUG   - The next offset will be: 803995648
2022-10-25 21:14:37,665 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "803995648", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:15:54,716 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:15:54,716 - DEBUG   - The next offset will be: 820772864
2022-10-25 21:15:54,725 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "820772864", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:17:40,336 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:17:40,336 - DEBUG   - The next offset will be: 837550080
2022-10-25 21:17:40,346 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "837550080", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:18:36,901 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:18:36,901 - DEBUG   - The next offset will be: 854327296
2022-10-25 21:18:36,910 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "854327296", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:19:33,678 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:19:33,679 - DEBUG   - The next offset will be: 871104512
2022-10-25 21:19:33,688 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "871104512", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:20:42,647 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:20:42,648 - DEBUG   - The next offset will be: 887881728
2022-10-25 21:20:42,657 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "887881728", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:21:38,699 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:21:38,699 - DEBUG   - The next offset will be: 904658944
2022-10-25 21:21:38,709 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "904658944", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:22:45,440 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:22:45,440 - DEBUG   - The next offset will be: 921436160
2022-10-25 21:22:45,449 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "921436160", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:23:50,576 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:23:50,576 - DEBUG   - The next offset will be: 938213376
2022-10-25 21:23:50,586 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "938213376", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:25:03,163 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:25:03,163 - DEBUG   - The next offset will be: 954990592
2022-10-25 21:25:03,173 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "954990592", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:26:00,091 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:26:00,091 - DEBUG   - The next offset will be: 971767808
2022-10-25 21:26:00,100 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "971767808", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:27:58,371 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:27:58,372 - DEBUG   - The next offset will be: 988545024
2022-10-25 21:27:58,382 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "988545024", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:29:06,974 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:29:06,974 - DEBUG   - The next offset will be: 1005322240
2022-10-25 21:29:06,985 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1005322240", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:30:11,100 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:30:11,100 - DEBUG   - The next offset will be: 1022099456
2022-10-25 21:30:11,111 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1022099456", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:31:20,733 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:31:20,734 - DEBUG   - The next offset will be: 1038876672
2022-10-25 21:31:20,743 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1038876672", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:32:18,415 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:32:18,416 - DEBUG   - The next offset will be: 1055653888
2022-10-25 21:32:18,425 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1055653888", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:33:18,204 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:33:18,205 - DEBUG   - The next offset will be: 1072431104
2022-10-25 21:33:18,214 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1072431104", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:34:17,612 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:34:17,612 - DEBUG   - The next offset will be: 1089208320
2022-10-25 21:34:17,621 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1089208320", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:35:42,658 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:35:42,658 - DEBUG   - The next offset will be: 1105985536
2022-10-25 21:35:42,667 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1105985536", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:36:45,977 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:36:45,977 - DEBUG   - The next offset will be: 1122762752
2022-10-25 21:36:45,987 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1122762752", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:38:03,932 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:38:03,932 - DEBUG   - The next offset will be: 1139539968
2022-10-25 21:38:03,944 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1139539968", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:39:06,529 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:39:06,530 - DEBUG   - The next offset will be: 1156317184
2022-10-25 21:39:06,543 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1156317184", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:40:26,952 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:40:26,953 - DEBUG   - The next offset will be: 1173094400
2022-10-25 21:40:26,965 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1173094400", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:41:34,222 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:41:34,223 - DEBUG   - The next offset will be: 1189871616
2022-10-25 21:41:34,236 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1189871616", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:43:47,765 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:43:47,765 - DEBUG   - The next offset will be: 1206648832
2022-10-25 21:43:47,779 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1206648832", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:45:32,005 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:45:32,005 - DEBUG   - The next offset will be: 1223426048
2022-10-25 21:45:32,017 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1223426048", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:46:30,169 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:46:30,169 - DEBUG   - The next offset will be: 1240203264
2022-10-25 21:46:30,182 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1240203264", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:47:28,878 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:47:28,878 - DEBUG   - The next offset will be: 1256980480
2022-10-25 21:47:28,891 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1256980480", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:55:11,490 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:55:11,490 - DEBUG   - The next offset will be: 1273757696
2022-10-25 21:55:11,503 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1273757696", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:56:11,497 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:56:11,498 - DEBUG   - The next offset will be: 1290534912
2022-10-25 21:56:11,510 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1290534912", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:57:15,539 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:57:15,540 - DEBUG   - The next offset will be: 1307312128
2022-10-25 21:57:15,553 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1307312128", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:58:11,499 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:58:11,500 - DEBUG   - The next offset will be: 1324089344
2022-10-25 21:58:11,512 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1324089344", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 21:59:59,089 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 21:59:59,089 - DEBUG   - The next offset will be: 1340866560
2022-10-25 21:59:59,101 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1340866560", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:00:56,383 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:00:56,383 - DEBUG   - The next offset will be: 1357643776
2022-10-25 22:00:56,396 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1357643776", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:01:52,925 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:01:52,925 - DEBUG   - The next offset will be: 1374420992
2022-10-25 22:01:52,936 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1374420992", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:03:25,585 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:03:25,585 - DEBUG   - The next offset will be: 1391198208
2022-10-25 22:03:25,598 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1391198208", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:07:39,852 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:07:39,852 - DEBUG   - The next offset will be: 1407975424
2022-10-25 22:07:39,865 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1407975424", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.9/http/client.py", line 1371, in getresponse
    response.begin()
  File "/usr/lib64/python3.9/http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.9/http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib64/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib64/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib64/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.9/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3.9/site-packages/urllib3/packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.9/http/client.py", line 1371, in getresponse
    response.begin()
  File "/usr/lib64/python3.9/http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.9/http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib64/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib64/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib64/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 98, in <module>
    main()
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 90, in main
    file_handle = service.upload(fp)
  File "/home/GITNE/src/GitHub/Mapillary tools/mapillary_tools/upload_api_v4.py", line 150, in upload
    resp = requests.post(
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

2022-10-25 22:21:03,494 - INFO    - Session key: TEST_SESSION
2022-10-25 22:21:03,494 - INFO    - Entity size: 2147483648
2022-10-25 22:21:03,494 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-10-25 22:21:07,510 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":1421967360}'
2022-10-25 22:21:07,511 - INFO    - Initial offset: 1421967360
2022-10-25 22:21:07,511 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-10-25 22:21:12,524 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":1421967360}'
2022-10-25 22:21:12,544 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-10-25 22:21:14,756 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":1421967360}'
2022-10-25 22:21:14,784 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1421967360", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:26:30,899 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:26:30,899 - DEBUG   - The next offset will be: 1438744576
2022-10-25 22:26:30,927 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1438744576", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:29:41,668 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:29:41,668 - DEBUG   - The next offset will be: 1455521792
2022-10-25 22:29:41,678 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1455521792", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:31:39,533 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:31:39,534 - DEBUG   - The next offset will be: 1472299008
2022-10-25 22:31:39,548 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1472299008", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:33:44,686 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:33:44,687 - DEBUG   - The next offset will be: 1489076224
2022-10-25 22:33:44,696 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1489076224", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:35:57,587 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:35:57,588 - DEBUG   - The next offset will be: 1505853440
2022-10-25 22:35:57,598 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1505853440", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:37:52,881 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:37:52,881 - DEBUG   - The next offset will be: 1522630656
2022-10-25 22:37:52,892 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1522630656", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:39:41,035 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:39:41,035 - DEBUG   - The next offset will be: 1539407872
2022-10-25 22:39:41,061 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1539407872", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:41:38,856 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:41:38,856 - DEBUG   - The next offset will be: 1556185088
2022-10-25 22:41:38,866 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1556185088", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:43:35,300 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:43:35,300 - DEBUG   - The next offset will be: 1572962304
2022-10-25 22:43:35,314 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1572962304", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:44:31,618 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:44:31,618 - DEBUG   - The next offset will be: 1589739520
2022-10-25 22:44:31,629 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1589739520", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:45:28,041 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:45:28,041 - DEBUG   - The next offset will be: 1606516736
2022-10-25 22:45:28,051 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1606516736", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:46:24,204 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:46:24,204 - DEBUG   - The next offset will be: 1623293952
2022-10-25 22:46:24,214 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1623293952", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:47:20,404 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:47:20,404 - DEBUG   - The next offset will be: 1640071168
2022-10-25 22:47:20,414 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1640071168", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:48:16,001 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:48:16,001 - DEBUG   - The next offset will be: 1656848384
2022-10-25 22:48:16,011 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1656848384", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:49:11,654 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:49:11,655 - DEBUG   - The next offset will be: 1673625600
2022-10-25 22:49:11,665 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1673625600", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:50:07,410 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:50:07,410 - DEBUG   - The next offset will be: 1690402816
2022-10-25 22:50:07,421 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1690402816", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:51:26,047 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:51:26,047 - DEBUG   - The next offset will be: 1707180032
2022-10-25 22:51:26,058 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1707180032", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:52:52,315 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:52:52,315 - DEBUG   - The next offset will be: 1723957248
2022-10-25 22:52:52,325 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1723957248", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:54:01,678 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:54:01,678 - DEBUG   - The next offset will be: 1740734464
2022-10-25 22:54:01,688 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1740734464", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:56:02,919 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:56:02,920 - DEBUG   - The next offset will be: 1757511680
2022-10-25 22:56:02,930 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1757511680", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:56:58,996 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:56:58,997 - DEBUG   - The next offset will be: 1774288896
2022-10-25 22:56:59,006 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1774288896", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:57:55,373 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:57:55,373 - DEBUG   - The next offset will be: 1791066112
2022-10-25 22:57:55,383 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1791066112", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 22:58:52,923 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 22:58:52,923 - DEBUG   - The next offset will be: 1807843328
2022-10-25 22:58:52,933 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1807843328", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:00:06,538 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:00:06,538 - DEBUG   - The next offset will be: 1824620544
2022-10-25 23:00:06,548 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1824620544", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:01:30,294 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:01:30,295 - DEBUG   - The next offset will be: 1841397760
2022-10-25 23:01:30,304 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1841397760", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:03:33,073 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:03:33,074 - DEBUG   - The next offset will be: 1858174976
2022-10-25 23:03:33,084 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1858174976", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:04:29,613 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:04:29,613 - DEBUG   - The next offset will be: 1874952192
2022-10-25 23:04:29,624 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1874952192", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:06:29,630 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:06:29,630 - DEBUG   - The next offset will be: 1891729408
2022-10-25 23:06:29,640 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1891729408", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:07:26,719 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:07:26,719 - DEBUG   - The next offset will be: 1908506624
2022-10-25 23:07:26,729 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1908506624", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:08:41,728 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:08:41,729 - DEBUG   - The next offset will be: 1925283840
2022-10-25 23:08:41,739 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1925283840", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:10:25,791 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:10:25,791 - DEBUG   - The next offset will be: 1942061056
2022-10-25 23:10:25,804 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1942061056", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.9/site-packages/urllib3/connection.py", line 352, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib64/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib64/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib64/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.9/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: SOCKSHTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/TEST_SESSION (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 98, in <module>
    main()
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 90, in main
    file_handle = service.upload(fp)
  File "/home/GITNE/src/GitHub/Mapillary tools/mapillary_tools/upload_api_v4.py", line 150, in upload
    resp = requests.post(
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: SOCKSHTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/TEST_SESSION (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))

2022-10-25 23:11:21,777 - INFO    - Session key: TEST_SESSION
2022-10-25 23:11:21,777 - INFO    - Entity size: 2147483648
2022-10-25 23:11:21,777 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-10-25 23:11:22,562 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":1942061056}'
2022-10-25 23:11:22,563 - INFO    - Initial offset: 1942061056
2022-10-25 23:11:22,563 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-10-25 23:11:23,709 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":1942061056}'
2022-10-25 23:11:23,718 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-10-25 23:11:24,995 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":1942061056}'
2022-10-25 23:11:25,022 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1942061056", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:13:23,733 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:13:23,734 - DEBUG   - The next offset will be: 1958838272
2022-10-25 23:13:23,759 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1958838272", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:14:26,416 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:14:26,416 - DEBUG   - The next offset will be: 1975615488
2022-10-25 23:14:26,427 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1975615488", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:16:00,195 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:16:00,195 - DEBUG   - The next offset will be: 1992392704
2022-10-25 23:16:00,206 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "1992392704", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:16:59,486 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:16:59,486 - DEBUG   - The next offset will be: 2009169920
2022-10-25 23:16:59,496 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "2009169920", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:17:56,784 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:17:56,784 - DEBUG   - The next offset will be: 2025947136
2022-10-25 23:17:56,795 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "2025947136", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:18:54,390 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:18:54,390 - DEBUG   - The next offset will be: 2042724352
2022-10-25 23:18:54,400 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "2042724352", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:19:50,560 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:19:50,561 - DEBUG   - The next offset will be: 2059501568
2022-10-25 23:19:50,570 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "2059501568", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:20:46,316 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:20:46,316 - DEBUG   - The next offset will be: 2076278784
2022-10-25 23:20:46,326 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "2076278784", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:23:28,678 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:23:28,679 - DEBUG   - The next offset will be: 2093056000
2022-10-25 23:23:28,688 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "2093056000", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:24:34,534 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:24:34,534 - DEBUG   - The next offset will be: 2109833216
2022-10-25 23:24:34,544 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "2109833216", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:25:34,267 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:25:34,267 - DEBUG   - The next offset will be: 2126610432
2022-10-25 23:25:34,277 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "2126610432", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:26:32,348 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-10-25 23:26:32,348 - DEBUG   - The next offset will be: 2143387648
2022-10-25 23:26:32,351 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "2143387648", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:26:47,729 - DEBUG   - HTTP response 200: b'{"h":"3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbuIYGp73fMFeBAnuoGUW3TICCH7rGrqpAAfS-tT077vzSZMYGCrx2tG6E7fj-WDb8KyoTfGKOvZCuvZBTOvyEpVa5sQzoq3U_ZM-2SEi9YVg:e:1667067315:ARbOnKC_bQmfHFK-9Vo\\n3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZdeJPtgm01h_l9drC8MqhgHyo2HzlVByph0fz14n1Ger4zSKm9HmEd-UPB34vqlcDdw6jVTKtXPMc92O2sLh_kDtJM2_5nhyAuSCTaZtLebw:e:1667067556:ARZhzskD7MOfn85O99M\\n3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbwiaV3xRuhOC1Z4SH08VqtkWBz7wTpVeOnvPk13yQcevgytNl1bO_iku9RYdWyggJ6qDyEyX8OGIMDWIex9a4txqSuGw1d...'
2022-10-25 23:26:47,729 - DEBUG   - The next offset will be: 2147483648
2022-10-25 23:26:47,729 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "2147483648", "X-Entity-Length": "2147483648", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-10-25 23:26:48,334 - DEBUG   - HTTP response 200: b'{"h":"3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbuIYGp73fMFeBAnuoGUW3TICCH7rGrqpAAfS-tT077vzSZMYGCrx2tG6E7fj-WDb8KyoTfGKOvZCuvZBTOvyEpVa5sQzoq3U_ZM-2SEi9YVg:e:1667067315:ARbOnKC_bQmfHFK-9Vo\\n3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZdeJPtgm01h_l9drC8MqhgHyo2HzlVByph0fz14n1Ger4zSKm9HmEd-UPB34vqlcDdw6jVTKtXPMc92O2sLh_kDtJM2_5nhyAuSCTaZtLebw:e:1667067556:ARZhzskD7MOfn85O99M\\n3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbwiaV3xRuhOC1Z4SH08VqtkWBz7wTpVeOnvPk13yQcevgytNl1bO_iku9RYdWyggJ6qDyEyX8OGIMDWIex9a4txqSuGw1d...'
2022-10-25 23:26:48,335 - DEBUG   - The next offset will be: 2147483648
2022-10-25 23:26:48,335 - INFO    - 3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbuIYGp73fMFeBAnuoGUW3TICCH7rGrqpAAfS-tT077vzSZMYGCrx2tG6E7fj-WDb8KyoTfGKOvZCuvZBTOvyEpVa5sQzoq3U_ZM-2SEi9YVg:e:1667067315:ARbOnKC_bQmfHFK-9Vo
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZdeJPtgm01h_l9drC8MqhgHyo2HzlVByph0fz14n1Ger4zSKm9HmEd-UPB34vqlcDdw6jVTKtXPMc92O2sLh_kDtJM2_5nhyAuSCTaZtLebw:e:1667067556:ARZhzskD7MOfn85O99M
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbwiaV3xRuhOC1Z4SH08VqtkWBz7wTpVeOnvPk13yQcevgytNl1bO_iku9RYdWyggJ6qDyEyX8OGIMDWIex9a4txqSuGw1daM0MVWFrpxFbUw:e:1667067649:ARa7Ur3Q5Xc2CCi1Sm0
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYqMM4n-0_TxqlsCfDRojHTBX2omstiHsI0uh6VZ-KESKL4oadhTpMdtRgRHG8KG5MksEJf2aI88K83f4iecwi5gfJoVAAxsJAFWSdxVFLZ4Q:e:1667067739:ARYOXWjGD7hiPBcz_5o
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARaC980zgH5GUR2tv6fy64t0cdNFhlySmrAiC_tpd4ol4zQURNt-lrIQqzxHj0RrseMTHtGpeBXZlXGu0FQ26Mu2B7jgSmVGkaplDbb9PctkBw:e:1667067824:ARYzezSD2LrPVjXE0QU
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYISSCdtsx7xuWPCxnfNAmqtfgAdGEhYasTFd1zL4h31py956uz2UdbfdBP9LzJOwyhdX7ptjqhXUmqPPjtuO5sNu3yVUpEkKLk5m90wuVzvA:e:1667067894:ARY6QoESKySulxDUWk0
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbTqhY3CCVjRla8wSRHXbC9w77Woz9RK4t0ToqJn8iIJJ8GJKph1O4oDPsZTtcvw824tFlgcWPj840mRyCxAdDJRNUm3hfY2nAFhjYYzSj21Q:e:1667067993:ARbsrAIBeJY9yvbyR1A
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbzgNSWgnCgxZiXaRuDlK4vgCautGC5deVZAVPNbnGjU-zuZCTwtnCbjd4DoVbtvnrcrG59SnRcgs8CYk9ZXFiXJ4eEhBqC-v6xsBJu_S2sXA:e:1667068067:ARbsf0qn80AsAPLOKNs
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbKhOHaIeXJVsOsN256jA6D0WoROcPrzDhDQKY1bwjbdYIr74ktAH7JNdkLjeGBS88-irwWr2zlwuDKScgocgdYaBiw0KkL_STe1-gWUgX01w:e:1667068142:ARa2vVfVeb5PM1GJ21k
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYcq3PmWCgGIRhk6QL44fDnE5jNsnN6Eu8nyFwm_M5Cp8gvPD4Z34qptoQLpAhoLKORkqxy2DcfDNnOsQEAfM-wOqLU3JFUxVT8Sz4rHPYDAg:e:1667068221:ARYiFnSGmZi77wqnf1k
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARaFL2UrRHUDQmhqadMgyfjSgfMKmtbuAkoEzQFa7H2LHo663c0yy927YNcMPrUESWuSY362ZbzBO5xd0aKfpYFF1x1tmL-iTERXjeRDj13Paw:e:1667068292:ARY0TmxItZCzcj-2JPM
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbglqbWPxtFKOpjXPNaFFah-LvVjOHxJfOXBW9r6skIoFMT6OpD5d3fnI883K0vfgByv4V29yO08DJYnDrRrthN2l1q_UJ5MaJ9u8YUI8nb0w:e:1667068423:ARYHWYFsaEkreI4mBeE
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARY6hI6L9_h7RSmuu9dbk8yV93acW8V1R2cGB4qkpuG4uzypttc0PGdT1JfJEgwUiPspNMsJflMMQIZpnGNIwFOfg4C3geILA4T0KiVWjM0UpA:e:1667068534:ARYipZx2P9I-WFprnmw
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARa-xa5woqcC_58pFYY-yHH2aPDNLP0-32O7i8iUQiEfb8NJjuxo3oU8IEanbsZHXRf7negw1AkGSyN-VwgNYNaF-afLa5-tVDN_kfhU8EB9wQ:e:1667068601:ARZREk5AfCP5qmHHnuY
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYXlrpEx7dFVnpVffbLO9I6Yc13yqTNeIYhodtEdbS0KEXpIAapAhxL0yfX93fAKk2zMrQjfr-PZTYcRtRm-1bYmZtbH9jr328btbrs4xfm7g:e:1667068718:ARaoR27N1kzJ5pyc5b4
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYw8CCG2aAzq_BnYNSswSM7pH5PGQ-PCWLkFij28Ida8EqEO6tQXRUvhfWriczVcRD0zdm4ZZWBvxM5fkrUzQdt6bkB_4Mt1x9lHSr2p5j-wg:e:1667068784:ARaTstws8nZCtdiEFdU
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARY_-4CH316Nff0S1Tj7wbsmhPxgJF9isYn7dYM1Qt91VZgGwiWP65g4bP1z3eXcVy2IwOk1qxFMfSsXc3Ww7jVtCoxqY-Dlc2J-qT0io-nejA:e:1667068888:ARb30IyRdTDykA_hcfI
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZa6CzLzR9J0T3Likg9rtF4g2ER_SEA9oLBHeZO2dbQFssYebp0IgaBw8iOaWAOoY_HX8A8XuaaEE8uSZtISZLYz14iCFcNXZ7IV6j4L1AnIg:e:1667068959:ARYfHtIQG4w3ZYKyHGM
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZQ7PG_4QPiTy9YuJjAc1BETNcCcTFKnTLsiMZAsFajALAjBrl4cNZiwh84QnOcVun9xcAk5kvfggFvq6puo8U3erqbqaK7cKwPvVPfjCaThw:e:1667069025:ARZ9vN0tkvwtumwy6kQ
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYos-u8h6WEwhehHQJCx4IkrJuOEiIbiFJy5kbvlKjNG5PLflMDoOe3i8bS9DRqYlEJQgX6wwb7UUypKwIM3b1Qr0vCkIDxEwLo6rET-GnXSQ:e:1667069093:ARZDFogGyMfISXQV_pI
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYQhpTQcKrVSU2ZVOuuiuM5uv37bcCokFD64tuSy_Hah5jdoN2KyKdSvW67D6vl_OAgrWWLO-fQbryqi4GSS92ktMhaYL4Filno39e6Yc2Bqw:e:1667069159:ARbkCJn-iR2GuUIjcsI
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbxoIxCTg-_EQ3oaV-ekUoN-SGN0T9s-c3wMO3Q-dHVkElMcx0iNm44C1YUvRmuwb9Xsrxyt21LIQ1oDonLY5drRVi-miVXIRLGLhWtzuDwXg:e:1667069227:ARZGPxRvTOLRaU7jO5U
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZSAQw0erKVuL2KcQbutu8SqFyoaU79TIwEWZ9w9V4T45md9qwNWhAvvVltXdVz_NVXdrSMJgeDh_z9d7askwrJG18bw830g_vRIIyVdF0mBQ:e:1667069293:ARZZkis5DZFoaSkfTI0
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZ-duOrdb_pjrGIXU9eSZt_2hNY87IwkhotqYPbZA1HZrdXB3SYizNzVwf7VMXipE8fdjAaRhuFZMdEVvy1Uhlqz0cg5-6uf593hAlAYHaVNQ:e:1667069359:ARa2r6ts6ASb6-bby50
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARa6AV5k7I_6lXNqzmRvpxLstJn_ocMztUw8qol-Vv8R5hx_2M5PJjmfRgnrCN5YSYbJJS72N9-w3pW15cC8ZeaaFrwDYzeru137uLq6CHSDOQ:e:1667069426:ARYlPMnLciVJ3CemS0Q
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARabGyDuoHIuiRc9wDrU1EnoiiQMgSdrhj1PX4TBozjEuiw9Mf_jwZWiFOu-uDZqk-xY_Cwq0phBoQvmZ2LDsxSqwrd5AafRxhpDb506i7obMQ:e:1667069493:ARbgKD62qSn1_l3hAz4
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZZ_-tiOPs3C9Ccsk4F7bUsD7Z9CB3gGxL3ln9vwe6bJQ8aYxBJ3-pPpdtP5KyBXHQeoSoLwoYG4IerNeUesEyUPzeUwYhqYyxhyFPSFGjn5w:e:1667069560:ARa9XkgF6SMhkqIy0s4
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZJmnxo6auoPwvhmIOtc7U5wv2TyjfsvFc_wM6Mf_phe_MCzpL8Nx4cm8yTagUenfbJ_dzwg9rUBr4wPmWyBscJo6jfgkCwJrA0nbgVWACvjQ:e:1667069627:ARZ3m7n5tcl68lt83BY
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYJcYiew-iyH_F9dN6ScddjEIquYF_ND4Re8yejVDsW4anUBWOPDES380cFuXbHn1AIOKgdhD-FPsPgxiTVgXyseKixHrW0ymSjQLXIq09IYg:e:1667069704:ARbTpfDVR81lUZUaLVw
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZdYPuYyKj5TUAPQA2-gJmY3CJcr4dq5QM9RjT-M_BKNEH89wcyJ7b7y_wmVZa1fZDF2Z2MULwfc5YXL0q7Z3018GjPfZSfJJsQ0HI1yWUyWQ:e:1667069772:ARZ7Mh5Sqy_rGgtW_ZU
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARY2_YrlfS9Yc177GcYGG7B9JipDSpcp9t_pUWJNIuRdY9a46d2ZIbWvW3904j07KZWPfSLtZ7VFhwk7qQIhHQAbcmzY1MKSC_2i1pSJSGL-qQ:e:1667069956:ARaa0jPxObJQEEti__0
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYaBiDisldkbSAW-j3jFRjbCRchwrV8AujqFK5Zn1wkClgB06ljWC9zOTiTDEooX_9kF9QArWbct4Q3snvtSKEAsBHRluB-Tsch3Rk3TdrdaA:e:1667070138:ARbZycD0-oacmC919pc
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbgWy8BnOmX8pM5j2fTdKYeVileRfSQv7Ky_fxN3sSKhWm1WBnm9konJV2adu3TpcFJy1rErygWHfjcvpMxcHiJuOmdt4ORZ5fU6g7tDQotLQ:e:1667070273:ARYanJO-n-hQ99pSxmE
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARb2lOXFwM_W6Q9vx328joYc0rgwofHPaC7NOtQNG0DhZqKk5VO86RO0I23v1m6ftoFfScnkJlY0q3-gEzPk8T3kks4Grpy8U0L3XBd5bWSFbw:e:1667070359:ARYWVv2xTHRsuR0HcWw
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZTy6kJa43WZbXcNzp7GBphr8UUjKIS9Xe85L8CqVAuoXO6u7tRArmKejotz_1kVNbPw9QseUoukbREDjx6nnaHDBKC2WMly1o4SH1ZyX0hBg:e:1667070433:ARbsuSRaQLRBGoedXVA
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYjlsF2uMeeiogsaTF1Sk7qWNolH0sQbPnol1QpsD1Akya4JIorLLWMG-nh__7dnijrgC5936M7_MPJ5Z6FQIXk7UyRTygHwY1lG8iuOJ2B9A:e:1667070503:ARa0flS8lKJE1vPiKzw
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZWCtmcH6B0ylmUxSQJhCVSvs0uKBJ4f1E6CjPApNDhhxXZCisseDtTIyV27pJ05lnIqzRRzZf_A68nsGFph4aF8bDKlYd0Y5sbNe1-8WDzdQ:e:1667070656:ARZiZaRsXUXDfy6APp0
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbiSXGnHsCTg0bQaRM8ykL2Ia2guS8KDQou0eiig4nCrBPmFVIzoHo7JEG457JMYw4sMKY2Jfs09-AXnxjhq6-gvtWq_U7ABPrVr115UwjV3w:e:1667070724:ARYIif9RW1MzrzzY7Lo
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZj7k59-NYYhs8DNabygefiLZA1c9ngjNU2U-iySilfV2jk8YHqJLbPA1-En5BYeCiGKBfzk3GusGztjbdomd5SObeH9i-ON9v-XoNqR2Ah9g:e:1667070792:ARZC1KHWnmi43zu0h-4
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARaj_3fqxvSY-5AfgNzuej8QR8lS9guuVQVR5qhxqTyk5CCZiBBz5dy-YUBB9ItwWkzxwkK8qg2EAhXkZycrcSDaOQbBvPrZKFmVqX0GoJjn5g:e:1667070865:ARaIONfnp_ivSDNMvwI
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbcUD5zE8ve1oEF4J9FFS0fNXSDGn2QYOqMvfMYP3eqhLo-rE9WouhsUq2YmZEQhZGlZXlCHeEurfgpZYfqhOTmbbm6eeD5qjuDPnVELOJb7g:e:1667070951:ARZyse0DtckPZ87B82k
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARaYX4P7LeEHsDk2-p5nS5Un6fNJlnxr6hN0Zv-NHjVSUkTo5zIaF38Yqq2pe0P2W12-kKFrBIlRdhnt57huVdTtKfHmhinh4a6DZTPzMWyDVQ:e:1667071070:ARaMV_kY89C_freWYF8
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbm_aMCaGIY46pzLuROSvyEFJ4dE34dhFQ7awL6U7TU6v9fQ0hv2b4QoHX0wQ_OyMm7tRql77001Vx8sSd3gPjrih1YSvF0UPvWScmIMQSsjg:e:1667071137:ARY1_iwmHmG29OSGUVc
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYwO8JSAjuTC_APOEmYGGSuDFevhl5E8REYAUZaky9g2z8xTW-O_1JisWTgfmEXGPGFQolbl5Xcrk8Az0J5GkwZe4TqZ3dzs1MO8qB4XISryA:e:1667071205:ARaVkLRoXCQp_CSUu7U
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbmf-znzss-o4y0G_cdA_sEAPo7s48SeJFaQATZwJWM87A2AlXbj9QqxyFEBp3fu_sa9yITHST4EfrsLBxnm3zexPH-3r4YUC9w7XAs8u6pTQ:e:1667071285:ARa2LdOnHidheq0Bv-U
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbktFR3H4cNmflwZ2Es05jezPDX1lFXjXlkoUS7MyxitTzDe731O1ed958sqU3H1mKsXSAdURPtdQI4N2GKt5wNyHdn1hP-s4yj-Zu-WkJeVw:e:1667071361:ARYzSv-nQG0O1p6dSEg
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYuKdoY5pKdqj6MN_wzYtQmxDcymudl71X0C1_ukZHS6ejSSufmt-pV6UKty0EZPdVAnRa0VP8z9LX6K4cyrF5-UB22K4Ptzo7rOcqu_zn1KA:e:1667071438:ARYMvTUEwrcvaPp3plc
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARaFmkg1MKw8Vy5_xUaWGAbnBLFY38PvnI5GMSQUCcu1NmjbhXgYa4TuyRN-nIqgvrRwiWTG3ko1dWzs8DL6PqaacK7mfirrqa7rKy11ZD6hTg:e:1667071522:ARZMiLCmDsMJaffnlWU
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZNqmwXqi_pR3fCz9kYlr5KSb9RNx54R_exx56Xy2Yh9-7RVlJX5Ko5W8aXGFRQZsIxnoeDwY94zC17leCwG_XWV01VjNLEnyZNuUTtymklGw:e:1667071589:ARZvmSw_pwzEvpMdbzQ
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZMI3pPKfPMcsJncoibpDSwFQQOVnLK0PTiXx7vPJCKxoIuj76UndYhKCn2c3Ks3vUXkD9BVOS8Sfo7yBZrPx9h7uLr85eIfAox3QYrWDYilw:e:1667071730:ARZ4291RIdY43-nw2Ho
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZj4-QPKtIwN_V_Ci7DpxN_an29aEp-ZONQ2HQhcreZF8D9k2MjuFhW_MusEdlS2MkZeI6Q9QkgbsgVW74bNJ4Va_Quk0cydbvfIRLnipnfVg:e:1667071804:ARbPyU1obw19s2Q9Uew
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYEfNIbBCh9B35oIMnoRv0cmpc4C48F3R5mQhVqAwMFc3PbZKmNZ3oB--JvqNuXanmIDwd3OGaGrrZmbO92i8qLmuf81D21o4JMlTO68WNbJQ:e:1667071887:ARYvWh5keST_XgTj-x8
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYRkkcxJlpERJslbJtN9bMWpF6kH9Ri3xc2_F5yaJsB9tfLN0RkLtVwkgyBVv8PdBIufm5_9vzjY2PU_D0zseLhqe6i4XkZhu0hMTYILSYfyg:e:1667071955:ARaAu5Ci-LTBew0c1sg
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARY8wyTYC3yweD4aFTWYagCTkklr3o8HfpjODgNoAgyV51928lbZTI3Ja_W2Q4Hp8Qo6n5rjrPgoQ92pDuuPQU6TCyjQRbS5CVYJZYBeJwBrOg:e:1667072028:ARa5SjV0zuyw6NTlPQ0
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYOSOW2iPzhzm15QeW0WUFLn4Vq_aQx0LIzKy6qcv9pjv-DN1p_8sZ4lL9OrnLKbW9uxwjgZH8fRCQzgmyV7dZO229Ea6IlW92ROelfZ13_vQ:e:1667072124:ARb8UnzhPYzdmZnNr1U
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYzSEOmh9fIi7AZNRO138P51a0bJrQKl7rkjhBlabbnQZrOW_wYBPspK2lDhYPtZqv59zayvxyFJ0YYWmjUKEolcfOMoQHosXsY6XO6QiQGsg:e:1667072197:ARYDKDKYqgg6FZU7IKk
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYIw3kNrp5pVAH2BVgW5FuCBHahs8hIpbtY2A2IPSKs3FsU_porsyBqf8tWs72g787lItqi2oNvdx6y7Cf03QsstuAc8ljaODSWq2Q5StZM-w:e:1667072288:ARYu0QEBW-MbsEboUDA
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARY4k0CrJqbLDtSZMIWL2F4CItOXxZt5yWJ6dob5yKGU1143IIfD1pdjwnI1IUhLnoDHsghKCM3GvFBQAgrbCQjhsmve8xZE8yhXfrM7aFht6A:e:1667072361:ARYoV6DmoPSIIfJAAcw
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYVoikYtuiaFFxS3xJc8NY7hbyvLzlY2ERpGafC-FH9ebwJHr8WB_i5s8F6vnVdUAJBLmk6Brt_fu4CBA1O5O1wY_8WRVEZ3fZvpJHd2MaN6A:e:1667072461:ARaYi0c2orQQj92ywY0
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARafriJsN1vzvKcBbbQm2hNdl4eV5mErtrIZHRzg3iKmmbBAsf-PjLi0nCxLkKyw05ATOVPaUw9upwRLNXvtwncbvLqWeuVyhOgtRF1VCuoaRg:e:1667072582:ARYycps6jfBpZ2OCyx0
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYkw8LZ0w80SQQZho7W9LdAxLF3ApiNjf2T8IyfR9Wwv31UyZX99Q-ybooTSALWropit46rB9ckWUWEpe3GVYYYYXGTN8dmNjttoscC234FhA:e:1667072708:ARZyLwhoYJxmHlkOgqc
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbhMUoTnem-IaNIaaC0NBUuRJhZPuPY_af9Qd-wgOdRqpRF8QKuSD-dYMa2sNco3koEMkgcv_cdfLofYl-0MeaFWjWQkcWalTJ8iBsUql1Flg:e:1667072790:ARYdY0UVan-OgriQ2AY
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYWhy-upQ-rog17sISOkggpZqMyU1HD7pfu5n3ZdKDhuiOmLCgNP0jaFicNmSwX4dM7mOcgrl91nP2IfvX-Fh6fThKSfRH5hBsvR-21OwbWWg:e:1667072871:ARZsEIB8M5Sq82YVukU
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbRoLWMOdzfo9A_Ugkyub8KAO5y3MiBWT1fE-IFimul2RzqvSQtdC1Op7DAmyHI0e5lQK2FuM8m2N3O8CTbb8aIrpPoEUuJ58Tf-_qLWmGzQg:e:1667073338:ARYuwb_Y4xghyPbAMY8
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZoOkTz2eCfY8oC4RRhXqG_hVX-QRrPjYW8sJ-NsEBT0rOnIHnuB620iec_PyFSVMPrKiGcK14H5UUq-E7yjX_yzCcyqD56rLnn-rd4ZhYOzA:e:1667073411:ARbZIwwUBa5KSGXjM5o
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZ3GIL-wx-J4nZqqTJZVVeKTqzB232BBBTdQRAmix37UMEJbEldH3pXGd46YyprvH-N_SQGEGMbEvGg6ubwxF2cDA2C1dg_6wgBDH1YYob9Aw:e:1667073479:ARbREnIvmNRKXotJuHc
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYzk4ljgug9VU11Vr5GiS5U7obAx8s-wvZIiBrEH8hC2wsxKsW7QsaRxfx78Fa-vtYNpsrEIlAH6umrRN4qXXCtKCxtf_vydeNQhnNMlvuu4Q:e:1667073597:ARY2uJ2d0guCo8iTjZ4
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbX1F90ATSb6-9wzAB42Vgw5uKmlK-zMHYWSg4cIx0DQ8MNLogWM9mAN6-LoXXJtW4BKKY68kzN8LbvCkqZIXhDy86m9vNsK1HCkxZJc4nk2Q:e:1667073666:ARZz8_pmVYFvutIwwoQ
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYr1qkQt5eXGdCtQDBlz-23P-9eEOkCVp7urkOEuAhWEUlCiluEl87yqf9GQvLB6xRwNJGz8hiPwtIVKH8dtvNbLmXQenfVLeBvUmlWOaEhjw:e:1667073733:ARYQ7dLNeLKJaQv1g0Q
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZsoYxgZGB3F2QI0BO2udFNRyUkY003p6FvVk_3d-zWKFc6dHmSrAs8gvsG_jVnhMjj_9Yfo8OGveFBA5LblyOL9LhrZQi71RqMMryFThOz9A:e:1667073999:ARZogd1kKqbo913Ag54
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZhBL28hiVMFQb1Q9FJENG_wism7ePTBhwNkRwN3PVhnG1BBwdRBbplgNzOrAHEkN2_D9nDZFziXh9N9YAeA7YNWKH53EJVjNt7TKs-QHDofg:e:1667074440:ARZOvbFSaRb8r8166gc
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbSl7temgiGA_JH-9x9jd6U_eYWnEV6QfOWGsT5NDKDKRNVIIMirGquzmpBMIGXE00GKzXwIcKmBFZXVPNWLKPf-H73mE30tiltQTqHk7K2ww:e:1667075204:ARbxzAn2uYAueeYPiX0
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbHVqVPi0MNPBXsT8OoyBh19iZNxeqKKkiXLcshwlNTWLy-owXOr__ESkBXQ02Bb7d4OLhC2CgmqWK0v4dMwdv5vR-BIngRjVJbseciwT7CQQ:e:1667075414:ARYBx60Ne3uuj_HbwH8
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARa7a3e7IIlnd7_SH5lYL3KS0H4UvDXfrd_7rJCPShGOz3dTM9pbGkFBvJXytlsG9Rh4_InfeWZyTc_OAPJJT3XLoJYzp3LO2Z_p2b9sBj6nKw:e:1667075538:ARZWsh-3ONPKNizIFu4
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYfqhizhSOXWXMvLugKcCKFfIG43oyS_GQCtLBF0bYwnYzTg5oMeyQNyWRVl_Oew9fMcW__MYCEi99Ad_zCF3db5YXt3WcKVwtCWawgOoPF1A:e:1667075711:ARbZO7cfdq6qHnzHwiE
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbrTT4b8QAbJbg3fY6Fsjley-0QRny4zAF9sGWd1j1K4nQZnI0xe48xvC8OwLmp4zDDOh18YxXixqqIdzWXTCTqWSMF3taHk1wUUWmN7uXi9w:e:1667075847:ARY4MnXyvTB8LIVH-vY
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARaMgtL-2dnLr6tqHYIpcU2YGagoCbUgrUrUp4pR08oyHXZm2XIJwpubQTB2LSuA61_tnDHy9bDMPT-U2gBVK9bOZnJnY0drwiyCeIBnn28ObA:e:1667075985:ARbuaH6y4G2L4SGTB48
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZOxFqUUYAeqc6y08OdhmiDuGIQyxAhFj5pxuiHlNz7eq6t9-LhxcIth5FMSiwSXhbjOxTB-LYKLI8bmx_Fliwh9l4BqNRIRfsDNtVdU6NaHA:e:1667076141:ARY-pPCLYEIQtIVqN98
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbTQphCnr211zHCZXQDl1F3JTUmGE5fx-_Ji3QXvHJJRM3i0ioz8N9hHnIM5VF8jE8bB-OSkYsga_dUz-mAzeFucPMXyNljU2I6sa_SeYZXWA:e:1667076240:ARYe7nQTv7S1JEUmuiY
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYaokHW09uh_bfgLHskY_1-ylSvRgs58ra1IlaNgQfQGBiIb4Xb7BusbjB4md7rnBlYmahB_yS3yiVoD57xCiG0JHE8uEiToqxSY3IwM_br6w:e:1667076307:ARYT0oAsDWMFiBOuVmw
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZrW1yfQFo6q1leR-WoKVtLgMQICS6X1eZtGgtZB6UjPJ8hj5YJBCesFenbxjMgVlxKBcAKjxeAfnX00bTojRFQORvHZV9Smlp7m0q-VuzE5g:e:1667076374:ARafKI_rNNLbrV-EUbA
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbJdR0Eh1RkjUxttmxkRHAJLtTtVz3YPDo-fRDoW3hcv10GBiifap3K3Lzz5mctTDyI_fYfRw7_J-VRWyEHLJMbLgdE8GrM_HZUNRZIQF3YOA:e:1667076441:ARZn13F6H1yIYqPe2Xs
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYqgSHEAE7VW5-lKXgoBBgil9KKAIUXhv9_YYAT-4pF6bxjDnGrLYt2HCe441YxhkIGNrYsT7HXRJfw98ds5uED42NHtD0WsYXLk2rCXXOpkw:e:1667076508:ARZJoVaOK77SlpHq47g
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbnRgH73y4Phlv3TCPv01R4t-cv3EqOviDZ9Mk781lFQNP4ix5AgoJcsrRgvzDcSc3AfBoKxsGA4JcEr6ZU-kBQgQ2c42au1H2YUFuROIhILQ:e:1667076574:ARb-A-Wjmttc78ZmHq0
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbDhQoqeeWaxjmmmpBJWO8tzIaqP97gYxBt-JWJxLjdhgFaJpu51rMsWP6mS47s0X_MCMtyJZq_9esb5iCxb4J5rrqhSXwpvCKD-2-AyT4Njg:e:1667076641:ARaMqZgtuVcoON8_yDk
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbHd1dsb9k4flyUYOG--h7GBvzvM5X_O9E2WlqPLkAlJFn1aoR5kgdSXeiqp91JODpgSTvs4ptRTLFhk6bKC3qdsQ_utLmlkt0eUJyaYC2BRw:e:1667076762:ARbYTT_GxX_DxUOjsdY
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYJIXk94lq_KJXNGIv6zsRtVPMOgkHioHyTR0OA0CQD-rYtrjtg4FUIHZVuXU_ztpiBziVF54wpvulcqGsPZUvsPuxghIL2ltiu43i-ZKaNlQ:e:1667076836:ARYKhq9jgSDYQ9P6kfE
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARb0sgDyIJdH5Oh1QYUGBX8vf7L4CGybYnXz9UNcSvVFQESwuQRinxGTZTH3dam8abk-NczrMc1mhLqLnLXSxyEPf-TXjM6S1S5LvTIBUYxmLw:e:1667076973:ARZJQb_-jbBb2wkBuKw
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYrUfn3ZNUHV8Nv7egpkWMAIibpW4LEcgxsHUwrUoL3Au1IueWjI8t1q9Oc54TE8HCNNyrbiXxOPbxL4HgXID92OlLzr7VlbgPZLlRIxcdi8g:e:1667077040:ARbZr-Oejsby8dvOOCc
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbCIf2fa0FiC7AXzv_i4r5D0ZiEz5YqFp35RrYBQx5gX5sVjwdYlLTwMEW5Zf7IhK_zaHCPBYPQ5T5acza1sAMBkMlwd-FUG-MSvC_d9XDrww:e:1667077107:ARYroQ1jET5Rn1tTusQ
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZbOm63sW7aBl5BWnBfoc_QD7BGfSzVl2ghCuLv_hB8uNKfNywsTLkFeIx66E_LzQyKaPwjZw8W8D7cw7hfBZJrEAMwWw7BBEXhFGthv-KQjw:e:1667077181:ARZY6zoYdQDTwmW1mAw
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARb7bq5paqMT4gv2w8nqL916aLwoJINLcAp_0r_LXaILro-h2XwP79eqL0SWLk2StWHmWMF8aD6_JoAlQ7MWznvJfEooyd1U8p8IlwiRdJ3Esg:e:1667077272:ARZSdnqOAcWF0xFDy_A
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARa-OGLrHSsze7OybL2z2WunRq5ZrVQyLXuO3OglR_3zxODa4SeTnNL4BguRqxSDmUCcElB-SJM48qeYqTmRao-4gyp98AjPFd9kK7Z2P575eQ:e:1667077422:ARYRFxe9kjvgZ2D9X1s
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYiDM8MThTulY6ZYmbAcrtze6SKXLBh5CcFszer2Ra9djfdU6mY09yjE67GrcF_D1B8Z1ncWmmVElB97mQt2uEYCW6R3XF1_eoeHvzuXBkcrw:e:1667077492:ARZTwcOEWPTj7X3zAFg
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZS2XgFxV5hSaDlVYLWdhTkz9la0b37CrqM1WiuU0XUg1-0VJkkkss7JQo6nr2Tu_hejtB80fPcFQrePWNDz2JT4N_CAChy0zpP0hWwAD9mrg:e:1667077620:ARbCzHIUfmG2vedeRM0
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZ7ddXg8SIdaRZT9NhGDRjRUwxnIq98Pfy8dOeFRlyJ-cOa-ulMQfJkvQ_xMRV3WkEDt4GwkWJzXjRgYZ8IoGESpfi49EbdEqNQzcZ8-o0NYw:e:1667077687:ARb04AontFh6fRZFmCU
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARY8jDJSWHDiQTExDXRxP0E5PMcP23EHkhL8iGRJVSPxVKUmLMeRqQ_M8-39XxceBY6WKKwp0a03mJkHkPHykmM_T90gYENPg62gf3sVuOwBZA:e:1667077813:ARZI77VFbX3ZRjfW5vE
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARaI1qAlBISPJBAPcJ6FGx7cO7U6RQLRbNAzU5WjMXXqgM7AokPPpTwu94z4qRtlwktxQ5FwVDjP89CsE0XLvfJI3nxSG5WgWCHZ28UORIQvMg:e:1667078009:ARYEyN1R7e7l-SwyJ6w
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARagcwd1hWqB3rH4KhM1mV31SN4Vuk6Mck_--2YAjElmpNhWTn7gTQ-7WlHyL9DeQ0bVtDgOHxLDVzTBKsxebzaQ4qN6k3XSHfvBpAsVcgFwHA:e:1667078083:ARZSoi4HvrFIuglx2FQ
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARb1TKoMYSb0Q9RxM9ht5-vKCdkLkOZoiVE9vsA8cML1Y_7posbPnmp726X-uECw05Eb6oRPZ4-9YYFqW8wKtqIBgxwA7h2wWADxHaJkraio1A:e:1667078190:ARZKED31fnoYyQQMzxk
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZVjsNBmTFoyObaesHx1ZgZB2njCR32zvXVttjeZyiDkY9eZyypwB6bufx7IgGemM5MgNMcEuI4lPg6n3dWbcJDXQ1lt3OmL_ZN3T1wHXjP1A:e:1667078257:ARbEL6Zdn0A4ptLtukc
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARa02y7s_HfrZgAg0Xa6w6klWAxSMSYKnma3XpWSdQD0vuEk0IbZThxOGZWIWw9vjMKp6bpLW-vsrAKRfVgjdImnJXBzsWt7j6ICE48M-OhP-g:e:1667078326:ARazkoAz4dkpjtsjn1I
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARYvuO2faWczu5mFn35Nerb4dOFbK4GHpZacdOGe4ONEoLFhJm7jRXVNpAauJwg0prjxXGRfuD9e72keuX6NdS-pQHACy2W5o6vugNlaoSbJ5w:e:1667078394:ARYjEZWJXcH2ftRlOXk
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbfTnDMvWWozIW7s-ic-t44khXBNBNy38Aitaju9529CsGJqx5t6wZnDUxFtUS-CJTdGltZCsM0Fx-3A-oqOstxsISiQc0rMUQGffAu-BzMQA:e:1667078519:ARbX9hDRehg0mx6u-gw
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARbtdPhSlcSE1kGIGSK3g7JBqXxmehDxxe1Hssq9-ajy2E33hF5u9aINz_hTwXilU7lcDp9JTTtdEbUQZoMFlRPNkcbElnMnzlZyDv-xZ9nD7w:e:1667078637:ARbr4x_orsU-ju3eFyk
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARb594zk60R_4-dxd44wij5RDNzcueUQ1Y6A_qxMIM-pcLZ7_jAl5T1MgQyOrdxR2DrAosJePeaCK2UmQqvtKTMGc_gkhIrbDKHgiLwT6csNrw:e:1667078710:ARbdPm21hM9pMRVWfkc
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARZoqEJESBjd-MPXiptWvUNCM_VEVCNuXASvT-U9tOFlaxuf66kIWr8rTOdjazFevPqMgOHlYbi_h3JoG0NKPm1uC91bVHiwjtqVDCBNvtPUUQ:e:1667078782:ARZioKANYY6GmCKMiFc
3:VEVTVF9TRVNTSU9O:YXBwbGljYXRpb24vemlw:ARa9QkRjgjC8MBfZ7BbkusEXsln4uQP5aqbytk98Qyodi9-N4uKUU_Vi6z2IUx889fs4k2VUYwrNRfseus6SZt0hnRalaL6iwRb2TpIYsOjV9Q:e:1667078807:ARYxFj3HYBXusrv2EtI

I hope this gives you a clue. :wink:

ptpt commented 2 years ago

Thanks @GITNE!

The logs you shared above didn't reproduce this HTTP 412 errors. However, I've reproduced it locally by switching on/off VPN during uploading. I think we should retry on these 412 errors instead of exiting. The reason why it took long for you is because:

I guess we should revert https://github.com/mapillary/mapillary_tools/pull/571, and let you configure the retry waiting time.

JakeSmarter commented 2 years ago

I guess we should revert #571, and let you configure the retry waiting time.

I don’t know, just adding some waiting time will not solve the core issue, which is that the offset is reset to 0 on a new IP address. All I know is that other upload services/servers do not exhibit this issue whenever I upload via a web browser. This issue did not exist with Mapillary when the web uploader was still available either. So, in my understanding there is something wrong with the server not accepting a live upload token from a different IP.

Could it be a Cloudflare, Cloudfront, or the like configuration issue? My best guess is that there is some sort of network security software in front of the upload server that sends error codes on client closed connections. And, when the client tries to reestablish an existing TLS session with the same (public) session key from a different IP address the network security software says “Nah! Multiple IP addresses cannot share a TLS (public) session key” (which is not completely devoid of some security logic but imho this matter should be left to TLS to deal with that situation).

Let me see if I can perhaps reproduce a HTTP 412 error code with that upload test script.

ptpt commented 2 years ago

@GITNE the offset reset is likely that when mapillary_tools resumes uploading from a different IP, it might upload to a different host, and the offset is not shared across hosts. Hence offset 0.

Note these logs:

2022-10-24 19:40:41,313 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_test_a981130cf2b7e09f4686dc273cf7187e
2022-10-24 19:40:43,019 - DEBUG   - HTTP response 200: b'{"dc":"lla2c16","offset":10076160}'

Do you get a different dc when the offset becomes 0?

While fixing the issue with the team, the best we can do is:

  1. retry on these 412 errors immediately (no sleep wait)
  2. upload from a stable network

Note on 1: it does not solve the problem, which is uploading the same data multiple times but it makes uploading faster

JakeSmarter commented 2 years ago

@GITNE the offset reset is likely that when mapillary_tools resumes uploading from a different IP, it might upload to a different host, and the offset is not shared across hosts. Hence offset 0.

Right, I have not thought of that; then I guess the upload token is not shared either.

  1. retry on these 412 errors immediately (no sleep wait)

If the above is true then this is indeed very reasonable. There is no need to pause.

Do you get a different dc when the offset becomes 0?

Could be, I have not payed attention to this. I guess mainly because I could not figure out what that cryptic dc might stand for. Anyway, I am going to pay attention to this now and report.

JakeSmarter commented 2 years ago 
2022-11-04 23:20:43,180 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-11-04 23:20:44,530 - DEBUG   - HTTP response 200: b'{"dc":"prn3c15","offset":0}'
2022-11-04 23:20:44,531 - INFO    - Session key: TEST_SESSION
2022-11-04 23:20:44,531 - INFO    - Entity size: 1073741824
2022-11-04 23:20:44,531 - INFO    - Initial offset: 0
2022-11-04 23:20:44,532 - INFO    - Chunk size: 16.0 MB
2022-11-04 23:20:44,580 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "0", "X-Entity-Length": "1073741824", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-11-04 23:21:51,812 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-11-04 23:21:51,813 - DEBUG   - The next offset will be: 16777216
2022-11-04 23:21:51,846 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "16777216", "X-Entity-Length": "1073741824", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-11-04 23:22:48,549 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-11-04 23:22:48,550 - DEBUG   - The next offset will be: 33554432
2022-11-04 23:22:48,581 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "33554432", "X-Entity-Length": "1073741824", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-11-04 23:23:45,089 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-11-04 23:23:45,090 - DEBUG   - The next offset will be: 50331648
2022-11-04 23:23:45,122 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "50331648", "X-Entity-Length": "1073741824", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-11-04 23:24:45,094 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-11-04 23:24:45,094 - DEBUG   - The next offset will be: 67108864
2022-11-04 23:24:45,107 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "67108864", "X-Entity-Length": "1073741824", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-11-04 23:26:08,628 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-11-04 23:26:08,629 - DEBUG   - The next offset will be: 83886080
2022-11-04 23:26:08,644 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "83886080", "X-Entity-Length": "1073741824", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-11-04 23:27:05,999 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-11-04 23:27:06,000 - DEBUG   - The next offset will be: 100663296
2022-11-04 23:27:06,013 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "100663296", "X-Entity-Length": "1073741824", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-11-04 23:28:02,698 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-11-04 23:28:02,698 - DEBUG   - The next offset will be: 117440512
2022-11-04 23:28:02,713 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "117440512", "X-Entity-Length": "1073741824", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-11-04 23:29:01,134 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-11-04 23:29:01,134 - DEBUG   - The next offset will be: 134217728
2022-11-04 23:29:01,148 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "134217728", "X-Entity-Length": "1073741824", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1073, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.9/http/client.py", line 995, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.9/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.9/ssl.py", line 1173, in send
    return self._sslobj.write(data)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.9/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3.9/site-packages/urllib3/packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1073, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.9/http/client.py", line 995, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.9/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.9/ssl.py", line 1173, in send
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 103, in <module>
    main()
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 95, in main
    file_handle = service.upload(fp, initial_offset)
  File "/home/GITNE/src/GitHub/Mapillary tools/mapillary_tools/upload_api_v4.py", line 151, in upload
    resp = requests.post(
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2022-11-04 23:29:51,880 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-11-04 23:29:53,135 - DEBUG   - HTTP response 200: b'{"dc":"prn3c15","offset":134217728}'
2022-11-04 23:29:53,136 - INFO    - Session key: TEST_SESSION
2022-11-04 23:29:53,136 - INFO    - Entity size: 1073741824
2022-11-04 23:29:53,137 - INFO    - Initial offset: 134217728
2022-11-04 23:29:53,137 - INFO    - Chunk size: 16.0 MB
2022-11-04 23:29:53,174 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "134217728", "X-Entity-Length": "1073741824", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-11-04 23:31:08,562 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-11-04 23:31:08,562 - DEBUG   - The next offset will be: 150994944
2022-11-04 23:31:08,593 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "150994944", "X-Entity-Length": "1073741824", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1073, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.9/http/client.py", line 995, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.9/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.9/ssl.py", line 1173, in send
    return self._sslobj.write(data)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.9/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3.9/site-packages/urllib3/packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1073, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.9/http/client.py", line 995, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.9/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.9/ssl.py", line 1173, in send
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 103, in <module>
    main()
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 95, in main
    file_handle = service.upload(fp, initial_offset)
  File "/home/GITNE/src/GitHub/Mapillary tools/mapillary_tools/upload_api_v4.py", line 151, in upload
    resp = requests.post(
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2022-11-04 23:32:01,254 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION
2022-11-04 23:32:02,346 - DEBUG   - HTTP response 200: b'{"dc":"odn1c03","offset":0}'
2022-11-04 23:32:02,347 - INFO    - Session key: TEST_SESSION
2022-11-04 23:32:02,347 - INFO    - Entity size: 1073741824
2022-11-04 23:32:02,348 - INFO    - Initial offset: 0
2022-11-04 23:32:02,348 - INFO    - Chunk size: 16.0 MB
2022-11-04 23:32:02,383 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "0", "X-Entity-Length": "1073741824", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
2022-11-04 23:33:00,330 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-11-04 23:33:00,330 - DEBUG   - The next offset will be: 16777216
2022-11-04 23:33:00,356 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/TEST_SESSION HEADERS {"Offset": "16777216", "X-Entity-Length": "1073741824", "X-Entity-Name": "TEST_SESSION", "X-Entity-Type": "application/zip"}
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1073, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.9/http/client.py", line 995, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.9/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.9/ssl.py", line 1173, in send
    return self._sslobj.write(data)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.9/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3.9/site-packages/urllib3/packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1073, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.9/http/client.py", line 995, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.9/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.9/ssl.py", line 1173, in send
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 103, in <module>
    main()
  File "/home/GITNE/src/GitHub/Mapillary tools/tests/cli/upload_api_v4.py", line 95, in main
    file_handle = service.upload(fp, initial_offset)
  File "/home/GITNE/src/GitHub/Mapillary tools/mapillary_tools/upload_api_v4.py", line 151, in upload
    resp = requests.post(
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Indeed, the offset resets on dc change.

JakeSmarter commented 2 years ago

v0.9.5 still resets to 0 on dc change:

2022-11-09 12:22:47,489 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_c6b206f8deb6c6db838575a9eef8a2c7.zip HEADERS {"Offset": "184549376", "X-Entity-Length": "800369843", "X-Entity-Name": "mly_tools_c6b206f8deb6c6db838575a9eef8a2c7.zip", "X-Entity-Type": "application/zip"}
2022-11-09 12:22:48,815 - WARNING - Error uploading chunk_size 16777216 at begin_offset 0: SSLError: SOCKSHTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_c6b206f8deb6c6db838575a9eef8a2c7.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2393)')))
2022-11-09 12:22:48,815 - INFO    - Retrying in 2 seconds (1/200)
2022-11-09 12:22:50,819 - DEBUG   - GET https://rupload.facebook.com/mapillary_public_uploads/mly_tools_c6b206f8deb6c6db838575a9eef8a2c7.zip
2022-11-09 12:22:52,022 - DEBUG   - HTTP response 200: b'{"dc":"odn1c04","offset":0}'
2022-11-09 12:22:52,023 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 1, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_f2431491ec6cf97213cac3f32b26f387.zip', 'sequence_image_count': 420, 'entity_size': 800369843, 'md5sum': 'c6b206f8deb6c6db838575a9eef8a2c7', 'upload_start_time': 1667992342.4282176, 'upload_total_time': 624.8997466564178, 'offset': 0, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 16777216, 'upload_last_restart_time': 1667992972.023177}
2022-11-09 12:22:52,037 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_c6b206f8deb6c6db838575a9eef8a2c7.zip HEADERS {"Offset": "0", "X-Entity-Length": "800369843", "X-Entity-Name": "mly_tools_c6b206f8deb6c6db838575a9eef8a2c7.zip", "X-Entity-Type": "application/zip"}
2022-11-09 12:24:59,870 - DEBUG   - HTTP response 206: b'{"debug_info":{"retriable":true,"type":"PartialRequestError","message":"Partial request (did not match length of file)"}}'
2022-11-09 12:24:59,870 - DEBUG   - The next offset will be: 16777216
2022-11-09 12:24:59,871 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 1, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_f2431491ec6cf97213cac3f32b26f387.zip', 'sequence_image_count': 420, 'entity_size': 800369843, 'md5sum': 'c6b206f8deb6c6db838575a9eef8a2c7', 'upload_start_time': 1667992342.4282176, 'upload_total_time': 624.8997466564178, 'offset': 16777216, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 16777216, 'upload_last_restart_time': 1667992972.023177}
2022-11-09 12:24:59,884 - DEBUG   - POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_c6b206f8deb6c6db838575a9eef8a2c7.zip HEADERS {"Offset": "16777216", "X-Entity-Length": "800369843", "X-Entity-Name": "mly_tools_c6b206f8deb6c6db838575a9eef8a2c7.zip", "X-Entity-Type": "application/zip"}
ptpt commented 2 years ago

Yeah, the client can't do much with this issue, it has to be fixed at the server side.

JakeSmarter commented 2 months ago 
$ mapillary_tools --version
mapillary_tools version 0.11.2

So basically this issue persists. This time, I have a different but similar setup. I upload via a cell phone network. It is sloooow :snail: but it basically works. The issue for me is now that the zip file to upload is huge (like gigabytes in size). During the day, I need my phone for daily use and cannot use it for upload but I can use it to upload during night hours. Whenever I tether to my desktop computer :desktop_computer: (where I do image and metadata processing before upload) to start or continue uploading I usually get a new public IP address. @ptpt It would be nice to have the upload session live for at least like 24 hours and could be resumed independent of the client’s public IP address.

$ MAPILLARY_TOOLS_UPLOAD_CHUNK_SIZE_MB='0.125' mapillary_tools upload_zip .
Uploading ZIP mly_tools_502bc4d7ccad1116cddc81e133534f73.zip (1/1):   4%|████▉                                                                                                                        | 83.0M/2.05G [18:03<78:02:40, 7.53kB/s]
Traceback (most recent call last):
  File "/usr/local/bin/mapillary_tools", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/mapillary_tools/commands/__main__.py", line 168, in main
    args.func(argvars)
  File "/usr/local/lib/python3.10/dist-packages/mapillary_tools/commands/upload_zip.py", line 30, in run
    upload.upload(
  File "/usr/local/lib/python3.10/dist-packages/mapillary_tools/upload.py", line 707, in upload
    _upload_zipfiles(mly_uploader, zip_paths)
  File "/usr/local/lib/python3.10/dist-packages/mapillary_tools/upload.py", line 863, in _upload_zipfiles
    cluster_id = mly_uploader.upload_zipfile(
  File "/usr/local/lib/python3.10/dist-packages/mapillary_tools/uploader.py", line 132, in upload_zipfile
    return self.upload_stream(
  File "/usr/local/lib/python3.10/dist-packages/mapillary_tools/uploader.py", line 221, in upload_stream
    return _upload_stream(
  File "/usr/local/lib/python3.10/dist-packages/mapillary_tools/uploader.py", line 400, in _upload_stream
    file_handle = upload_service.upload(fp, offset=begin_offset)
  File "/usr/local/lib/python3.10/dist-packages/mapillary_tools/upload_api_v4.py", line 142, in upload
    resp = requests.post(
  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 402, in request
    self.send(chunk)
  File "/usr/lib/python3.10/http/client.py", line 999, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.10/ssl.py", line 1266, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib/python3.10/ssl.py", line 1235, in send
    return self._sslobj.write(data)
KeyboardInterrupt
Uploading ZIP mly_tools_502bc4d7ccad1116cddc81e133534f73.zip (1/1):   4%|████▉                                                                                                                        | 83.0M/2.05G [18:04<85:16:11, 6.89kB/s]

Ctrl+C at end of first night. Continue in the evening.

$ MAPILLARY_TOOLS_UPLOAD_CHUNK_SIZE_MB='0.25' mapillary_tools upload_zip .
Uploading ZIP mly_tools_502bc4d7ccad1116cddc81e133534f73.zip (1/1):   0%|▌                                                                                                                            | 9.25M/2.05G [21:09<80:02:45, 7.61kB/s]
2024-09-09 13:28:42,838 - WARNING - Error uploading chunk_size 262144 at begin_offset 0: SSLError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_7c406feee46ec832ecfdc0c5d6507794.zip (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2426)')))
2024-09-09 13:28:42,847 - INFO    - Retrying in 2 seconds (1/200)
Uploading ZIP mly_tools_502bc4d7ccad1116cddc81e133534f73.zip (1/1):   0%|▌                                                                                                                            | 9.25M/2.05G [21:15<80:02:57, 7.61kB/s]
Uploading ZIP mly_tools_502bc4d7ccad1116cddc81e133534f73.zip (1/1):   1%|▊                                                                                                                            | 13.2M/2.05G [31:50<79:52:47, 7.61kB/s]

As you can see the upload session resets. The uploading :desktop_computer: does not reboot. A 16 MB chunk size works on a slow connection too but is way too much to recover on any network errors. Furthermore, as you can see TLS can get some errors. Next time, I am going to switch from the default GnuTLS implementation to OpenSSL. Maybe this is going to make uploads work more reliably.

JakeSmarter commented 2 months ago

Sometimes, the phone can switch between different cell network operators (because of weather conditions etc), which can also lead to public IP address changes. @ptpt you really have to finally get this resolved because otherwise it becomes impossible to complete uploads! You just cannot assume in a crowd sourced undertaking that every contributor is going to have a stable high bandwidth connection, like you have in your office. You have to take all of this into consideration.

JakeSmarter commented 2 months ago

For now, I reroute my upload traffic to a public static IP address but this is only an ugly workaround. Voluntary contributors who upload stuff for free should not need to go to these lengths to upload images. You have to fix this!

JakeSmarter commented 1 month ago

:frowning_face: The upload session lifetime is too short. :cry: And, I had almost 9%! I won’t upload any further, unless you fix this.

JakeSmarter commented 1 month ago

@nickplesha Can you fix this on the server side too? You do not have to read everything above, you can start reading since https://github.com/mapillary/mapillary_tools/issues/569#issuecomment-2337974674. Thank you!

ptpt commented 1 month ago

I'll discuss with the team and get back.

JakeSmarter commented 1 month ago

There is an alternative to modifying network settings on the server (if that is not an option to you): Like before, contributors could upload a sequence in batches and the backend would join them together into a sequence by virtue of the SequenceUUID. Note that because SequenceUUID (and PhotoUUID) is de facto fixed length binary data, its data type should or could optionally be a base64 string rather than (only) the human readable UUID hex notation.

{
  "SequenceUUID": "01234567-89AB-CDEF-0123-456789ABCDEF"
}
{
  "SequenceUUID": "ASNFZ4mrze8BI0VniavN7w"
}
JakeSmarter commented 1 month ago

So, I read through the issue posts again in order to better understand what is going on.

It comes down to: When a client wants to resume an interrupted upload from the same IP address then it gets the same bucket from before and things work as expected. But, when the client wants to resume from a different IP address then it gets another bucket (or there is a great chance of happening so). And, because upload session state is not shared between buckets, then in the later case, it counts as a new upload session. :bulb: Well that, and there is this issue with the upload session’s lifetime on long upload pauses.

JakeSmarter commented 4 weeks ago

Did you change anything on the server side?

Yesterday, I tried to upload a large zip file via a static IP address and it went fine until it reset to 0 after about 24 hours. So, what I meant by upload session lifetime of about 24 hours was that the lifetime timer should reset to 0 whenever a packet or chunk (whatever) is received. I am not sure whether you really need a total lifetime timer. Though, I am aware that long running uploads may pose some security threat but this can easily be mitigated by limiting the number of concurrent uploads per remote IP address. Note that limiting the number of concurrent upload sessions per IP address to one may block other clients from uploading which are behind a NAT on the same public IP address.