Uploading API is slow, sometimes replies with errors

SviMik commented 3 years ago

No matter how many servers I use, I'm able to upload only 250k images per day, or 5MB/s in total. Judging by the fact that creating a sequence takes sometimes up to 14 seconds it is likely that the problem is the database server performance rather than bandwidth.

Sometimes I receive errors from the API, here is my collection:

{"status":{"apiCode":400,"apiMessage":"An exception occurred while executing \u0027INSERT INTO osv_photos(`status`,osv_photos.`visibility`,osv_photos.`lat`,osv_photos.`lng`,osv_photos.`sequence_id`,osv_photos.`sequence_index`,osv_photos.`headers`,osv_photos.`projection`,osv_photos.`shot_date`,osv_photos.`name`,osv_photos.`filesize`,osv_photos.`filepath`,osv_photos.`filemd5`,osv_photos.`width`,osv_photos.`height`,osv_photos.`storage`,osv_photos.`auto_img_processing_result`,osv_photos.`coord`) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,ST_GeomFromText(\u0027POINT(30.4836142 50.504962)\u0027))\u0027 with params [\u0022active\u0022, \u0022public\u0022, \u002250.504962\u0022, \u002230.4836142\u0022, 3027054, 249, \u0022352.493\u0022, \u0022PLANE\u0022, \u00222019-01-21 11:04:37\u0022, \u00223027054_8af9f_5f615bd4ce96f.jpg\u0022, 1928009, \u00222020\\\/9\\\/15\\\/{{sizeprefix}}\\\/3027054_8af9f_5f615bd4ce96f.jpg\u0022, \u002299b4e413aa6c0ea7620723e427242b2a\u0022, 4000, 3000, \u0022storage13\u0022, \u0022ORIGINAL\u00

{"status":{"apiCode":400,"apiMessage":"An exception occurred while executing \u0027UPDATE osv_photos SET `auto_img_processing_status` = ? WHERE id = ?\u0027 with params [\u0022PROCESSING\u0022, \u00221099807202\u0022]:\n\nSQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction","httpCode":400,"httpMessage":"Bad Request"}}

{"status":{"apiCode":400,"apiMessage":"Fail:\nCode: 500\nValue: Operation could not be completed within the specified time.\ndetails (if any): \ufeff\u003C?xml version=\u00221.0\u0022 encoding=\u0022utf-8\u0022?\u003E\u003CError\u003E\u003CCode\u003EOperationTimedOut\u003C\/Code\u003E\u003CMessage\u003EOperation could not be completed within the specified time.\nRequestId:0f21178c-801e-003d-0b6c-8d8bb0000000\nTime:2020-09-18T03:32:10.2933629Z\u003C\/Message\u003E\u003C\/Error\u003E.","httpCode":400,"httpMessage":"Bad Request"}}

{"status":{"apiCode":400,"apiMessage":"cURL error 56: TCP connection reset by peer (see http:\/\/curl.haxx.se\/libcurl\/c\/libcurl-errors.html)","httpCode":400,"httpMessage":"Bad Request"}}

While this error sounds like I did something wrong with the auth, in fact just repeating the exact same request again succeeds:

{"code": "client_id_differ", "message": "The provided client id differs from auth token"}

I know these messages aren't particularly useful, but sometimes the API just gives a 502 or 504 error:

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>Microsoft-Azure-Application-Gateway/v2</center>
</body>
</html>

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Bad Gateway</title>
</head><body>
<h1>Bad Gateway</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
</p>
</body></html>

<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

tigren commented 3 years ago

@SviMik may I know how do you upload the images? with a script?

SviMik commented 3 years ago

I wrote my own uploading program using your API documentation and some examples I found on the Internet.