Some uploads to Google Drive fail.

[Angstman]

I just want you to see this issue. some but not all uploads to google drive fail. I also have a lot of "mismatch size errors" which may keep the files from uploading, too. version is .62. I had increased threads to 45 but have decreased to 25 to see if that helped. there is no big load on the server at 45 but the bandwidth is saturated sufficiently at 25. see this error and the one below:

11/12 07:59:58 [Upload 32][ERROR]: Error uploading file M:\bitlist_temp\437d099be73c807ae3a5a0f2dbf16817f4298075 Traceback (most recent call last): File "M:\BitcasaFileLister\includes\lib\gdrive.py", line 55, in upload_file return self.get_service().files().insert(body=body, media_body=media_body).execute() File "M:\Python\lib\site-packages\oauth2client\util.py", line 129, in positional_wrapper return wrapped(_args, kwargs) File "M:\BitcasaFileLister\includes\lib\googleapiclient\http.py", line 688, in execute , body = self.next_chunk(http=http, num_retries=num_retries) File "M:\Python\lib\site-packages\oauth2client\util.py", line 129, in positional_wrapper return wrapped(_args, _kwargs) File "M:\BitcasaFileLister\includes\lib\googleapiclient\http.py", line 867, in next_chunk headers=headers) File "M:\Python\lib\site-packages\oauth2client\util.py", line 129, in positional_wrapper return wrapped(_args, *_kwargs) File "M:\BitcasaFileLister\includes\lib\oauth2client\client.py", line 540, in new_request redirections, connection_type) File "M:\BitcasaFileLister\includes\lib\httplib2init.py", line 1593, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "M:\BitcasaFileLister\includes\lib\httplib2init.py", line 1335, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "M:\BitcasaFileLister\includes\lib\httplib2init.py", line 1291, in _conn_request response = conn.getresponse() File "M:\Python\lib\httplib.py", line 1067, in getresponse response.begin() File "M:\Python\lib\httplib.py", line 409, in begin version, status, reason = self._read_status() File "M:\Python\lib\httplib.py", line 373, in _read_status raise BadStatusLine(line) BadStatusLine: ''

---

11/12 08:04:08 [Download 4][ERROR]: The Last Song (2010 PG).mkv File size mismatch. Will retry 5 more times Traceback (most recent call last): File "BitcasaFileFetcher\threads\download.py", line 116, in download raise SizeMismatchError("Download size mismatch downloaded %s expected %s" % (sizecopied, size_bytes)) SizeMismatchError: Download size mismatch downloaded 734800182 expected 2440373227

[Angstman]

Another tidbit after looking at the "successfiles" in excel, i crashed python. this confirms that not much uploaded despite being on a vps with copious bandwidth. take a look:

11/12 14:00:36 [Upload 20][ERROR]: Error. Could not write to M:\bitlist_temp\successfiles.csv. Ending Traceback (most recent call last): File "M:\BitcasaFileLister\includes\helpers\results.py", line 17, in writeSuccess with codecs.open(self.successfiles, 'a', 'utf-8') as myfile: File "M:\Python\lib\codecs.py", line 878, in open file = builtin.open(filename, mode, buffering) IOError: [Errno 13] Permission denied: 'M:\bitlist_temp\successfiles.csv' 11/12 14:00:37 [Download 5][INFO]: Stopping download 11/12 14:00:38 [MainThread][INFO]: Total download time: 1.03 days 11/12 14:00:38 [MainThread][INFO]: Total upload time: 1.47 hours 11/12 14:00:38 [MainThread][INFO]: Total run time: 4.75 hours 11/12 14:00:38 [MainThread][INFO]: Download speed: 34.42 GB at 404.54 KB/s 11/12 14:00:38 [MainThread][INFO]: Upload speed: 4.35 GB at 863.35 KB/s 11/12 14:00:38 [MainThread][INFO]: Program run speed: 34.42 GB at 2.06 MB/s 11/12 14:00:40 [MainThread][INFO]: Done 11/12 14:00:40 [Download 9][INFO]: Stopping download 11/12 14:00:40 [Download 10][INFO]: Stopping download

[reallistic]

The downloadsize mismatch is a "safety" precaution for you. What that really means is bitcasa prematurely hung up the download but it should continue instead of attempting to re-download the whole thing. There is unfortunately nothing that can be done on my side for that.

From what I recall, excel "locks" the file when you open it. This means nothing else can access it during that time. If you really want to peruse it during the download, use a text editor like sublime or good ol' fashion notepad (textedit on mac). As for the upload issues, that is mostly on Google Drive's side but there is one thing you can try that may help: Change the value on line 48 of includes/lib/gdrive.py from:

media_body = MediaFileUpload(filepath, resumable=True, mimetype="", chunksize=-1)

to

media_body = MediaFileUpload(filepath, resumable=True, mimetype="")

You can also play around with different chunksizes in multiples of 1024.

[Angstman]

So I could change it to media_body = MediaFileUpload(filepath, resumable=True, mimetype="", chunksize=4096) or media_body = MediaFileUpload(filepath, resumable=True, mimetype="")

for example?

[reallistic]

@Angstman that is correct. If I remember right the latter is actually the same as:

media_body = MediaFileUpload(filepath, resumable=True, mimetype="", chunksize=1024)

[Angstman]

Ok I don't have a good way of knowing if is working other than letting it run for a while and looking at whether there is progress. if i edit the script while it is running will that cause problems?

[reallistic]

No it will not cause problems but it won't change it either. You would have to stop the script and re-run it for the changes to take effect. You can do this by types Ctrl+C, or one of the following: y, q, quit. Followed by enter.

[Angstman]

Using multiples of 1024 did not seem to work. I got upload errors. I will let it run for a few hours without a chunksize specified and report back.

[Angstman]

I wonder if the number of threads caused problems with google? I had 45 and 25 when I tried this. I finally killed it and transferred locally since there are only 2 days left I can't afford to have it go wrong. I have gdrive installed so I am going to dump the files there and have them sync to my account.

[reallistic]

It could be the number of threads by I'm not entirely sure. Please try 53342220548e82b7f90adc16decfff2e4dc81960 and let me know if that helps.

[Angstman]

ok. I will spin that up tomorrow. If i recall correctly if I kill the process while i have files moving from the temp directory to the final destination they will have to be redownloaded. I had all those files queued to upload in the temp directory and with your instruction i was able to save them locally. but alas they are still "moving".

[reallistic]

Closing based on https://github.com/google/google-api-python-client/issues/36

Also see http://stackoverflow.com/questions/17180433/google-mirror-api-throwing-badstatusline-exception-python