jjohanson
commented
11 years ago Hello again,
I have looked some more at using 'response.raw'. As mentioned above you may end up with zip'ed files when setting 'stream=True' and using 'response.raw'. However, it turns out that the 'read()' method has an optional parameter 'decode_content' that can be set to True to have the response decoded (unzipped in this case):
response.raw.read(decode_content=True)
So the following code are using 'stream=True', and response.raw (with response.raw.read(decode_content=True) called in a loop to copy all data).
The documentation on .read() states that you can not specify an amount to read and at the same time set 'response.raw.read(decode_content=True)'. So this still leaves the question about the ability to read (very) large files. I have tried reading files up to about 600MB and I can see that the file is transferred in a single call to .read() (the while loop is only passed one). So, how to transfer really large files?
Jørgen
--- /home/jojo/Downloads/easywebdav-1.0.7/easywebdav/client.py 2012-11-13 23:31:47.000000000 +0100
+++ /usr/local/lib/python2.7/dist-packages/easywebdav-1.0.7-py2.7.egg/easywebdav/client.py 2013-02-24 09:51:50.638723255 +0100
@@ -74,7 +74,7 @@
self.session.auth = (username, password)
def _send(self, method, path, expected_code, **kwargs):
url = self._get_url(path)
- response = self.session.request(method, url, allow_redirects=False, **kwargs)
+ response = self.session.request(method, url, allow_redirects=False, stream=True, **kwargs)
if isinstance(expected_code, Number) and response.status_code != expected_code \
or not isinstance(expected_code, Number) and response.status_code not in expected_code:
raise OperationFailed(method, path, expected_code, response.status_code)
@@ -124,7 +124,12 @@
response = self._send('GET', remote_path, 200)
with open(local_path, 'wb') as f:
#f.write(response.content)
- shutil.copyfileobj(response.raw, f)
+ #shutil.copyfileobj(response.raw, f)
+ line = response.raw.read(decode_content=True)
+ while line:
+ f.write(line)
+ line = response.raw.read(decode_content=True)
+
def ls(self, remote_path='.'):
headers = {'Depth': '1'}
response = self._send('PROPFIND', remote_path, (207, 301), headers=headers)
read(): http://urllib3.readthedocs.org/en/latest/helpers.html#module-urllib3.response
http://docs.python-requests.org/en/latest/user/advanced/ http://docs.python-requests.org/en/latest/api/
amnong
blootsvoets
Jay54520
Hello!
When using the latest version of easywebdav,
easywebdav 1.0.7 requests 1.1.0 webdav server: mod_dav, apache2 ubuntu 12.10
I have a problem using downloading files. When doing a download (by calling the download() method), the file is created on the local machine with the correct name, but the file is empty (zero size).
When looking at the code I can see response.raw is used as an input to shutil.copyfileobj(). However, I think that the documentation for requests states that 'stream=True' must be used in the call to session.request for .raw to be valid (http://docs.python-requests.org/en/latest/api/):
"raw = None File-like object representation of response (for advanced usage). Requires that 'stream=True' on the request."
I have tried setting stream=True (line 77 in client.py). and the received file then contains data! However, when looking at the header I can see that my webdav server gzip'ing the files, so the files downloaded have to be unszipped.
*** client-new.py 2013-02-20 08:27:29.941967542 +0100 --- client-org.py 2013-02-18 14:17:59.114443000 +0100 *************** *** 76,78 **** url = self._get_url(path) ! response = self.session.request(method, url, allow_redirects=False, stream=True, **kwargs) if isinstance(expected_code, Number) and response.status_code != expected_code \ --- 76,78 ---- url = self._get_url(path) ! response = self.session.request(method, url, allow_redirects=False, **kwargs) if isinstance(expected_code, Number) and response.status_code != expected_code \To get around this problem I have just uncommented the 'f.write(response.content)' on line 126 in client.py (and added a comment to 'shutil.copyfileobj(response.raw, f)' on line 127).
As far as I can see using response.raw and something like shutil.copyfileobj() is however necessary to be able to download really large files.