search

Duke-GCB / DukeDSClient

Command line program to allow uploading, downloading, and managing projects in the duke-data-service.
MIT License
5 stars 6 forks source link

User received TooLargeChunkDownloadError exception #214

Closed johnbradley closed 6 years ago

johnbradley commented 6 years ago

User was downloading a project via ddsclient download -p <projectname> --include <filepath>.

Error:

  File ".../conda/envs/bespin/lib/python3.7/site-packages/ddsc/core/download.py", line 455, 
in _write_response_to_file
    self._on_bytes_read(len(chunk))
  File ".../conda/envs/bespin/lib/python3.7/site-packages/ddsc/core/download.py", line 464, 
in _on_bytes_read
    raise TooLargeChunkDownloadError(self.actual_bytes_read, self.bytes_to_read, 
  self.local_path)

ddsc.core.download.TooLargeChunkDownloadError: 
Received too many bytes downloading part of a file. 
Actual: 44159228 Expected: 32963486 File: <filename>

I was unable to reproduce the issue the same file downloaded fine for me. In this place in the code we are reading from a GET request with Range headers delimited to the part of the file we are downloading. So we should only receive the amount we are expecting. This exception was added to catch the unlikely case that we received more than we asked for.

johnbradley commented 6 years ago

This might could occur if we received too few bytes from an earlier GET request, which raises PartialChunkDownloadError which will then retry, in this scenario actual_bytes_read doesn't get reset which could cause this error.