Partial downloaded

[sjperkins]

@smasoka Lets have a think about this. The question raised in the comment is:

• If the file is already in the cache, can we assume that it's hash is correct?

I suspect we may be able to by changing the code. The reasons being that:

• We download into a file with the .partial extension. See here and here.
• If the download completes, then the file with the .partial extension is renamed to the actual filename. See here for example.

It might make more sense to do the hash check before the move. Then we know that if the file (without the .partial extension) exists, it has the right hash. One could also use an exception to indicate when this fails. e.g.

   class FileHashMismatch(ValueError):
       pass

    ...

    with open_and_hash_file(part_filename) as (size, sha256hash, f):
        if size > 0:
            log.info("Resuming download of %s at %d", filename, size)
            # Some of this file has already been downloaded
            # Request the rest of it
            total_size = int(response.headers['Content-Length'])
            response.close()
            headers = {'Range': 'bytes=%d-%d' % (size, total_size)}
            response = session.get(url, params=params,
                                   headers=headers, stream=True)

        if response.status_code != 416:
            try:
                response.raise_for_status()
            except requests.exceptions.HTTPError as e:
                raise Exception(e)

            for chunk in response.iter_content(CHUNK_SIZE):
                if chunk:  # filter out keep-alive new chunks
                    sha256hash.update(chunk)
                    f.write(chunk)

    # Changes start here
    sha256hash = sha256hash.hexdigest()

    if sha256hash != entry['hash']:
        raise FileHashMismatch(f"{sha256hash} doesn't match {entry['hash'}")

    shutil.move(part_filename, filename)
    return sha256hash

[smasoka]

Yeah, I agree and code in download handles that well. The assumption that if the filename is there (under cache) than its complete should stand.

I am testing the partial part of the code and I get this, I'm just adding that and testing

[sjperkins]

I am testing the partial part of the code and I get this, I'm just adding that and testing

Nice catch.I suspect that there are going to be more of these subtleties.