Add support for verifying downloaded file's SHA-512 hash

EricRahm commented 9 years ago

Each build has an associated <product>...<locale><platform>.checksums file. The SHA-512 hash contained in the file could be used to verify the hash of the download.

Proposal:

Add a |--verify| flag
Download the checksums file and verify the the SHA-512 hash for the download matches
If the hash fails follow the standard retry logic already in place

whimboo commented 9 years ago

@EricRahm I assume this hash only applies to the installer binaries to download, but not all the other files including the .zip file for Windows.

EricRahm commented 9 years ago

@whimboo There's a hash for every build file. For example: https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/01/2015-01-29-03-02-02-mozilla-central/firefox-38.0a1.en-US.win64-x86_64.checksums includes both the installer and the zip.

whimboo commented 9 years ago

Ah, great. Thanks. Sounds like a good improvement.

parthbakshi commented 9 years ago

I would like to work on this bug, whats the best way to work on it.

EricRahm commented 9 years ago

@whimboo Do you have any notes for @parthbakshi?

whimboo commented 9 years ago

Hi @parthbakshi. Great to hear that you have interests to work on this issue! Sorry for the late response but I was away for a full month.

So the best way is to fork the repository and create a local clone of it. Then you should make yourself familiar with the different kinds of Firefox builds (Nightly, Aurora, Beta, Release, candidate builds, tinderbox builds) which are supported by mozdownload. Once done it would be good to know how this checksum file looks like for those different kinds of builds. After that the download() method of the scraper would have to be extended to being able to also get the checksum file, parse the entries, and do the checks if specified via the command line (--verify).

Let me know if you have further questions. You can also reach me on the moznet IRC network in the #automation channel.

parthbakshi commented 9 years ago

Ok i have forked the repo, Is there a doc where i can read about the various builds and how the checksums/SHA-512 actually look?

whimboo commented 9 years ago

The different builds are also called channels. You can get short introductions on MDN. Maybe by following the links you will get more comprehensive information. For the checksum you might want to have a look at the hashlib module documentation of Python.

parthbakshi commented 9 years ago

Whenever i run the mozdownload on my machine im getting an error executing it, is there anything that i am missing?

(venv)pbakshi@parthbakshi:~/personal-projects/mozilla/mozdownload$ ../venv/bin/mozdownload --version=latest
Traceback (most recent call last):
  File "../venv/bin/mozdownload", line 9, in <module>
    load_entry_point('mozdownload==1.14', 'console_scripts', 'mozdownload')()
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/pkg_resources/__init__.py", line 546, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/pkg_resources/__init__.py", line 2666, in load_entry_point
    return ep.load()
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/pkg_resources/__init__.py", line 2339, in load
    return self.resolve()
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/pkg_resources/__init__.py", line 2345, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozdownload-1.14-py2.6.egg/mozdownload/__init__.py", line 5, in <module>
    from scraper import *
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozdownload-1.14-py2.6.egg/mozdownload/scraper.py", line 19, in <module>
    import mozlog
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozlog-2.11-py2.6.egg/mozlog/__init__.py", line 19, in <module>
    import structured
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozlog-2.11-py2.6.egg/mozlog/structured/__init__.py", line 5, in <module>
    import commandline
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozlog-2.11-py2.6.egg/mozlog/structured/commandline.py", line 10, in <module>
    from structuredlog import StructuredLogger, set_default_logger
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozlog-2.11-py2.6.egg/mozlog/structured/structuredlog.py", line 14, in <module>
    from logtypes import Unicode, TestId, Status, SubStatus, Dict, List, Int, Any
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozlog-2.11-py2.6.egg/mozlog/structured/logtypes.py", line 97
    known_kwargs = {name: value for name, value in kwargs.iteritems()
                                  ^
SyntaxError: invalid syntax

whimboo commented 9 years ago

As mentioned on IRC this is most likely Python version 2.6 which is causing those problems. Please upgrade to latest 2.7 and it should work.

whimboo commented 8 years ago

@parthbakshi, given the amount of time which has been passed by here, I assume you will not work on this issue anymore? It's somewhat important for us now given that in the last days we had a couple of corrupted downloads without seeing a failure from mozdownload.

parthbakshi commented 8 years ago

Sorry work got in the way you can reassign it to someone else.

On Thu, Oct 29, 2015 at 6:03 PM Henrik Skupin notifications@github.com wrote:

@parthbakshi https://github.com/parthbakshi, given the amount of time which has been passed by here, I assume you will not work on this issue anymore? It's somewhat important for us now given that in the last days we had a couple of corrupted downloads without seeing a failure from mozdownload.

— Reply to this email directly or view it on GitHub https://github.com/mozilla/mozdownload/issues/241#issuecomment-152341042 .

whimboo commented 8 years ago

Thanks for letting me know!

haouech commented 8 years ago

Hello, I'm wondering, we need to pass a checksum as a parameter for the download() method and after the _download() has finished we need to generate and compare hashes?

whimboo commented 8 years ago

So for each platform we have a checksums file located right next to the builds. This file contains checksums for all files related to the platform. So we should check first if such a file is present, download it, and then we can generate the hash locally and compare it with the one from the checksums file.

haouech commented 8 years ago

I would like to work on this issue

whimboo commented 8 years ago

Sure, go ahead and let us know how you proceed and where you need help. Don't hesitate to ask your questions on IRC. You can find me in the #automation channel. Thanks.

Hyperion101010 commented 6 years ago

is it still open i would like to work on it

whimboo commented 6 years ago

The active PR for this issue is https://github.com/mozilla/mozdownload/pull/454. But we haven't gotten a reply from @haouech for a long time. Maybe you could reply so we can open up this issue again for someone else to work on? Thanks.

Hyperion101010 commented 5 years ago

hi there @whimboo for the #454 issue as you said :you were going to distribute the code in pieces like write it systematically i would be happy to help just two things 1)can you give me the names of files responsible for check_sum in the mozilla build , also a short intro on what are you planning like a short set-up will be quite helpful 2)and yeah open the issue let's work together (crytography has always been a fantasy)

sayanmondal2098 commented 5 years ago

I would like to work on this issue

shanudjn commented 5 years ago

I am new here . I want to work on this bug . Looking for mentorship to get started. Thank you.

m-agboola commented 5 years ago

@whimboo Is this issue still open? If Yes, I'd like to work on it

whimboo commented 5 years ago

It turned out that this particular issue was too complicated to get fixed in the past as mentored issue. As such I will remove the mentored flag.

kaushiksk commented 4 years ago

Hey @whimboo I'd like to take a dig at this. Seemed like @rei2hu got most of the things right from what I can see in his PR. I have one question, are all checksum file names similar to what was mentioned earlier in this thread? i.e. extension of the release file replaced with .checksum or is there a specific format followed for each release? Any resource to help me with this?

mozilla / mozdownload

Add support for verifying downloaded file's SHA-512 hash #241