search

biopython / biopython.github.io

Source of biopython.org website, migrated from MediaWiki
biopython.org
154 stars 283 forks source link

Providing all the past releases via GitHub Pages #7

Open peterjc opened 8 years ago

peterjc commented 8 years ago

PyPI has our (recent?) *.tar.gz releases, and in future we'd like to do wheels there too.

However, can we and should we simply host all the old *.tar.gz, *.zip, *.exe and recently *.msi files via GitHub pages?

peterjc commented 8 years ago

I've created https://github.com/biopython/DIST which will handle the DIST/ folder as a separate repository; this seems more elegant than forcing everything under https://github.com/biopython/biopython.github.io

(I will now rewrite the history here to remove the DIST/ files)

peterjc commented 8 years ago

I've uploaded the most recent Windows installers to https://github.com/biopython/DIST - taking the time to double check their MD5 checksums and note this in the commit comments. The good news is the download via GitHub pages seems to work fine:

e.g. https://biopython.github.io/DIST/biopython-1.66.win32-py3.5.msi

vincentdavis commented 8 years ago

@peterjc I can try working on migrating the remaining installers.

MarkusPiotrowski commented 8 years ago

I'm actually working on the Download page. @peterjc can you go on uploading more releases, maybe starting with the more recent ones?

peterjc commented 8 years ago

I've asked on the mailing list for help confirming checksums before posting the remaining releases:

peterjc commented 8 years ago

Progress: I think all the *.tar.gz files are back, thanks to community validation of checksums and the Gentoo packaging manifest records.

MarkusPiotrowski commented 8 years ago

Can we use the validated *.tar.gz files to generate the *.zip files? Is this a simple conversion or is the content different?

peterjc commented 8 years ago

Good idea. I always assumed the *.zip files might be built with Windows line endings, but no - probably as these were always produced together on the same (Linux) machine, their contents seem to be the same.

Based on a single test:

$ diff -r biopython-1.61-tar-gz/ biopython-1.61-zip/ && echo "Returned $?"
Returned 0

OK, that ought to let me verify the contents of all the *.zip files against the previously validated *.tar.gz files.

peterjc commented 8 years ago

Other than having biopython-1.21/Tests/*.pyc in biopython-1.21.zip all the zip file contents seem to match the tar-balls :)

https://github.com/biopython/DIST/commit/1a6013af7d29d2fcc32f7db0113b10167ebf88ae

If anyone is interested, the quick-and-dirty comparison script I used was:

import os
import sys

def untar(tar, folder):
    assert os.path.isfile(tar)
    if os.path.isdir(folder):
        print("Assuming %s already untarred to %s" % (tar, folder))
        return
    os.mkdir(folder)
    cmd = 'tar -zxvf "%s" -C "%s"' % (tar, folder)
    print(cmd)
    rc = os.system(cmd)
    if rc:
        sys.exit("Error %i from %s" % (rc, cmd))

def unzip(zip, folder):
    assert os.path.isfile(zip), zip
    if os.path.isdir(folder):
        print("Assuming %s already unzipped to %s" % (zip, folder))
        return
    os.mkdir(folder)
    cmd = 'unzip "%s" -d "%s"' % (zip, folder)
    print(cmd)
    rc = os.system(cmd)
    if rc:
        sys.exit("Error %i from %s" % (rc, cmd))

def tar_vs_zip(tar, zip, stem):

    print("%s vs %s" % (tar, zip))

    assert tar == stem + ".tar.gz"
    tar_folder = "/tmp/%s-tar-gz" % stem
    untar(tar, tar_folder)

    assert zip == stem + ".zip"
    zip_folder = "/tmp/%s-zip" % stem
    unzip(zip, zip_folder)

    cmd = 'diff -r "%s" "%s"' % (tar_folder, zip_folder)
    rc = os.system(cmd)
    if rc:
        sys.exit("Error %i from %s" % (rc, cmd))

    cmd = 'git add "%s" "%s"' % (tar, zip)
    rc = os.system(cmd)
    if rc:
        sys.exit("Error %i from %s" % (rc, cmd))
    md5.extend([tar, zip])

md5 = []
for tar in os.listdir("."):
    if not tar.endswith(".tar.gz"):
        continue
    stem = tar[:-7]
    zip = stem + ".zip"
    if not os.path.isfile(zip):
        continue
    tar_vs_zip(tar, zip, stem)

cmd = "md5 %s" % " ".join(md5)
os.system(cmd)

cmd = "shasum -a 256 %s" % " ".join(md5)
os.system(cmd)

Plus use of rm /tmp/biopython-1.21-zip/biopython-1.21/Tests/*.pyc to get the diff to pass.

peterjc commented 8 years ago

Another contributor pointed out that archive.org actually does record binary files, and I was able to download almost all of the past releases from them and compare them to the files from the old web server - and everything matched: https://github.com/biopython/DIST/commit/d5f7b010e345eaf27e6a09d2dc22374098127ba7

We're still missing verification for just a handful of files:

peterjc commented 8 years ago

Just FYI, GitHub is allowing us to host the installers, but does grumble about it:

The page build completed successfully, but returned the following warning:

It looks like you're using GitHub Pages to distribute binary files. We strongly suggest that you use releases to ship projects on GitHub. Releases are GitHub's way of packaging and providing software to your users. You can think of it as a replacement to using downloads to provide software. We found the following file(s) which may be a good candidate for releases: biopython-1.65.win32-py2.6.exe, ... For more information, see https://help.github.com/articles/about-releases/.

For information on troubleshooting Jekyll see:

https://help.github.com/articles/troubleshooting-jekyll-builds

If you have any questions you can contact us by replying to this email.

I'm expecting in future we won't need to host new releases on http://biopython.org/DIST/ and can use PyPI instead.

peterjc commented 8 years ago

Biopython 1.62 and 1.63b are not currently linked to from the old releases section downloads page. They were publicly announced via http://lists.open-bio.org/pipermail/biopython/2013-August/014917.html and http://lists.open-bio.org/pipermail/biopython/2013-November/014995.html so I think it was just an oversight in bfbdb5aceaca8d3347c46cfea8e6c1cca88da3e6 and bfbdb5aceaca8d3347c46cfea8e6c1cca88da3e6