How to handle packages that Spack cannot download?

[eschnett]

I want to package CosmoMC. To download, you have to enter your email address on a web site and accept a licence. A download url is then emailed to you. How do I tell Spack about this? I assume I will download the tarball manually and then somehow point Spack to it.

The packaging guide has a section on "licensed software" https://spack.readthedocs.io/en/latest/packaging_guide.html#licensed-software, but this seems to be something slightly different, namely software that requires a license to install or run.

If this is already documented, is there a particular keyword for which I should search?

[alalazo]

@eschnett You can create a mirror and put the tarball inside it. spack will try to fetch from mirrors before trying download urls.

[adamjstewart]

It isn't really documented, but @alalazo is right. See http://spack.readthedocs.io/en/latest/mirrors.html for details on how to create a mirror.

Check out the following packages for software that behaves similarly to yours:

• converge
• cuda
• daal
• intel
• intel-parallel-studio
• ipp
• mkl
• pgi
• turbomole
• vizglow

The unofficial standard that we started using was to use a URL like:

url = "file://%s/pgi-16.3.tar.gz" % os.getcwd()

That way, Spack can find the tarball if it is in your current directory, or in a mirror. Also, feel free to copy the blurb in the docstring of any of these packages that explains how to download it and create a mirror.

[citibeth]

I added the "documentation" tag to this. Someday, someone will make (another) sweep through open Issues and implement all the "documentation"-tagged issues.

[citibeth]

Closing, as this has been moved to #2505.

[citibeth]

The decision was to just leave this as an open thread until it makes its way into the docs.

[markcmiller86]

So, related to this...what if the owners of a package host it in such a way that automatic download is not possible (e.g. behind some kind of human-in-the-loop registration process)? An example is Chombo, https://anag-repo.lbl.gov/chombo-3.2/access.html.

Turns out a number of people out there in the wild have turned around and are now hosting Chombo on github. There are like 20-30 copies of it there! Most are prestine copies of the official subversion repo.

Is there anything terribly wrong with having Spack side-step the official site and using one of those? I suppose for security's sake, we could add logic to tar up and checksum the lot to ensure it matches what we know is the official code.

[citibeth]

Is there anything terribly wrong with having Spack side-step the official site and using one of those?

Spack wants to cultivate the goodwill of upstream authors. It therefore needs to respect those authors' wishes on how their stuff is downloaded.

However, somebody might wish to contact Chombo authors and share with them how non-automatic downloads hurts the user experience with Spack. Upstream authors have an incentive to make it as easy as possible to install their software.

That said, the solution for now is to manually download the tarball and place it in a manually created Spack mirror. You only have to do it once, then everything else "just works."

Is there anything terribly wrong with having Spack side-step the official

site and using one of those? I suppose for security's sake, we could add logic to tar up and checksum the lot to ensure it matches what we know is the official code.

Or we could host it ourselves. But the point remains that this disrespects the wishes of the upstream authors.

Some authors have a manual download because (a) they didn't think it would be a burden, and (b) they want to keep track of who is using their software. Maybe Spack could offer a standardized / automatable way of doing that. When you install Spack, you would give it basic info about who you are. And then Spack would share that, in an automated way, when downloading packages like Chombo.

[adamjstewart]

Is there anything terribly wrong with having Spack side-step the official site and using one of those?

I have to admit, I've done this once for the stream package. The official downloads page is https://www.cs.virginia.edu/stream/FTP/Code/ but there is no single tarball to download from. I asked the developer if there was any chance he could create a tarball, but he didn't seem to understand why I couldn't just download each file individually like everyone else. Luckily I found a mirror https://github.com/jeffhammond/STREAM.git and was able to use that. I mentioned this to the original author and he didn't seem upset.

It's always a good idea to contact the authors and see if they have a solution. But if you have a deadline to meet, I won't hold it against you if you use a mirror. We can always switch it later.

[alalazo]

The question here seems to be answered. Should we close this issue?

[markcmiller86]

Hmm. Not sure I actually know what the answer is ;) Before closing, should we summarize the outcome?

[alalazo]

See #2648 for a particular case related to this that should be better documented.

[alephpiece]

It seems that spack could fail on finding a tarball in mirrors when the tarball name is not in lower case. Spack complains that python-3.7.6.tgz could not be found just because the official package is named as Python-3.7.6.tgz. The log shows spack do recoganize the official name:

==> Fetching https://www.python.org/ftp/python/3.7.6/Python-3.7.6.tgz

It's just getting confusing for me.

[adamjstewart]

This seems to be even more complex now that the downloads cache uses hashes, see #21002

[monellz]

Any update on this?

[monellz]

It seems that we can find the hash from package.py.

For example, I want spack to install llvm@18.1.7. ${SPACK_HOME}/var/spack/repos/builtin/packages/llvm/package.py shows its sha256=b60df7cbe02cef2523f7357120fb0d46cbb443791cde3a5fb36b82c335c0afc9. So the target cache dir and file is ${SPACK_HOME}/var/spack/cache/_source-cache/archive/b6/b60df7cbe02cef2523f7357120fb0d46cbb443791cde3a5fb36b82c335c0afc9.tar.gz

Just copy and rename the corresponding file and then spack can use the cache.