Improve setup

[felipedau]

Feedback from @david415 suggested changes to setup.py and use [dev-]requirements.txt files. It should be updated based on sphinxmixcrypto's setup.py.

[felipedau]

Hi @david415 and @meejah, I just pushed a few changes to my develop branch to use _metadata.py and the requirements[-dev].txt files (bae25362cc49ac2f6edfac79b6f9ca5d7ccc0424 and 834757df6c20d517d443913d73b0c33fc20888da).

Some questions I have:

Is it important to have __version__ in _metadata.py? I have not found a nice way to do that when using Versioneer - didn't try too hard though.

I like that way crossbario/crossbar named their requirement files, so I appended the extra keyword instead. Is that okay?

To what versions should I pin the requirements? I think it would be better to pin the lowest version that unMessage needs, but at the same time I am not sure what it might cause to the user's environment and the packages might get old. Should we instruct users to use a virtualenv?

This is not too related to this task (more related to #5 and meejah/txtorcon#229), but: do you know the lowest pip and setuptools versions we could ask users to have? Hash checking has been introduced in pip 8.0, so maybe we shouldn't even consider the ones < 8.0. Then I could check what was setuptool's current version when pip 8.0 was released and pick that one as well. Is there an automated way to test multiple versions to find the lowest one?

I think that if we are going to use hash checking, we probably do not trust the infrastructure providing the packages. That said, the right way to do this is finding an authenticated way to check them with the authors, right? Or maybe just get from multiple sources and TOFU to prevent a future attack, which might be better than not checking?

Let me know what you think.

Thanks!

[meejah]

I'm inclined more towards "pin against the latest versions" of everything.

As to getting "authenticated" hashes, unfortunately that's still an "exercise for the reader" until something like https://theupdateframework.github.io/ is rolled out on PyPI.

Meanwhile, basically the only way is to satisfy yourself "somehow"; for packages that provide signatures you can check them of course. For packages lacking signatures yeah I'd say all you can do is TOFU :/

[felipedau]

I'm inclined more towards "pin against the latest versions" of everything.

Alright, I'll make sure to update the versions on every release. The only concern I have here is pinning a version of a package that is also a dependency of another package (currently installed in the user's machine) and then break that package because this new dependency has some kind of incompatibility. Does that make sense or is it already handled? That's why I think we should suggest that users use virtualenvs.

As to getting "authenticated" hashes, unfortunately that's still an "exercise for the reader" until something like https://theupdateframework.github.io/ is rolled out on PyPI.

Got it. Thanks, TUF is indeed very interesting!

Meanwhile, basically the only way is to satisfy yourself "somehow"; for packages that provide signatures you can check them of course. For packages lacking signatures yeah I'd say all you can do is TOFU :/

Alright!

And... "do you know the lowest pip and setuptools versions we could ask users to have?"

Thanks!

[meejah]

Yes, users should use virtualenvs. For "system packages" (e.g. if unmessage got Debian packaged) then it's their problem to resolve the dependencies .. so having some notion of "the minimum required" would have some value here.

I don't know what the absolute minimum pip/setuptools that e.g. txtorcon requires is, but it's definitely "newer than what Debian ships right now". Demanding at least 8.0 to get the hash-checking is totally reasonable (and Debian/Ubuntu users are going to have to upgrade their pip anyway). The advice I usually give here is:

1. apt-get install python-virtualenv
2. create virtualenv using system-supplied package.
3. immediately pip install --upgrade pip setuptools in all new virtualenvs

[felipedau]

Cool! Thanks :)

[felipedau]

Hey @meejah, I talked to @david415 and we decided to make the hash checking an extra task (described in #49) in case we have more time available later.

I made the changes we discussed here in #48 and closed the task. Let me know if I missed anything and what you think about #49.

Thanks!