BGLibPy patch hinders reproducibility

[wvangeit]

Could you please remove / find a solution for the following patch:

https://github.com/BlueBrain/bbp-nixpkgs/blob/d4dfc8f00bdefa6ec730afc1c0c229d7460436a1/bbp/nse/bglibpy/patch_version.patch#L509

This will easily become outdated. As it is already now, it points to the wrong date and the wrong git hash. It hinders reproducibility for people that store the version number of BGLibPy in their scripts.

I guess an obviously solution would be to just use the devpi server to install bglibpy.

[adevress]

Using the devpi server as a source would not solve the issue.

The reason of being of this patch is python versioneer.

I see two way to solve that:

• Giving an option to somwhere specify manually rev, version and co and disable versioneer.
• Generate dynamically the patch out of the Nix config with the good version / rev number.

[wvangeit]

So how does nix install a pip package ? It doesn't 'pip install' it ? In the pip case versioneer should normally work, because in that case the bglibpy/_version.py should have all the info necessary (version, revision, etc), as compared to the files downloaded from git

[adevress]

Nix install from source and then call the correct installer depending of what you give : wheels, source + setup.py, eggs... or other.... In the case of the NSE python package, we install them from source, directly from git. Because this is the only thing accessible outside of BBP networks for the HBP / external installation.

[adevress]

Do you have a clean way with versioneer to tell him "use this version + git hash for this software and don't try to guess around" ? I have all the necessary information inside the Nix recipe for that.

You might disagree with that, but auto-generating version are generally a bad practice in my mind. Package managers software hate that, including Nix, RPM, DPKG, Easybuild and Spack. Because they all strip away the .git folder for reproducibility reason.

[wvangeit]

Well, if using devpi / pypi (which I would recommend, because for that version we're 100% sure all tests have passed), there is not need for the .git folder. The version is contained inside the package.

[wvangeit]

Btw, forgot to mention, the pretty well-known package 'numba' also using versioneer. It seems there is a nix package for that here (without a patch as far as I can see): https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/numba/default.nix

So is there something different between how they use versioneer (or how our system functions) ?

[adevress]

They fetch from Pypi, meaning they don't have the problem.

I can not use the internal BBP PyPi due to firewall reason like I said before. But if you can guarantee me that the official python PyPi BluePyOpt is always up to date and ok to use. It might be a good solution

[wvangeit]

Yes, maybe it makes sense to solve the firewall issues then ? I guess it would anyway useful for people to be able to pip install packages on these systems.

[adevress]

Yes, maybe it makes sense to solve the firewall issues then ?

It's political I think. Your BBP devpi contains proprietary package.

[wvangeit]

Ah, you sorry, I misread, you meant the open source pypi. For now it has to stay on bbp devpi.

But since bbpcode git is also private, we just need to make sure bbp devpi can be accessed from the same systems as bbpcode. (You know if there is a ticket somewhere for that ?)

[adevress]

But since bbpcode git is also private, we just need to make sure bbp devpi can be accessed from the same systems as bbpcode. (You know if there is a ticket somewhere for that ?)

I already discuss of that with the BBP sysadmins, after @arsenius7 had the same idea. And it is out of question.

The main reason is that contrary to bbpcode, devpi is without any authentication, and that white list entire IP ranges from Spanish, Italian and German HBP institutes for them to install the software is not acceptable from a sys-admin perspective.

Adding HTTP auth to devpi is also problematic, because it will means hardcoded users credential in netrc files letf and right to make pip works in the users scripts.

[adevress]

But to come back on that: I see bluepyopt on the official Pypi, I suppose it has been OSS ? Can I just use this one ?

https://pypi.python.org/pypi/bluepyopt/1.6.27

[wvangeit]

I guess you're confusing bluepyopt with bglibpy ?

[adevress]

Indeed << Any hope to see bglibpy OSS one day ?

[wvangeit]

we first need neurodamus :-) (and all the other dependencies)

[adevress]

ahah, that's a challenge by itself indeed ^^