search

giotto-ai / giotto-tda

A high-performance topological machine learning toolbox in Python
https://giotto-ai.github.io/gtda-docs
Other
847 stars 173 forks source link

Naming of the nightly builds #283

Open rth opened 4 years ago

rth commented 4 years ago

Follow up on https://github.com/giotto-ai/giotto-tda/issues/109. Currently nightly builds are named giotto-tda-nightly and are uploaded on PyPi. This issue aims to discuss whether the approach is optimal, with respect to,

There are two approaches.

  1. what is done now: i.e. different name, and uploading to pypi. This is for instance done by tensorflow
  2. Use the same name, the only difference is the version. Upload to some external storage. To my knowledge this is done by most scientific python packahges: numpy, scipy, scikit-learn pandas etc. This has the advantage that installing from sources and nightly wheels produces the same name, and IMO might allow to simplify the CI setup somewhat (because there is no need to test 2 configurations). For instance, when installing pytorch nightly ones run, 
    pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html

    or scikit-learn https://scikit-learn.org/stable/developers/advanced_installation.html#installing-nightly-builds.

See related discussion in https://discuss.python.org/t/publishing-nightly-builds-on-test-pypi-org-with-a-time-based-retention-policy/3152

Both approaches work, just wanted to raise this discussion. Likely not too critical in the near future. There are advantages and disadvantages in both cases.

cc @lewtun

ulupo commented 4 years ago

@rth thanks for formally initiating this discussion here so it reaches beyond the confines of our private chats. I guess I would see some merit to having two different names if they could somehow be used simultaneously. But I suppose (correct me if I'm wrong) that for this to happen the module names should also be changed (e.g. from gtda to gtda_nightly), and I don't think this is what we do now.

ulupo commented 4 years ago

Just for context, there are inependent reasons why we need nightly wheels quite quickly, and moving to a setup like sklearn's is a bit of a further intimidating dev-ops undertaking, at least for me at present.

rth commented 4 years ago

I guess I would see some merit to having two different names if they could somehow be used simultaneously.

Right, but that exactly the situation one wants to avoid, I think? Because otherwise when users have 2 versions of a package installed and are getting errors, it's quite hard to debug what's going on. Also nightly wheels should be roughly equivalent to building master from sources, in which case it can be confusing that the package names are different.

Just for context, there are inependent reasons why we need nightly wheels quite quickly, and moving to a setup like sklearn's is a bit of a further intimidating dev-ops undertaking, at least for me at present.

Thanks for the additional context! Definitely, I'm not saying it's a large concern now, I just opened this issue for future reference.

ulupo commented 4 years ago

Right, but that exactly the situation one wants to avoid, I think? Because otherwise when users have 2 versions of a package installed and are getting errors, it's quite hard to debug what's going on.

I realise I was very unclear: I meant to say that e.g. both impor gtda and import gtda_nightly would be available if both the nightly and stable version are installed. What do you think about this? I do take the point about similarity between the nightly and dev versions though...

holmbuar commented 4 years ago

I personally would support having both versions on pip. Particularly if it is possible to throw an exception if and when the user tries to install more than one version in the same environment. For example having gtda installed in gtda-env and then pip install gtda_nightly without creating a new env. Does that make sense?

rth commented 4 years ago

For example having gtda installed in gtda-env and then pip install gtda_nightly without creating a new env

It's possible indeed, but likely not very simple since gtda could be installed by either one of conda or pip. In any case documenting that both shouldn't be installed at the same time would definitely be useful.