Update requirements.txt / environment.yaml #44

Closed RandomDefaultUser closed 2 years ago

In GitLab by @RandomDefaultUser on Feb 11, 2021, 11:26

Related to #38. The environment.yaml is currently outdated (and I believe the requirements.txt too). The former does not include oapackage and the latter does not install swig (I think, I am not sure and I can't test it at the moment). Furthermore it seems that we now have a restriction on which python verison to use, at least on the cluster. The total energy package there is built with a specific python version (it was originally python3.6.5, Mani might be changing it). Maybe we should talk about how we proceed here in our next meeting.

In GitLab by @RandomDefaultUser on Feb 11, 2021, 13:52

I support the idea of providing a fully working virtual environment. Once we release part of the code, most people interested will be users rather than developers. Having this would benefit making the code popular.

By Cangi, Dr. Attila (FWU) - 139621 on 2021-02-11T13:52:44 (imported from GitLab)

In GitLab by @RandomDefaultUser on Feb 11, 2021, 13:02

It is possible to use pip inside of conda environments. If the oapackage is availible via pip, you may consider to install pip inside the conda environment and then pip install oapackage. You can also export this to a Conda yaml file which lists then a pip section.

By Kotik, Daniel (FWU) - 140179 on 2021-02-11T13:02:32 (imported from GitLab)

In GitLab by @RandomDefaultUser on Feb 11, 2021, 11:35

To clarify: oapackage is not in the environment.yaml because it is not available from any channel afaik. So we would also have to update the installation notes in this regard.

By Fiedler, Lenz (FWU) - 146409 on 2021-02-11T13:02:32 (imported from GitLab)

In GitLab by @RandomDefaultUser on Feb 11, 2021, 13:08

Yes, that is what I have done so far. My suggestion is simply to put this into the README.md for installation too. I have not done that yet because I thought it might be wise to adress all these issues at once.

By Fiedler, Lenz (FWU) - 146409 on 2021-02-11T13:08:18 (imported from GitLab)

In GitLab by @RandomDefaultUser on Feb 17, 2021, 10:08

This rabbit hole actually goes a little bit deeper as well. Python3.6.5 and Python3.8.x are not fully compatible in terms of the pickle and pytorch packages (I/O). The current state of the hemera packages forces us/me to sort of "interoperate" (LAMMPS only works with python 3.8.0, QE only with 3.6.5, I foolishly installed my horovod with python 3.8.5). So for now I am adding some hotfixes to ensure that we can use the the save and load functionalities throughout. But this comes at a cost in terms of file size. We should really restrict the versions more so we don't run into these problems.

By Fiedler, Lenz (FWU) - 146409 on 2021-02-17T10:08:22 (imported from GitLab)

In GitLab by @RandomDefaultUser on Feb 23, 2021, 16:39

I guess we have a little misunderstanding here. Although there is no conda channel with oapackage available, one has the freedom to use pip inside conda environments. If you create a conda environment, pip is installed per default and you can make use of it. In this way conda allows you to specify oapackage inside a conda yml file, see fesl_cpu_base_environment.yml:

name: fesl-cpu
channels:
  - conda-forge
  - defaults
dependencies:
  - python>=3.6, <3.9
  - pip
  - numpy
  - scipy
  - optuna
  - ase
  - pytorch-cpu
  - mpmath
  - pip:
    - oapackage

This was done in !33.

By Kotik, Daniel (FWU) - 140179 on 2021-02-23T16:40:34 (imported from GitLab)

In GitLab by @RandomDefaultUser on Feb 11, 2021, 12:59

This is definitely important and needs to be done also due to the following reasons:

In order to speed up CI pipline runtime: every single fesl test job inside the test stage requires all the dependencies to be installed (naturally). At the moment this installation procedure happens for every single fesl test job inside the test stage. This installation process alone covers ~4min and can be avoided by caching the build stage and reusing the installed dependencies.
As I've pointed out earlier, it is good practice to have a reproducible environment. This means having all the dependencies specified and fixed. From my understanding, you want to distribute FESL as a single package (that is why there is a setup.py) and that is why you don't specify the dependencies in requirments.txt in full detail (meaning exact versions) and leaving some out, e.g. pytorch - to leave it to the user which version (CPU/GPU) to use (for example). However, we also should provide a virtual environment with a fully specified and working list of dependencies (needed for point one anyway). This may be a Conda environment and/or Pipenv (better a Poetry environment).

By Kotik, Daniel (FWU) - 140179 on 2021-02-23T16:50:41 (imported from GitLab)

In GitLab by @RandomDefaultUser on Feb 23, 2021, 17:04

This was done in !33. We have now to versions of yml files: one with direct dependencies and one with explicit (versioned) dependencies including sub-dependencies. See the updated documentation for more details and https://pythonspeed.com/articles/conda-dependency-management/ for background information on this topic. Furthermore I've added two skeleton yml files for use on GPU architectures (here also CUDA versions become relevant). They need to be updated when the time comes.

By Kotik, Daniel (FWU) - 140179 on 2021-02-23T17:04:40 (imported from GitLab)

In GitLab by @RandomDefaultUser on Feb 23, 2021, 23:10

The on only reason these packages are there because this is the installation string the pytorch website gives you when using their only wizard for installation... There is no need for them. I just forgot that I never deleted them. You are right, we should just exclude them if we never use them anyway.

By Fiedler, Lenz (FWU) - 146409 on 2021-02-23T23:10:36 (imported from GitLab)

In GitLab by @RandomDefaultUser on Feb 23, 2021, 16:49

Regarding pip: in the install sec of the documentation it is written

$ pip install torch==1.7.1+cpu torchvision==0.8.2+cpu \
    torchaudio==0.7.2 -f \
    https://download.pytorch.org/whl/torch_stable.html

I wonder if torchvision and torchaudio are actually needed? This seems not to be the case (at least by now). So I have not put these packages into the current yml files and everything seems to work. If these packages are obsolete, the pip installation instructions should be updated.

By Kotik, Daniel (FWU) - 140179 on 2021-02-23T23:10:36 (imported from GitLab)

In GitLab by @RandomDefaultUser on Feb 23, 2021, 23:11

Oh! I didn't know you could do that. That's great!

By Fiedler, Lenz (FWU) - 146409 on 2021-02-23T23:11:17 (imported from GitLab)

I think we should (re-)address this now. As pointed out by Eugene on Slack there are some issues here:

We need to include SWIG
We need to investigate what causes this libz error to occasionally happen and how to fix it
We should work on the GPU yml files

OAPackage is optional now, so points 1 and 2 are now on the user. This only leaves the GPU yml file.

With respect to the GPU environment, we should probably discuss what is to be done here. If we continue not to include torch in the environment, what are the big differences between thos then?

GPU environment.yml progress will be tracked in #278 now.

mala-project / mala

Update requirements.txt / environment.yaml #44