Conda package broken due to Julia installation

[bouweandela]

The latest conda package is broken because of issue #1534.

The correct solution would be to change the julia installation script so it installs all packages at once and the order in which it installs things no longer matters. Next we should make a new release and new conda package.

[valeriupredoi]

why is it broken?

[bouweandela]

why is it broken?

Because of the issue mentioned above. This issue was 'solved' by changing the order of the dependency list, but that will only work until the next time things go wrong, manually sorting lists of dependencies in the hope that you find a compatible list of versions is not the way to go. The pull request referenced makes use of a Julia environment and leaves picking a set of compatible versions to the Julia package manager. After that is merged, i will make a new release and conda package.

[bouweandela]

The new release (v2.0.0b3) and associated conda package solved this, but unfortunately Julia diagnostics do not work in the new package. Will investigate.

[bouweandela]

Removing the directory ~/.julia and re-installing the conda package solved the problem.

[jvegreg]

The nightly conda install is still failing with the same error. Any clue?

[bouweandela]

For some reason conda tries to install an old version if you do not specify one. The latest package works fine.

[jvegreg]

For some reason conda tries to install an old version if you do not specify one. The latest package works fine.

Is trying to use the linux x64 distribution instead of the noarch, maybe?

https://anaconda.org/ESMValGroup/esmvaltool

See the download stats

[bouweandela]

For some reason it seems to find that environment easier to solve:

conda create -n test esmvaltool=2.0.0b3 -c conda-forge -c esmvalgroup -d

takes about 75 seconds to run on my computer, while

conda create -n test esmvaltool -c conda-forge -c esmvalgroup -d

takes just 15 seconds, but then it plans to install v2.0.0b0 of esmvaltool instead of the latest version.

[jvegreg]

Ok. I tried it using the --debug flag. It looks that first it tries to use the linux-64. If that is denied because of the version specification, it tries noarch

[jvegreg]

Is there a way to remove / deprecate the linux-64 packages in conda?

[jvegreg]

For some reason it seems to find that environment easier to solve

The arch version have stricter dependencies, so there are quite a few combinations that conda must not discard by itself

Edit: in particular, deciding which python to use takes a while in my machine

[bouweandela]

Yes, we could delete the older packages, but I'm not sure if that's a good idea. What if someone is using them?

[jvegreg]

Yes, we could delete the older packages, but I'm not sure if that's a good idea. What if someone is using them?

That's why I asked for deprecating it (i.e. leave it installable but impossible to found unless you specify that particular version).

If that is not possible, I think is worse that installing ESMValTool without any version will retrieve a random old one instead of the latest

[bouweandela]

I think is possible using e.g. deprecated label, people could then install the old version using something like

conda create -n test esmvaltool=2.0.0a0 -c conda-forge -c esmvalgroup/label/deprecated

This will require anyone using such an old version to change their installation scripts though.

[valeriupredoi]

@bouweandela there is an issue with b3 - look at the file sizes

[valeriupredoi]

there was an error in the production of b3: b2: info/ is 36M with all the necessary codes in info/recipe/esmvaltool b3: info/ is a diminutive 300k with no info/recipe/esmvaltool

Gotta rebuild it, bro :beer:

[bouweandela]

@valeriupredoi Actually there was a mistake in all the old builds which makes them huge. The info directory contains only metadata, see the conda documentation. Conda will copy files that are in the same directory as the conda recipe into the info/recipe directory, e.g. all the pictures from the documentation. The actual package is located in the site-packages directory.

[jvegreg]

This will require anyone using such an old version to change their installation scripts though.

This is a price I'm willing to pay to avoid installing an old version by default

[valeriupredoi]

Ill have to investigate a bit more the quality of the latest b3 package as conda packed it. There are two reasons why conda decides to install an older version without expllicitly telling it to do so: either there is an issue with the newer package or the environment you ask for solves better for the older version. Since circleci solves the conda env no problemo I suspect it's the first cause - Bouwe, when you turned off that option to not copy all the stuff to info/ have you done smth else too? :beer:

[jvegreg]

Ill have to investigate a bit more the quality of the latest b3 package as conda packed it. There are two reasons why conda decides to install an older version without expllicitly telling it to do so: either there is an issue with the newer package or the environment you ask for solves better for the older version. Since circleci solves the conda env no problemo I suspect it's the first cause - Bouwe, when you turned off that option to not copy all the stuff to info/ have you done smth else too? beer

I did a quick test with labels:

• Put everything linux-64 in a new label called deprecated

• Create a new environment with the --dry-run

• Conda now installs v2.0.0b3 as expected

So we can finally conclude that this is the problem: conda prefers the specific versions above the generic. I think it assumes that if there is an architecture dependant version along with a noarch it is because the noarch is not desirable in that architecture for whatever reason

I reverted the changes, so don't try it by yourself, you'll need to trust my word on this

[bouweandela]

Ok, let's do the deprecated labels thing then, I'll add them again.

[bouweandela]

I'll also see if I can unpin some packages, to make it easier to solve the environment again.

[jvegreg]

I'll also see if I can unpin some packages, to make it easier to solve the environment again.

For me, it is way faster when I add the specific version of Python I want to install. Maybe we should add it as a hint in the documentation

[valeriupredoi]

Ah yes, even channel searching conda puts noarch after linux64, how about we build the linux64 too or remove all linux64 older packages? I don't like the deprecated flag

[valeriupredoi]

No, adding versions, subversions and hashes is something regular Joe user doesn't really know, let's not complicate the installation process even more :beer:

[bouweandela]

Bouwe, when you turned off that option to not copy all the stuff to info/ have you done smth else too?

@valeriupredoi Working on this topic in https://github.com/ESMValGroup/ESMValTool/pull/1570

remove all linux64 older packages?

I'm not keen on removing packages, removing packages from conda puts people relying on those packages in trouble: I regularly spend a few hours fixing our installation procedure because someone decided they can remove stuff from conda or CRAN. I think the deprecated flag is a good first step. We can remove the deprecated stuff later.

For me, it is way faster when I add the specific version of Python I want to install. Maybe we should add it as a hint in the documentation

I'm not sure, people are already complaining that the installation procedure is lengthy and complicated.

[valeriupredoi]

Yepers, agreed! :beer:

[bouweandela]

Just successfully installed esmvaltool from conda and ran a few recipes. OK to close this issue? @jvegasbsc @valeriupredoi?

[valeriupredoi]

good job guys! Now by doing

conda create --name esmvaltool
conda install esmvaltool -c conda-forge -c esmvalgroup

it indeed wants to install:

  esmvalcore         esmvalgroup/noarch::esmvalcore-2.0.0b8-py_0
  esmvaltool         esmvalgroup/noarch::esmvaltool-2.0.0b3-py_0

Note that we can put a note saying that if you really want to install the pkg from linux-64 you can do it via:

conda install esmvaltool -c esmvalgroup/linux-64 -c conda-forge

which, at the moment, results in installing the noarch ones since all the others are deprecated. We must tell users not to use the esmvalgroup/label/deprecated option because that results in installing some really ancient package:

esmvaltool         esmvalgroup/label/deprecated/linux-64::esmvaltool-2.0a2-py36_0

[valeriupredoi]

OK to close with a few pointers I'd add to the documentation:

• we can build linux-64 specific packages but we should put them in a separate dir (like deprecated but call it something else) and tell the users how to do the installation via channel label BUT honestly I dont see the point of doing that
• we must tell the users not to use the deprecated label (see comment above)
• :beer: