cdms2 3.0.1 incompatible with latest conda-build

[xylar]

The conda recipe for cdma2=3.0.1 has syntax that the latest conda-build doesn't support:

Traceback (most recent call last):
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda_build/environ.py", line 749, in get_install_actions
    actions = install_actions(prefix, index, specs, force=True)
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda/common/io.py", line 85, in decorated
    return f(*args, **kwds)
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda/plan.py", line 473, in install_actions
    txn = solver.solve_for_transaction(prune=prune, ignore_pinned=not pinned)
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda/core/solve.py", line 107, in solve_for_transaction
    force_remove, force_reinstall)
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda/core/solve.py", line 145, in solve_for_diff
    force_remove)
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda/core/solve.py", line 242, in solve_final_state
    ssc = self._run_sat(ssc)
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda/common/io.py", line 85, in decorated
    return f(*args, **kwds)
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda/core/solve.py", line 465, in _run_sat
    conflicting_specs = ssc.r.get_conflicting_specs(tuple(final_environment_specs))
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda/resolve.py", line 852, in get_conflicting_specs
    reduced_index = self.get_reduced_index(specs)
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda/common/io.py", line 85, in decorated
    return f(*args, **kwds)
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda/resolve.py", line 356, in get_reduced_index
    specs, features = self.verify_specs(specs)
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda/resolve.py", line 244, in verify_specs
    raise ResolvePackageNotFound(bad_deps)
conda.exceptions.ResolvePackageNotFound: 
  - cdms2==3.0.1 -> esmf[version='>=7.1.*']
  - cdms2==3.0.1 -> esmpy[version='>=7.1.*']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xylar/miniconda3/bin/conda-metapackage", line 11, in <module>
    sys.exit(main())
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda_build/cli/main_metapackage.py", line 125, in main
    return execute(sys.argv[1:])
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda_build/cli/main_metapackage.py", line 121, in execute
    api.create_metapackage(channel_urls=channel_urls, **args.__dict__)
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda_build/api.py", line 374, in create_metapackage
    license_name=license_name, summary=summary, config=config)
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda_build/metapackage.py", line 29, in create_metapackage
    return build(m, config=config, need_source_download=False)
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda_build/api.py", line 207, in build
    notest=notest, need_source_download=need_source_download, variants=variants)
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda_build/build.py", line 2300, in build_tree
    notest=notest,
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda_build/build.py", line 1399, in build
    raise e
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda_build/build.py", line 1390, in build
    channel_urls=tuple(m.config.channel_urls))
  File "/home/xylar/miniconda3/lib/python3.6/site-packages/conda_build/environ.py", line 751, in get_install_actions
    raise DependencyNeedsBuildingError(exc, subdir=subdir)
conda_build.exceptions.DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {"esmpy[version='>=7.1.*']", "esmf[version='>=7.1.*']"}

The newest versions of cdms2 on conda-forge (3.1.2) don't have this problem but isn't compatible with e3sm_diags.

[xylar]

@zshaheen, I'm not going to be able to make a new version of e3sm-unified until this issue gets fixed. Is there any chance you could try a build of e3sm_diags with a newer version of cdms2 and perhaps other cdat packages?

[zshaheen]

@dnadeau4, @doutriaux1, didn't you guys have a similar issue like this? I think you guys said that for some reason, the newest version on conda-build gave issues, specifically regarding cdms2. Any input?

[xylar]

It's been fixed on conda-forge (not sure about the cdat channel). It's just that v.3.0.1 hasn't been rebuilt with the fix. Is the easiest solution to rebuild the old version with the new recipe? Is there a good reason not to update e3sm_diags to a more recent version?

[xylar]

Digging into this a bit further, I now see that the latest release of cdat (8.0) only works with v3.0.1, so unless we're good with using a nightly build of cdat, there's not a good option to upgrade. I'll request a rebuild of 3.0.1

[zshaheen]

@xylar, I can build e3sm_diags with the latest version of of cdms, but I don't know if that would fix it. @dnadeau4, is whatever causing this issue be fixed in the later versions of cdms?

I'm asking because building v3.0.1 with an older version of conda-buildworked fine for me. So this issue might not have been addressed. If not, like Xylar said, we'd need a rebuild.

[xylar]

I have a PR here for rebuilding: https://github.com/conda-forge/cdms2-feedstock/pull/27

I'm not saying that's the best option but it might work.

I'm asking because building v3.0.1 with an older version of conda-buildworked fine for me. So this issue might not have been addressed. If not, like Xylar said, we'd need a rebuild.

Yes, I can build a metapackage with just cdms2 ==3.1.2 just fine. That's because the unsupported syntax for specifying the versions of esmf and esmpy were removed in later versions. The problem is finding a version of all the other cdat packages that works with that version of cdms2. If you have such a set, @zshaheen, I think that's by far the better solution and I can abandon my branch on cdms2-feedstock.

[xylar]

Okay, so I think I now understand the problem better. The latest conda/conda-build doesn't think that esmf=7.1.0r satisfies esmf>=7.1.*, presumably because it is a release candidate. Presumably, this is a new behavior and release candidates were handled okay before. I'll see if there's anything more I can figure out about this but currently there is no such release version of ESMF (or ESMPy) available. So conda doesn't think cdms2 is supported.

[xylar]

Ah, there's already an issue about this: https://github.com/conda/conda/issues/8309

[xylar]

If I go back to conda=4.5.12, the issue goes away when I use conda create.... I may have to go back to an earlier conda-build as well, we'll see. But maybe this is the solution for now.

[alaniwi]

@xylar For your info, it seems that conda=4.5.13 (i.e. the latest 4.5.x) was the latest version where it worked, and conda=4.6.1 was the first version where it didn't.

(By the way, sorry, I am not going to follow this thread.)