Support for "conda-devenv" extension

[pabloGillariCes]

First of all, I'd like to thank you guys for this awesome project! 🚀

Background

The conda-devenv extension adds the ability to work with multiple projects in development mode, by processing environment.devenv.yml files, similar to regular environment.yml files, with features like Jinja2 templating, files inclusion, and environment variables.

https://github.com/ESSS/conda-devenv

Scenario

Environment devenv files are parsed incorrectly and the action throws an error.

# .github/workflows/simple-workflow.yml
  ...
  with:
    activate-environment: ${{ steps.configure.outputs.env-name }}
    environment-file: ${{ steps.configure.outputs.env-file }}
    python-version: ${{ steps.detect-version.outputs.version }}
  ...

# environment.devenv.yml
name: a-simple-service
dependencies:
  - gcc_linux-64 # [linux]
includes:
  - environment.yml

# environment.yml
name: a-simple-service
channels:
  - conda-forge
  - defaults
dependencies:
  - mypy
  - pip
  - python=3.8.5
  - pytest

The error

The response of the action setup is next:

> Updating 'a-simple-service' env from conda env update...
  /usr/share/miniconda/condabin/conda env update --name a-simple-service --file a-simple-service/environment.devenv.yml
  /home/runner/work/_actions/conda-incubator/setup-miniconda/v2/dist/setup/index.js:8193
                              throw new Error(`"${command}" failed with "${forced_error}"`);
                              ^

  Error: "/usr/share/miniconda/condabin/conda,env,update,--name,a-simple-service,--file,a-simple-service/environment.devenv.yml" failed with "EnvironmentSectionNotValid"
      at Object.stdout (/home/runner/work/_actions/conda-incubator/setup-miniconda/v2/dist/setup/index.js:8193:35)
      at Socket.<anonymous> (/home/runner/work/_actions/conda-incubator/setup-miniconda/v2/dist/setup/index.js:823:52)
      at Socket.emit (events.js:210:5)
      at addChunk (_stream_readable.js:309:12)
      at readableAddChunk (_stream_readable.js:290:11)
      at Socket.Readable.push (_stream_readable.js:224:10)
      at Pipe.onStreamRead (internal/stream_base_commons.js:182:23)

Thank you very much!

[jaimergp]

With the current code, I believe you could get it working with two steps:

1. Setup Miniconda with an env that contains only conda-devenv and the desired Python, for now.
2. Use conda-devenv to update the configured environment using the provided .devenv.yml

---

To provide native support in the tool we would need to add a new keyword (use-devenv or similar) and install conda-devenv on base, then process the environment file with devenv. We could also auto-detect the need for devenv by looking at the yml file extension, if that's a standard practice.

Thoughts @goanpeca @bollwyvl ?

[bollwyvl]

I think the previous suggestion is the path forward for us here (use setup-miniconda, then insert-your-tool-here). This clearly separates the problem of "get non-ancient conda" from "do stuff (with conda)"... And would allow you to swap out for, say provision-micromamva

If you can lobby conda+mamba to support your file format (good luck) I'm all for it!

nox, anaconda-project, etc. have all been used successfully with this action, and with infinite maintainer time, I could imagine supporting all of them... But as that isn't the case...

[pabloGillariCes]

With the current code, I believe you could get it working with two steps:

1. Setup Miniconda with an env that contains only conda-devenv and the desired Python, for now.
2. Use conda-devenv to update the configured environment using the provided .devenv.yml

Hey, thanks for the prompt response.

Most of the problem is the activation of the environment.

I tried to create a minimal environment.yml file to use when calling the action and then adding an extra install step with the conda-devenv configuration and the actual environment.devenv.yml file I want to use. But even the environment is listed using conda info --envs from the previous conda-incubator/setup-miniconda@v2 action, it seems to ignore any conda activate command or any other environment-related command.

I assume it's an issue with the login shells, that it won't happen within the action.

[bollwyvl]

A reproducer would help, but there's an example of using base directly (which is probably appropriate in this case) in the docs.

- uses: conda-incubator/setup-miniconda@v2
  with:
    auto-activate-base: true
    activate-environment: ""

I don't remember if that will work with environment-file, but probably does.

[pabloGillariCes]

Thank you, people. You pointed me in the right direction about activating the environment. Then adding the environment's bin files into GITHUB_PATH was the trick/hack needed.

Updates

Conda installation ran with the environment name only:

# workflow.yml
  ...
  - id: setup-python
    if: ${{ steps.detect-language-and-version.outputs.language == 'python' }}
    name: Setup Python environment
    uses: conda-incubator/setup-miniconda@v2
    with:
      activate-environment: ${{ steps.configure.outputs.environment-name }}
  ...

Prior to this, I created a dummy environment.yml file only with name and channels objects for the action to use, previously replacing the environment name with the value fetched from environment.devenv.yml like a Jinja2 template:

# dummy environment.yml.j2 > environment.yml
name: {{ service-name }}
channels:
  - conda-forge
  - defaults

The conda-incubator/setup-miniconda is creating and activating my (empty) environment correctly.

What I do next is to install conda-devenv, by creating an install target in a Makefile:

.ONESHELL:
SHELL=/bin/bash

.PHONY: install
install:
    conda update -n base -c defaults conda
    conda install conda-devenv -c conda-forge -y
    conda devenv
    echo "${CONDA}/envs/${ENV_NAME}/envs/${ENV_NAME}/bin" >> ${GITHUB_PATH}

At this step, conda devenv is using the environment.devenv.yml I want, without even adding an extra configuration.

As I try to keep make targets simple and portable for Python, Node, or any other application languages, all of the steps below setup-python are generic. They only need a login shell that may not be necessary for all:

  ...
  - id: install
    name: Install
    shell: bash -l {0}
    run: |
      make install
  ...

I don't need to activate the environment since it's already done and it's being updated by this command from devenv:

> Executing: conda env update --file /home/runner/work/......./a-simple-service/environment.yml --prune

Thanks again for this action. Best regards.

[jaimergp]

Nice! Thanks for the detailed overview!

[sarnold]

I've been using this action for some time to setup GH CI for both projects with full meta.yaml as well as projects with a conda-devenv file (if at all possible, across all the GH runner platforms, etc). Both types of conda env files used to work, but recently conda-devenv started failing on windows just like issue #9

I tried various suggested (action) config options but the previously documented setup with activate-environment set to an empty string fails with an error: python-version: 3.XX requires 'activate-environment: true'

I could really use some suggestions here; if it's possible to just activate base (using the action) and then run shell commands to generate/activate conda-devenv I could work with that. The latest failed action and yaml config can be seen here: https://github.com/berkeley-abc/abc/actions/runs/7082648502

Thanks in advance for any suggestions.

[sarnold]

Aaand I reverted some previous change bits and managed to get the environment activated manually and this works now across all the runners:

    - uses: conda-incubator/setup-miniconda@v3
      with:
        auto-update-conda: true
        auto-activate-base: true
        activate-environment: ''
        channels: conda-forge
        channel-priority: flexible

    - name: Configure condadev environment
      shell: bash -l {0}
      env:
        PY_VER: ${{ matrix.python-version }}
      run: |
        conda config --set always_yes yes --set changeps1 no
        conda install conda-devenv=3.2.0

    - name: Build and test
      shell: bash -l {0}
      env:
        PY_VER: ${{ matrix.python-version }}
      run: |
        conda devenv
        conda activate abc
        ctest --build-generator "${{ matrix.generator }}" \
          --build-and-test . build \
          --build-options ${CMAKE_ARGS} ${{ matrix.extra_args }} \
          -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=${{ matrix.build_type }} \
          --test-command ctest --rerun-failed --output-on-failure -V

Thanks!