Closed JohannesWiesner closed 1 year ago
If you compare the runner-images of ubuntu-latest
and windows-latest
, you can see the following:
Miniconda 23.3.1
Miniconda 23.3.1 (pre-installed on the image but not added to PATH)
So we can see that conda
is on path on linux but not on windows.
Since setup-micromamba doesn't do anything with conda, conda is still not in PATH in the windows runner.
You could fix this like that:
- run: echo $CONDA >> $GITHUB_PATH
shell: bash
Or use the absolute path for the conda executable
# probably C:\Miniconda\conda.exe (?)
- run: C:\Miniconda\conda.exe env export -p [...]
Alright I just checked what the conda installation on windows-latest
looks like
# ls /c/Miniconda/
_conda.exe api-ms-win-core-string-l1-1-0.dll api-ms-win-crt-utility-l1-1-0.dll python.exe
api-ms-win-core-console-l1-1-0.dll api-ms-win-core-synch-l1-1-0.dll concrt140.dll python.pdb
api-ms-win-core-datetime-l1-1-0.dll api-ms-win-core-synch-l1-2-0.dll condabin python3.dll
api-ms-win-core-debug-l1-1-0.dll api-ms-win-core-sysinfo-l1-1-0.dll conda-meta python310.dll
api-ms-win-core-errorhandling-l1-1-0.dll api-ms-win-core-timezone-l1-1-0.dll cwp.py python310.pdb
api-ms-win-core-file-l1-1-0.dll api-ms-win-core-util-l1-1-0.dll DLLs pythonw.exe
api-ms-win-core-file-l1-2-0.dll api-ms-win-crt-conio-l1-1-0.dll envs pythonw.pdb
api-ms-win-core-file-l2-1-0.dll api-ms-win-crt-convert-l1-1-0.dll etc Scripts
api-ms-win-core-handle-l1-1-0.dll api-ms-win-crt-environment-l1-1-0.dll include share
api-ms-win-core-heap-l1-1-0.dll api-ms-win-crt-filesystem-l1-1-0.dll Lib shell
api-ms-win-core-interlocked-l1-1-0.dll api-ms-win-crt-heap-l1-1-0.dll Library Tools
api-ms-win-core-libraryloader-l1-1-0.dll api-ms-win-crt-locale-l1-1-0.dll libs ucrtbase.dll
api-ms-win-core-localization-l1-2-0.dll api-ms-win-crt-math-l1-1-0.dll LICENSE_PYTHON.txt Uninstall-Miniconda3.exe
api-ms-win-core-memory-l1-1-0.dll api-ms-win-crt-multibyte-l1-1-0.dll Menu vccorlib140.dll
api-ms-win-core-namedpipe-l1-1-0.dll api-ms-win-crt-private-l1-1-0.dll msvcp140.dll vcomp140.dll
api-ms-win-core-processenvironment-l1-1-0.dll api-ms-win-crt-process-l1-1-0.dll msvcp140_1.dll vcruntime140.dll
api-ms-win-core-processthreads-l1-1-0.dll api-ms-win-crt-runtime-l1-1-0.dll msvcp140_2.dll vcruntime140_1.dll
api-ms-win-core-processthreads-l1-1-1.dll api-ms-win-crt-stdio-l1-1-0.dll msvcp140_codecvt_ids.dll zlib.dll
api-ms-win-core-profile-l1-1-0.dll api-ms-win-crt-string-l1-1-0.dll pkgs
api-ms-win-core-rtlsupport-l1-1-0.dll api-ms-win-crt-time-l1-1-0.dll pre_uninstall.bat
So you should use C:\Miniconda\_conda.exe env export ...
(when using cmd.exe
or powershell
) or alternatively /c/Miniconda/_conda.exe env export ...
(when using bash
)...
It looks like you can also just use Micromamba instead of Conda.
I think the conda env export part (especially with pip dependencies) is not directly transferable to micromamba.
So this works fine (generating a .yml file):
/c/Users/runneradmin/micromamba-bin/micromamba.exe env export -p "${MAMBA_ROOT_PREFIX}/envs/${{ runner.os }}" | grep -v "^prefix: " > ./environments/csp_${{ runner.os }}_solved.yml
but when executing this (generate an explicit spec-file):
/c/Users/runneradmin/micromamba-bin/micromamba.exe list --explicit -p "${MAMBA_ROOT_PREFIX}/envs/${{ runner.os }}" > ./environments/csp_${{ runner.os }}_solved.txt`
I got this:
The following argument was not expected: --explicit
which makes sense because micromamba list
seems to be implemented in micromamba but not micromamba list --explicit
. If you want to get an explicit list you still have to rely on conda list --expcicit
as described in the docs or run:
`/c/Users/runneradmin/micromamba-bin/micromamba.exe env export -p "${MAMBA_ROOT_PREFIX}/envs/${{ runner.os }}" --explicit > ./environments/csp_${{ runner.os }}_solved.txt`
Alright it seems that the docs are out of date. They should point to micromamba env export --explicit
for explicit files.
But then I would say the easiest solution would be
micromamba env export -n "${{ matrix.os }}" > ./environments/csp_${{ matrix.os }}_solved.yml
micromamba env export --explicit -n "${{ matrix.os }}" > ./environments/csp_${{ matrix.os }}_solved.txt
If you rely on pip dependencies being added to the export, then you need to call conda
(which can be done on windows runners with /c/Miniconda/_conda.exe env export ...
)
If you rely on pip dependencies being added to the export, then you need to call
conda
(which can be done on windows runners with/c/Miniconda/_conda.exe env export ...
)
Just to be clear here: /c/Miniconda/
will automatically be installed when using mamba-org/setup-micromamba@v1
?
No, only Micromamba will be installed, no Conda
/c/Miniconda/
is already shipped when using windows-latest
.
Okay, I completely had to switch back to conda commands for both the Linux and Windows jobs. This is how my code looks now (see full code here):
if [ "$RUNNER_OS" == "Windows" ]; then
/c/Miniconda/_conda.exe env export -p "${MAMBA_ROOT_PREFIX}/envs/${{ matrix.os }}" | grep -v "^prefix: " > ./environments/csp_${{ matrix.os }}_solved.yml
/c/Miniconda/_conda.exe list -p "${MAMBA_ROOT_PREFIX}/envs/${{ matrix.os }}" --explicit > ./environments/csp_${{ matrix.os }}_solved.txt
elif [ "$RUNNER_OS" == "Linux" ]; then
conda env export -p "${MAMBA_ROOT_PREFIX}/envs/${{ matrix.os }}" | grep -v "^prefix: " > ./environments/csp_${{ matrix.os }}_solved.yml
conda list -p "${MAMBA_ROOT_PREFIX}/envs/${{ matrix.os }}" --explicit > ./environments/csp_${{ matrix.os }}_solved.txt
fi
For the linux-runner, everything works fine but for the windows-runner, I get:
Run if [ "$RUNNER_OS" == "Windows" ]; then
if [ "$RUNNER_OS" == "Windows" ]; then
/c/Miniconda/_conda.exe env export -p "${MAMBA_ROOT_PREFIX}/envs/windows-latest" | grep -v "^prefix: " > ./environments/csp_windows-latest_solved.yml
/c/Miniconda/_conda.exe list -p "${MAMBA_ROOT_PREFIX}/envs/windows-latest" --explicit > ./environments/csp_windows-latest_solved.txt
elif [ "$RUNNER_OS" == "Linux" ]; then
conda env export -p "${MAMBA_ROOT_PREFIX}/envs/windows-latest" | grep -v "^prefix: " > ./environments/csp_windows-latest_solved.yml
conda list -p "${MAMBA_ROOT_PREFIX}/envs/windows-latest" --explicit > ./environments/csp_windows-latest_solved.txt
fi
shell: C:\Program Files\Git\usr\bin\bash.EXE -el {0}
env:
pythonLocation: C:\hostedtoolcache\windows\Python\3.7.9\x64
PKG_CONFIG_PATH: C:\hostedtoolcache\windows\Python\3.7.9\x64/lib/pkgconfig
Python_ROOT_DIR: C:\hostedtoolcache\windows\Python\3.7.9\x64
Python[2](https://github.com/JohannesWiesner/tcy/actions/runs/5322826699/jobs/9639798139#step:7:2)_ROOT_DIR: C:\hostedtoolcache\windows\Python\[3](https://github.com/JohannesWiesner/tcy/actions/runs/5322826699/jobs/9639798139#step:7:3).7.9\x6[4](https://github.com/JohannesWiesner/tcy/actions/runs/5322826699/jobs/9639798139#step:7:4)
Python3_ROOT_DIR: C:\hostedtoolcache\windows\Python\3.7.9\x[6](https://github.com/JohannesWiesner/tcy/actions/runs/5322826699/jobs/9639798139#step:7:6)4
MAMBA_ROOT_PREFIX: C:\Users\runneradmin\micromamba
MAMBA_EXE: C:\Users\runneradmin\micromamba-bin\micromamba.exe
CONDARC: C:\Users\runneradmin\micromamba-bin\.condarc
CommandNotFoundError: No command 'conda env'.
A few take-aways:
1.) Generating a spec-file (as documented here): The command conda env export --explicit
(in contrast to micromamba env export --explicit
) does not exist. For conda the equivalent to micromamba env export --explicit
is conda list --explicit
. I am aware that this file will not include pip packages which is addressed in this issue and which is out of scope of setup-micromamba@v1
. A possible workaround is to create a separate requirements.txt
file as mentioned in this comment.
2.) Generating a .yml
file (which can serve as an alternative to a spec-file.txt + requirements.txt and which will include pip-packages. The drawback here is, that it's not an exact recipe): In order to get a file that also includes pip-packages you have to use conda env export -p "${MAMBA_ROOT_PREFIX}/envs/name_of_my_environment"
, whereas micromamba env export -p "${MAMBA_ROOT_PREFIX}/envs/name_of_my_environment"
will not include pip-packages as mentioned here
Btw you can also use this syntax in GitHub actions
- run: |
conda env export -p "${MAMBA_ROOT_PREFIX}/envs/${{ matrix.os }}" | grep -v "^prefix: " > ./environments/csp_${{ matrix.os }}_solved.yml
conda list -p "${MAMBA_ROOT_PREFIX}/envs/${{ matrix.os }}" --explicit > ./environments/csp_${{ matrix.os }}_solved.txt
if: runner.os == 'Linux'
- run: |
/c/Miniconda/_conda.exe env export -p "${MAMBA_ROOT_PREFIX}/envs/${{ matrix.os }}" | grep -v "^prefix: " > ./environments/csp_${{ matrix.os }}_solved.yml
/c/Miniconda/_conda.exe list -p "${MAMBA_ROOT_PREFIX}/envs/${{ matrix.os }}" --explicit > ./environments/csp_${{ matrix.os }}_solved.txt
if: runner.os == 'Windows'
CommandNotFoundError: No command 'conda env'.
My guess would be that conda needs to be initialized for it to work... Did you try running conda init, see here.
Okay, I solved it.
My guess would be that conda needs to be initialized for it to work... Did you try running conda init, see here.
I followed the steps from this medium post and added the following step to my workflow to initialize conda for Git Bash:
- name: initialize conda for Git Bash (will be skipped for Linux)
run: |
if [ "$RUNNER_OS" == "Windows" ]; then
echo ". /c/Miniconda/etc/profile.d/conda.sh" >> ~/.bashrc
fi
Interestingly, this step has to be done before running mamba-org/setup-micromamba@v1
otherwise I ended up with the same error (CommandNotFoundError: No command 'conda env'
), I don't know why the order is important here.
With this approach, it's now also not necessary to use /c/Miniconda/_conda.exe
since conda
is now a valid command. Feel free to close this issue.
Interestingly, this step has to be done before running mamba-org/setup-micromamba@v1
The ~/.bashrc
only gets sourced when using the interactive shell mode (i.e. bash -i
). This is the default on Linux hence it is described in the Medium post.
On macOS and Windows on the other hand, the ~/.bash_profile
gets sourced when starting bash. You can (and do in your code) simulate this behavior in CI using bash -l
.
Since the bashrc doesn't get sourced in CI (since it is not an interactive shell and you didn't specify bash -i
), the command didn't do anything
The pip install ...
step in your CI fixed your setup
I would suggest that you replace the bashrc part with bash_profile
- name: initialize conda for Git Bash (will be skipped for Linux)
run: |
if [ "$RUNNER_OS" == "Windows" ]; then
echo ". /c/Miniconda/etc/profile.d/conda.sh" >> ~/.bash_profile
fi
Then it should work no matter if you put it in front of the setup-micromamba step or behind it.
I am using a matrix to create a windows and linux job that solve an input .yml file and should output a solved .yml file / spec-file.txt using:
The linux job works fine but for the windows job I get:
conda: command not found
It seems that the shell has not been initialized correctly here? Here's the link to the full code. And here is the output when setting up
setup-micromamba@v1
for the Windows-job: