Issue with scoop/windows installation of 22.11.1-2

tuanpham96 commented 1 year ago

Solution to issue cannot be found in the documentation.

[x] I checked the documentation.

Issue

We have an installation job in our CI using scoop on Windows Server images (mcr.microsoft.com/windows/servercore:1809). Previously via scoop, the job was successful with mambaforge@22.9.0-2. Calling conda in Powershell was ok.

However, with 22.11.1-2, it is now not successfully installed. It was missing even python at first. But even after installing python then reinstalling mambaforge, conda is not callable.

Here are the links to the logs of:

:x: 22.11.1-2: installed with

scoop install git 
scoop install python # had to add this
scoop bucket add extras
scoop install mambaforge # no version specified # LINE 96 of log
conda init powershell # error here

:heavy_check_mark: and when forcing to fall back on previous version 22.9.0-2, installed with

scoop install git 
scoop install python # optional, dont really need this
scoop bucket add extras
scoop install mambaforge@22.9.0-2 # fallback # LINE 96 of log
conda init powershell # ok here moving forward

Installed packages

NA

Environment info

NA

hmaarrfk commented 1 year ago

It is somewhat hard for us to troubleshoot scoop, a shim we don't support.

Can you help us find the commands it runs so somebody with a windows machine can troubleshoot.

tuanpham96 commented 1 year ago

help us find the commands it runs so

Did you mean scoop as it? If so, it is installed via powershell with admin rights in the CI or local docker image

# in powershell
iex "& {$(irm get.scoop.sh)} -RunAsAdmin"  # maybe can remove the run as admin if run on local machine? 
scoop install git 
scoop install python 
scoop bucket add extras

# if run install without version, the conda command cannot be found 
scoop install mambaforge 
conda init powershell

# if run install WITH version, the conda command is usable
scoop install mambaforge@22.9.0-2 
conda init powershell

hmaarrfk commented 1 year ago

Sorry, i didn't phrase my question well enough.

What is the exact bat or powershell command used by scoop, to install mambaforge?

hmaarrfk commented 1 year ago

cc @jaimergp

jaimergp commented 1 year ago

Wasn't the automated initialization removed? That might be making 'conda' not reachable on PATH.

hmaarrfk commented 1 year ago

Ah. Was it different between windows and Unix?

I tried to maintain the same defaults for Unix, but didn't test on windows.

jaimergp commented 1 year ago

My first intuition is to look at the effect of these lines, changed in https://github.com/conda-forge/miniforge/pull/415:

# During the interactive installation, these variables are checked.
# During batch installation, conda is never initialized
initialize_conda: True
initialize_by_default: False  # used to be True

I think scoop might be relying on the (previous) automated inclusion of conda somewhere in PATH as a byproduct of initialization. I don't know where scoop places Miniforge, but I am assuming that using something like /abs/path/to/bin/conda init ... would indeed work. After that, calls to a bare conda should work. Alternatively, if scoop allows passing arguments to the installer, users could re-enable the default initialization via CLI.

tuanpham96 commented 1 year ago

@hmaarrfk It should be just scoop bucket add extras and scoop install mambaforge together. Not sure about what goes on in the background.

So I checked the the directory in which scoop should be installing mambaforge into. For earlier versions, there would be many files, as expected. But for this version, there are 2 files (install.json and manifest.json) which I assume are from scoop?

The directory would be ~/scoop/apps/mambaforge/<version>/

EDIT: So versions 22.9.0-2 22.9.0-3 work. But versions from 22.11.1-0 to 22.11.1-2 do not

jaimergp commented 1 year ago

This line suggests that extra options cannot be passed. However, /AddToPath was always 0, so it was never initialized to begin with (or this option was ignored?).

@tuanpham96 can you check the value of where conda for functional and problematic versions, after scoop install mambaforge? Thanks!

tuanpham96 commented 1 year ago

For both working and not-working versions, where conda would not return anything.

But get-command conda would return the source as C:\Users\ContainerAdministrator\scoop\apps\mambaforge\current\scripts\conda.exe when 22.9.0-X is installed; but it would return an error for 22.11.1-X

jaimergp commented 1 year ago

So 22.9 was indeed modifying PATH. Can you check whether it was a system-wide change, user-wide change or just local to that Powershell session?

tuanpham96 commented 1 year ago

whether it was a system-wide change, user-wide change or just local to that Powershell session?

I'm not quite sure how to test that unfortunately, I don't use Windows as my main OS.

But I suspect it is only local to that session only if conda init powershell is not called first.

When I have 22.9 installed:

If I do not do conda init powershell -> exit powershell -> enter again: conda and even scoop commands cannot be found; the environment $PATH inside the new session does not reflect any changes made before
If I do conda init powershell -> exit powershell -> enter again: the PS1 has (base) in it, conda is callable, and the environment $PATH inside the new session does have paths of condabin. However, scoop cannot be called anymore.

I'm not sure how or why scoop cannot be called again honestly.

jaimergp commented 1 year ago

I usually search for "Environment variables" in Start Menu. The main result should bring up this dialog:

And the selected button should take you to this other one:

(screenshots are Win7 but same content)

Top panel is user-only env vars. Bottom panel is system-wide.

If scoop (or the installer) is modifying PATH temporarily on that session only, you can compare the values before and after in the shell (echo $env:PATH maybe?). If it's being set in the user or system level (I doubt it now), it should also appear in the UI dialog I mentioned above. It won't auto refresh so make sure to open it up from scratch before and after the scoop install.

To be honest at this point we might need to look into this locally so the amount of information I can give now is limited. It does look like the previous conda command (without initialization) availability was not intended and the current (22.11) behavior is the expected one, but I would agree that it was convenient, so I'll be happy to look into this a bit more to see if there's an easy solution. Otherwise, the abs path to conda (e.g. ~\scoop\apps\mambaforge\current\scripts\conda.exe init) should work.

hmaarrfk commented 1 year ago

xref: https://github.com/ScoopInstaller/Extras/issues/10420

c-ortbauer commented 1 year ago

I'm pretty sure the root of the failed scoop install is the check for an empty install directory. Scoop tries to install into the same folder the installer itself is located in. This worked with the previous version but not with the new one.

I already created a PR that fixes the issue by first moving the installer to different directory. ScoopInstaller/Extras#10457

tuanpham96 commented 1 year ago

@jaimergp Thanks for the info! I was doing it in a Windows Server docker image so the UI wouldn't be an option. I arrived at my statement above about whether paths are modified by checking $env:PATH in powershell actually.

@c-ortbauer Thank you for creating the PR. Unfortunately it still didn't work for me. I tried to install with your modified mambaforge.json in a fresh Windows Server image but I still ran into the same issues as above.

jaimergp commented 1 year ago

I'm pretty sure the root of the failed scoop install is the check for an empty install directory.

Hm, in /Silent mode, the installer should just ignore the warning and continue. We added this line precisely to avoid the installer hanging indefinitely for an answer, but maybe it's having a different outcome?

tuanpham96 commented 1 year ago

Based on this diff between 22.9 to 22.11, there are 3 main code changes:

pinning setuptools 65.*
upgrade to constructor>=3.4.2"
and removal of nsis for windows

if [[ "${TARGET_PLATFORM}" == win-* ]]; then
    conda install --yes "nsis=3.01" --channel conda-forge --override-channels
fi

The last one seems very specific to windows, which @c-ortbauer also had issues with. Could that be the reason?

jaimergp commented 1 year ago

The big change is moving to constructor 3.4.x from 3.3.x. The diff is quite large:

https://github.com/conda/constructor/compare/3.3.1..3.4.2

We have been ironing this out for a long time but it's a project with many rough edges. I am not surprised if we discover that previous unintended (or unaccounted) behaviour was in use and it's now broken (but technically "fixed" in intent).

I need to spin a VM to check this, so I'll see what I can do! Thanks for the continued feedback, really appreciated!

jaimergp commented 1 year ago

So the scoop JSON has this block:

...
    "env_add_path": [
        "scripts",
        "Library\\bin"
    ],
...

, which is what adds things to PATH. That mystery is solved.

I think @c-ortbauer's solution in https://github.com/ScoopInstaller/Extras/pull/10457 should fix things. If I remove the /S switch from the JSON I can reproduce the "directory not empty" error.

YWtheGod commented 1 year ago

in /S Silent mode, the installer just do nothing and quit, and this cause the issue Update: I check this agian, and found the installer do install the application, but it just quit immediately and let the installing job run in background. so when the scoop script got the control again, no files were actually installed, so the script failed to find the file pythonw.exe

jaimergp commented 1 year ago

Hm, the process is being -Waited here and our CI also waits successfully (from here):

miniforge
---------
coverage run --branch --append -m constructor -v D:\a\constructor\constructor\examples\miniforge --output-dir C:\Users\runneradmin\AppData\Local\Temp\tmpcyw5ltmq\miniforge-93ds2cvt --conda-exe C:\Miniconda\envs\constructor/standalone_conda/conda.exe
--- Done in 0:00:48.839204
--- Testing Miniforge3-4.10.1-0-Windows-x86_64.exe
cmd.exe /c start /wait C:\Users\runneradmin\AppData\Local\Temp\tmpcyw5ltmq\miniforge-93ds2cvt\Miniforge3-4.10.1-0-Windows-x86_64.exe /S /D=C:\Users\runneradmin\AppData\Local\Temp\tmpcyw5ltmq\miniforge-93ds2cvt\tmpomxh63g4
--- Done in 0:00:28.099688
cmd.exe /c start /wait C:\Users\runneradmin\AppData\Local\Temp\tmpcyw5ltmq\miniforge-93ds2cvt\tmpomxh63g4\Uninstall-Miniforge3.exe /S _?=C:\Users\runneradmin\AppData\Local\Temp\tmpcyw5ltmq\miniforge-93ds2cvt\tmpomxh63g4
--- Done in 0:00:04.734893

vdsbenoit commented 1 year ago

I have a similar issue without using scoop. Since v22.11.1-0, it seems the AddToPath option is discarded by the installer.

If I build this Dockefile and then run a container, conda & mamba are in the path:

FROM mcr.microsoft.com/windows/servercore:ltsc2019

RUN curl -curl.exe -fSLo mambaforge.exe https://github.com/conda-forge/miniforge/releases/download/22.9.0-3/Mambaforge-Windows-x86_64.exe; `
    Start-Process .\mambaforge.exe -Wait -ArgumentList "/S /InstallationType=AllUsers /RegisterPython=1 /AddToPath=1\"

But starting from v22.11.1-0, the path is no longer updated:

FROM mcr.microsoft.com/windows/servercore:ltsc2019

RUN curl -curl.exe -fSLo mambaforge.exe https://github.com/conda-forge/miniforge/releases/download/22.11.1-0/Mambaforge-Windows-x86_64.exe; `
    Start-Process .\mambaforge.exe -Wait -ArgumentList "/S /InstallationType=AllUsers /RegisterPython=1 /AddToPath=1\"

Except the path, everything seems to be properly installed.

jaimergp commented 1 year ago

@vdsbenoit - that's intended and unfortunately there's no way to signal that from the silent mode. See https://github.com/conda/constructor/pull/584 for details.

vdsbenoit commented 1 year ago

Thank you so much for explanation @jaimergp ! I fixed my issue by passing the /InstallationType=JustMe to the installer.

ReimarBauer commented 1 year ago

@vdsbenoit May be you can help me to get mine working?

install-windows:
    runs-on: windows-latest
    defaults:
      run:
        shell: cmd
    steps:
      - uses: actions/checkout@v3

      - name: Install mambaforge (1/2)
        run: |
          cd %GITHUB_WORKSPACE%
          curl -L0 https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Windows-x86_64.exe --output mambaforge-installer.exe
          call mambaforge-installer.exe -Wait -ArgumentList "/S /InstallationType=JustMe /RegisterPython=1 /AddToPath=1"

I do see it downloading but somehow it gets not installed 'mambaforge-installer.exe' is not recognized as an internal or external command, operable program or batch file. Error: Process completed with exit code 1.

Could you may be provide a short example where I can learn from what I did wrong?

How can I check when it looks like it has something installed that I can use it.

vdsbenoit commented 1 year ago

It looks like another issue. Are you sure mambaforge-installer.exe was properly downloaded ?

Moreover, I see that you use -Wait and -ArgumentList, which are arguments of the Start-Process command (from PowerShell), while you use the call command (from cmd). Is there a reason to use cmd and call here ?

Could you try this instead ?

install-windows:
    runs-on: windows-latest
    defaults:
      run:
        shell: powershell
    steps:
      - uses: actions/checkout@v3

      - name: Install mambaforge (1/2)
        run: |
          cd %GITHUB_WORKSPACE%
          curl.exe -fsSLo mambaforge.exe https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Windows-x86_64.exe
          Start-Process .\mambaforge.exe -Wait -ArgumentList "/S /InstallationType=JustMe /RegisterPython=1 /AddToPath=1"

tuanpham96 commented 1 year ago

@vdsbenoit I tried your suggestions above and it didn't work.

Running docker build with this

FROM mcr.microsoft.com/windows/servercore:ltsc2019

RUN curl.exe -fSLo mambaforge.exe https://github.com/conda-forge/miniforge/releases/download/22.11.1-0/Mambaforge-Windows-x86_64.exe 
RUN .\mambaforge.exe -Wait -ArgumentList "/S /InstallationType=JustMe /RegisterPython=1 /AddToPath=1"

then running it, neither conda or mamba was found. The software was installed in C:\Users\All Users\mambaforge, but the path wasn't changed.

Hence, I tried add the path manually like the following, only then conda and mamba would be found.

FROM mcr.microsoft.com/windows/servercore:ltsc2019

RUN curl.exe -fSLo mambaforge.exe https://github.com/conda-forge/miniforge/releases/download/22.11.1-0/Mambaforge-Windows-x86_64.exe 
RUN .\mambaforge.exe -Wait -ArgumentList "/S /InstallationType=JustMe /RegisterPython=1 /AddToPath=1"
RUN setx path "%path%;C:\Users\All Users\mambaforge;C:\Users\All Users\mambaforge\Scripts;C:\Users\All Users\mambaforge\condabin"
RUN conda init powershell  # this is optional

ReimarBauer commented 1 year ago

Found my mistake, by looking up silent mode I got the essential part https://docs.anaconda.com/anaconda/install/silent-mode/

Take care to have /S and /D as the last option at the end

install-mamba-on-windows-pwsh:
    runs-on: windows-latest
    defaults:
      run:
        shell: powershell
    steps:
      - uses: actions/checkout@v3

      - name: Install mambaforge by curl(1/2)
        run: |
          curl.exe -fsSLo mambaforge.exe https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Windows-x86_64.exe
          Start-Process .\mambaforge.exe -Wait -ArgumentList "/InstallationType=JustMe /RegisterPython=1 /AddToPath=1 /S /D=$($env:USERPROFILE)\mambaforge"

      - name: check mamba info(2/2)
        run: |
          $env:Path += ";$($env:USERPROFILE)\mambaforge\condabin"
          mamba info

  install-mamba-on-windows-cmd:
    runs-on: windows-latest
    defaults:
      run:
        shell: cmd
    steps:
      - uses: actions/checkout@v3

      - name: Install mambaforge by curl(1/2)
        run: |
          curl.exe -fsSLo mambaforge.exe https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Windows-x86_64.exe
          start /wait "" mambaforge.exe /InstallationType=JustMe /RegisterPython=1 /AddToPath=1 /S /D=%UserProfile%\mambaforge
      - name: check mamba info(2/2)
        run: |
          set PATH=%PATH%;%USERPROFILE%\mambaforge\condabin
          mamba info

vdsbenoit commented 1 year ago

@tuanpham96 the snippets from your comment do not match with my suggestion. For some reasons, you stripped Start-Process from the second RUN instructions. And as I said in my previous comment, the -Wait and -ArgumentList arguments are arguments of the Start-Process command. Passing them straight to the mamba installer does not have any effect.

tuanpham96 commented 1 year ago

@vdsbenoit Sorry you're right. Will try again some time next week when I have time.

UPDATE: Unfortunately I still had to set the paths explicitly to include the installed package directory.

@jaimergp Is explicitly setting paths an intended action for Windows after installing?

jaimergp commented 1 year ago

@jaimergp Is explicitly setting paths an intended action for Windows after installing?

What do you mean? /AddToPath=1 is an option that you should not technically use (it pollutes the system too much). The usual way of enabling conda after the installation is having conda init do its work, which I think it's enabled by default?

Btw, is there a reason you can't use conda-incubator/setup-miniconda to install Mambaforge? The README in that GHA Action has a lot of info on some gotchas.

tuanpham96 commented 1 year ago

@jaimergp Sorry for the late response. Thanks for the clarification. I remember the last time I tried I still had to change the path manually after installation before calling conda init (since conda would not be found without changing the path). Seems that @ReimarBauer also had to do that from https://github.com/conda-forge/miniforge/issues/421#issuecomment-1474819033.

Btw, is there a reason you can't use conda-incubator/setup-miniconda to install Mambaforge?

I'm using Gitlab CI and I'm not aware if there's an equivalent for it.

jaimergp commented 1 year ago

I see! That makes sense now.

Let me expose how this works and let's see if we can find the issue.

/AddToPath=1 will only work with /InstallationType=JustMe for CVE reasons.
If that's checked, this code block will be run.
_nsis.py comes from this file. In particular, the add_to_path function.
- This function relies on this other module being present too in Lib/. This involves manipulating the Windows registry, so if it worked, we should see changes in some of the keys mentioned, like HKCU\Environment (JustMe) or (only in old installs) HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Environment (All Users).
- This is then broadcasted.

So, things to check:

INSTDIR\Lib contains _nsis.py and _add_to_path.py
Assuming all those steps correctly ran, everything should be working because we haven't touched that part in a while (only the CLI flags that control whether this is run or not for AllUsers). So one way to test whether it actually did what it's supposed to do is to dump the registry keys mentioned there after the install:

% reg query HKCU\Environment /v PATH
:: for good measure, let's echo it too:
% echo %PATH%

Let me know if that helps!

conda-forge / miniforge