Environments not being set correctly

This is in relation to a discussion here.

Basically, I have a labscript installation with gui launching shortcuts that used to work. Then one day they stopped working. I saw no indication of what was wrong, but I could launch from the terminal using python -m blacs just fine. I spent a little time today looking in to this and I suspect that something hinky is going on with the environment activation. To help me actually get tracebacks, I am running the blacs.exe instead of the blacs-gui.exe. For everything below, I've run the executable from the command line in the relevant environment Scripts directory. I'm running Windows 10 20H2 with a Miniconda installation.

If I run blacs.exe or blacs-gui.exe from my labscript environment (python 3.8.11), everything works.

If I run blacs.exe from the 'base' environment (python 3.7.13), or a bare command prompt, I get the following traceback:

(base) PS C:\Users\naqsL\Miniconda3\envs\labscript\Scripts> .\blacs.exe
Traceback (most recent call last):
File "C:\Users\naqsL\Miniconda3\envs\labscript\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\naqsL\Miniconda3\envs\labscript\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\naqsL\Miniconda3\envs\labscript\Scripts\blacs.exe\__main__.py", line 4, in <module>
File "C:\Users\naqsL\Miniconda3\envs\labscript\lib\site-packages\desktop_app\__init__.py", line 3, in <module>
from .shell import set_process_appid, install, uninstall
File "C:\Users\naqsL\Miniconda3\envs\labscript\lib\site-packages\desktop_app\shell.py", line 17, in <module>
from .windows import (
File "C:\Users\naqsL\Miniconda3\envs\labscript\lib\site-packages\desktop_app\windows.py", line 9, in <module>
from win32com.shell import shell, shellcon
File "C:\Users\naqsL\Miniconda3\envs\labscript\lib\site-packages\win32com\__init__.py", line 5, in <module>
import win32api, sys, os
ImportError: DLL load failed while importing win32api: The specified module could not be found.

Given that part of the point of using desktop-app is so that conda environments are properly activated, I suspect some subtle bug. That said, only one labscript installation of the five I actively run shows this problem, so I'm a little suspicious that it may actually be an issue with conda environments in general (and my one random computer has some random thing about the Miniconda installation that is borking everything, making reproduction of the problem pretty hard).

Anyway, if you can think of obvious tests I can try to get to the bottom of this, I'd be happy to try them out.

I might see if I can reproduce the issue, but if you want to debug further given you can reproduce it, here are some details that might help.

The only thing desktop-app does to "activate" an environment is modify three environment variables, CONDA_DEFAULT_ENV, CONDA_PREFIX, and PATH:

https://github.com/chrisjbillington/desktop-app/blob/e26b9cc565070022396f7afd938c1ca44026e6a0/desktop_app/environment.py#L136

CONDA_DEFAULT_ENV should be the name of the environment, CONDA_PREFIX the directory containing that environment, and PATH should be a bunch of things, prepended to whatever the PATH was before activation:

env['PATH'] = os.path.pathsep.join(
            [
                str(prefix),
                str(prefix / "Library" / "mingw-w64" / "bin"),
                str(prefix / "Library" / "usr" / "bin"),
                str(prefix / "Library" / "bin"),
                str(prefix / "Scripts"),
                env['PATH'],
            ]

PATH is the one that should matter for finding DLLs. The DLLs should be directly in the environment's prefix directory (str(prefix) above).

You might try adding some printlines to BLACS/__main__.py prior to the desktop_app import to see if any problem with these environment variables is apparent.

You might also or instead add printlines to desktop_app.environment.detect_conda_env() to see if it's the detection of an environment that is failing, rather than the activation.

Conda sets other environment variables too, but most aren't needed for running an app, so we don't bother with them. But, it's always possible that has changed. You might want to have a Python script print all environment variables with CONDA in their names to if anything differs between a normally activated conda environment on a computer where things work correctly vs the computer where things are breaking.

OK, so I have a small update.

I added print lines to blacs/__main__.py and got sensible things when running from the labscript environment. Saving them here so I don't have to look them up all the time.

Conda Name: labscript
Conda Prefix: C:\Users\naqsL\Miniconda3\envs\labscript
Path: C:\Users\naqsL\Miniconda3\envs\labscript;C:\Users\naqsL\Miniconda3\envs\labscript\Library\mingw-w64\bin;C:\Users\naqsL\Miniconda3\envs\labscript\Library\usr\bin;C:\Users\naqsL\Miniconda3\envs\labscript\Library\bin;C:\Users\naqsL\Miniconda3\envs\labscript\Scripts;C:\Users\naqsL\Miniconda3\envs\labscript\bin;C:\Users\naqsL\Miniconda3\condabin;C:\Program Files\Microsoft MPI\Bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\WINDOWS\System32\OpenSSH;C:\Program Files (x86)\IVI Foundation\VISA\WinNT\Bin;C:\Program Files\IVI Foundation\VISA\Win64\Bin;C:\Program Files (x86)\IVI Foundation\VISA\WinNT\Bin;C:\Program Files\Intel\WiFi\bin;C:\Program Files\Common Files\Intel\WirelessCommon;C:\Program Files\TortoiseHg;C:\Program Files\MiKTeX 2.9\miktex\bin\x64;.;C:\BitFlow SDK 6.30\Bin64;C:\BitFlow SDK 6.30\Bin32;C:\Program Files\CameraLink\Serial;C:\Program Files (x86)\CameraLink\Serial;C:\Program Files\FLIR Systems\Spinnaker\bin64\vs2015;C:\Program Files\FLIR Systems\Spinnaker\bin\vs2015;C:\Program Files\Wolfram Research\WolframScript;C:\Program Files\Graphviz\bin;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\dotnet;C:\Users\naqsL\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Andor SDK3;C:\Users\naqsL\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files (x86)\Silicon Laboratories\ClockBuilder Pro\Bin;C:\Program Files\Elmer 9.0-Release\bin;C:\Users\naqsL\AppData\Local\GitHubDesktop\bin

Trying to run from base, I still get the same error as before. So I moved the print statements to desktop_app.windows just before the win32com imports. When called from the base environment, I get

Conda Name: base
Conda Prefix: C:\Users\naqsL\Miniconda3

What is confusing to me is that this environment has the pywin32 module, so it should still work. On a hunch, I upgraded the base environment's python to 3.8 (which matches the labscript environment). Now calling blacs.exe from base works. Leaving the print statements in windows, I see four calls: the first shows Conda Name: base, the rest show Conda Name: labscript.

So I guess this comes down to stupidity with conda environments sometimes finding DLLs in the wrong environments and failing to load if the python versions don't match. Unfortunately this is something I've seen before and it is crazy hard to debug since the failure is difficult to replicate as it necessitates having multiple environments with different python versions. Yuck. Maybe I'll try again to find a repeatable failure mode so I can submit another bug to conda.

Ah, and it appears that changing to base python version did not fix the actual shortcuts or running from a bare command prompt, which is curious. Checking a little, it appears it finds any random ol' python on the system's path and uses it. Suppose that shouldn't surprise me too much. This computer uses a python that came bundled with an entirely different application.

chrisjbillington / desktop-app

Environments not being set correctly #20