Envs built in Python 3.7 in anaconda powershell, not useable in Sublime

[tomslade1]

I built an environment in the anaconda console (my base Python environment has Python 3.7 so the environment was created using that version of Python) installing pandas & pandas-gbq.

I activated the environment using Conda and I attempted to run:

import pandas as pd

I got the following error log: Conda error.txt

If I attempt to run this directly via the anaconda powershell, it works fine.

Equally if I try and create the environment from scratch in the Conda package in Sublime, I select one of Python 3.6, 3.5 or 2.7; if I pick Python version 3.6 and then install the packages it works fine.

Is the Sublime Conda package not yet compatible with Python 3.7, and is that why this fails to run?

I've created other environments in the anaconda powershell, with the Python version set to 3.7 and they run fine Sublime (still activating via Conda), but they didn't require the pandas package.

[mandeep]

This shouldn't have anything to do with the python version. I think this more likely has to do with the way anaconda console handles paths for packages like pandas, numpy, etc. I'll look into this but in the meantime I'll release version 6.0 of sublime-text-conda that allows for the creation of 3.7 conda envs.

[tomslade1]

Thanks for adding the support for Python 3.7, you were (unfortunately) correct - looks like a pandas specific issue, as still getting the same error

[mandeep]

But everything works correctly if you solely use anaconda console?

[tomslade1]

Yep correct - if I load up the exact same environment in anaconda console and run the same script; it runs fine.

If I activate the environment in Sublime using Conda, then hit ctrl+b to run the script I get the above attached error log.

[mandeep]

Seems to be an issue with paths. I'm not sure there is a solution for this but I'll see if I can find something.

[jim-hart]

@mandeep It is a path issue; I've had similar issues with packages like pandas in sublime before I started using the Conda plugin. The only solution I found was to prepend the equivalent of conda activate {env_name} to the list of cmd args in the sublime-build file.

For example, if I had an environment named py37, I the cmd key would have a value of the following:

["\\${HOME}\\Anaconda3\\Scripts\\activate.bat", "py37",  "&&",  # activate command
 "\\${HOME}\\Anaconda3\\envs\\py37\\python.", "-u", "$file"]    # standard call to python 

I use the absolute path to activate.bat as Anaconda isn't on my system path; the same pattern should work for those with the Anaconda dir on their PATH.

To make this work with the Conda plugin, I've modified ExecuteCondaEnvironmentCommand to insert the necessary arguments:

class ExecuteCondaEnvironmentCommand(CondaCommand):
    """Override Sublime Text's default ExecCommand with a targeted build."""

    def run(self, **kwargs):
        try:
            environment = self.project_data['conda_environment']
            use_pythonw = self.settings.get('use_pythonw', False)
            run_through_shell = self.settings.get('run_through_shell', False)

            python_executable = 'pythonw' if use_pythonw else 'python'
            kwargs['cmd'] = self._build_cmd_list(
                python_executable, environment, kwargs['cmd'])
            kwargs['shell'] = run_through_shell

        except KeyError:
            pass

        if kwargs.get('kill') is not None:
            kwargs = {'kill': True}
        self.window.run_command('exec', kwargs)

    def _build_cmd_list(self, python_exe, environment, current_args):
        """
        Builds the argument sequence that will be used as the value for the `cmd` key of the
        build-system
        """
        if sys.platform == 'win32':
            bat_file = os.path.join(self.root_directory, 'Scripts', 'activate.bat')
            env_name = os.path.basename(environment)
            exe_path = os.path.normpath('{}\\{}'.format(environment, python_exe))
            prefix = [bat_file, env_name, '&&', exe_path]
        else:
            exe_path = '{}/bin/{}'.format(environment, python_exe)
            prefix = [os.path.normpath(exe_path)]
        return prefix + current_args[1:]

The behavior remains unchanged for non-win32 platform. I don't know if this is the best way to do it, but this method has always worked for me.

[mandeep]

@jim-hart thanks for letting me know!

I don't have a Windows machine on me so unfortunately I can't play around with things, but I'm interested in seeing why activate.bat is necessary. I really don't see why Conda should be on PATH for things to work correctly.

[jim-hart]

No problem!

I think it has something to do with DLL dependency and load semantics used by some packages (in our case, Pandas by extension through Numpy). This section on numpy troubleshooting sheds some light on the issue.

The problem seems to be package specific; I can load up a package like requests without any errors using the unmodified version of the plugin (no activate.bat required).

While a call to activate.bat isn't ideal, it avoids permanently inserting Anaconda somewhere on PATH.

EDIT: For reference, I get this error if I don't call activate.bat when trying to import numpy

Traceback (most recent call last):
  File "Q:\Git_Repos\#Projects#\PseudoStarch\test.py", line 2, in <module>
    import numpy
  File "C:\Users\J\Anaconda3\envs\py37\lib\site-packages\numpy\__init__.py", line 140, in <module>
    from . import _distributor_init
  File "C:\Users\J\Anaconda3\envs\py37\lib\site-packages\numpy\_distributor_init.py", line 34, in <module>
    from . import _mklinit
ImportError: DLL load failed: The specified module could not be found.

I believe this is the dependency Pandas is failing to import in the error log posted by @tomslade1

[ianhbell]

I have precisely the same problem. While conda+sublime used to work (yesterday), having just installed visual studio and Intel Fortran played with my path and now I see the same failure as the other users here. I agree that a conda activate env is likely the necessary solution.

[mandeep]

@ianhbell thanks for letting me know! I'll see if I can dig more into conda activate env and push a fix for this. If anyone wants to submit a PR I'd be more than happy to accept it!

[mandeep]

@jim-hart Were you able to check if the edited build script still works for older versions of Conda? I'm afraid that this change would break older Conda users.

[ianhbell]

I'm a newer conda user and this fix works fine for me. The fix from @jim-hart uses a backwards-compatible approach of calling the activate batch script. The new conda activate env approach is not likely to work on old conda. Of course you could also have a config flag to decide which approach to try.

[jim-hart]

@mandeep sorry for the delayed response, I didn't catch the tag notification when it was originally sent.

The fix should work for older versions, but I'm happy to verify. How far should I go back?

[mandeep]

No need to verify :). I expect it to work, but in the end I'd like to find a solution that does not involve using any of the activation scripts.

[jim-hart]

@mandeep I agree, I'd like to avoid using the activation scripts as well.

I did some digging and found a potential alternative. I compared the output of a call to os.environ when using the original ExecuteCondaEnvironmentCommand against the edited version that calls the activation script. I pulled out the differences (targeting new entries to os.envrion['PATH']) and narrowed down the issue to a single directory, ~\Anaconda3\envs\py37\Library\bin.

If this directory is not on Windows' path, it seems pandas and/or numpy can't find some of the compiled libraries they rely on.

To address this, we can add the following to ExecuteCondaEnvironmentCommand:

class ExecuteCondaEnvironmentCommand(CondaCommand):
    """Override Sublime Text's default ExecCommand with a targeted build."""

    os_env_path = os.environ['PATH']

    def __enter__(self):
        if sys.platform == 'win32':
            env_path = self.project_data['conda_environment']
            bin_path = os.path.join(env_path, 'Library', 'bin')
            os.environ['PATH'] = os.pathsep.join((bin_path, self.os_env_path))
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        os.environ['PATH'] = self.os_env_path

    def run(self, **kwargs):

        # .... snip ....

        with self:
            self.window.run_command('exec', kwargs)

I went with context manager protocols for the following reasons:

1. Modifications to os.envrion['PATH'] persist between file executions and propagate to child processes, so the PATH entry needs to be reset to its original value to avoid duplicate entries. It also avoids running files in the context of a modified environment PATH longer than necessary.
2. I felt it was the cleanest way to leave .run() untouched (with the exception of a single additional line).

While the operation is still a bit invasive, it avoids activation scripts entirely and limits environment modifications primarily to the target file's runtime.

What are your thoughts?

[ianhbell]

I can test this, but I am not yet convinced that we can avoid the environment activation script approach, but I would be happy to be wrong. I have both Intel development stack and conda environments on my computer, and I its not clear that adding the bin folder would be sufficient to resolve imports. But it would be easy enough to check. Can somebody push this to a branch so that I can checkout the proposed code?

[mandeep]

@jim-hart I like the idea and would happily accept a PR for this change. Many thanks for the effort! Having said that, I know the conda folks used to warn against modifying PATH (I'm not sure if this is still the case) as it could cause problems with other packages such as Qt5. I will try to go through the conda documentation to see if there's anything that might cause problems, and will reach out to the conda team if need be.

Here is what the activation script is doing so I guess it's okay to modify PATH in this manner: https://github.com/conda/conda/blob/7b16a9dfb0f4162908dc654eb51cd04f698e6485/conda/cli/activate.py#L24

We should probably add a check to make sure the conda version is 4.6 and higher.

[jim-hart]

@mandeep I want to say the warnings are about the install option that permanently places Anaconda on Windows' PATH, but I can see even temporary placement causing problems, and I'm all for erring on the side of caution here.

I'm using 4.7 locally and it places Library/bin on the PATH in addition to a number of other paths and entries. I've seen other packages that rely on compiled dependencies perform some variation of this process on their own through a .pth file placed in Lib/site-packages, so it may be a problem inherent to path resolution on Windows. I'm curious if there are alternative solutions though, so I will keep an eye out and report back if I ever find anything.

Anyway, its no problem, and thank you for looking into this further! I created PR #15 to address the issue.

[mandeep]

fixed by #15