C API (`Python.h`) broken

[dmsul]

Actual Behavior

If Vim 8.1 for Windows accesses python37.dll, Vim crashes with the following error:

Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000874 (most recent call first):

This does not occur with python37.dll file from python.org.

Expected Behavior

Expect python37.dll to function without crashing other applications.

Steps to Reproduce

0. Fresh Windows 10 install (probably not necessary, but that's where I'm at).
1. Install Miniconda 3.7 64-bit. Add to PATH. (This is not absolutely necessary to reproduce but makes it easier on the Vim side.)
2. Install 64-bit Vim 8.1, e.g., from here (EDIT: specific file here. However, I have also compiled Vim myself from source and the problem remains.
3. Run Vim and call :py3 print('hello'). Should output 'hello', but instead crashes and kills Vim.

Anaconda or Miniconda version:

Miniconda 3.7.1. Problem persists after conda install python=3.7.2.

Operating System:

Windows 10 64-bit

conda info

     active environment : None
       user config file : C:\Users\Daniel\.condarc
 populated config files : C:\Users\Daniel\.condarc
          conda version : 4.6.4
    conda-build version : not installed
         python version : 3.7.2.final.0
       base environment : C:\Anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
                          https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : C:\Anaconda3\pkgs
                          C:\Users\Daniel\.conda\pkgs
                          C:\Users\Daniel\AppData\Local\conda\conda\pkgs
       envs directories : C:\Anaconda3\envs
                          C:\Users\Daniel\.conda\envs
                          C:\Users\Daniel\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.6.4 requests/2.21.0 CPython/3.7.2 Windows/10 Windows/10.0.10240
          administrator : False
             netrc file : None
           offline mode : False

conda list --show-channel-urls

$ conda list --show-channel-urls
# packages in environment at C:\Anaconda3:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py37_0    defaults
ca-certificates           2019.1.23                     0    defaults
certifi                   2018.11.29               py37_0    defaults
cffi                      1.11.5           py37h74b6da3_1    defaults
chardet                   3.0.4                    py37_1    defaults
conda                     4.6.4                    py37_0    defaults
conda-env                 2.6.0                         1    defaults
console_shortcut          0.1.1                         3    defaults
cryptography              2.4.2            py37h7a1dbc1_0    defaults
idna                      2.8                      py37_0    defaults
menuinst                  1.4.14           py37hfa6e2cd_0    defaults
openssl                   1.1.1a               he774522_0    defaults
pip                       18.1                     py37_0    defaults
pycosat                   0.6.3            py37hfa6e2cd_0    defaults
pycparser                 2.19                     py37_0    defaults
pyopenssl                 18.0.0                   py37_0    defaults
pysocks                   1.6.8                    py37_0    defaults
python                    3.7.2                h8c8aaf0_2    defaults
pywin32                   223              py37hfa6e2cd_1    defaults
requests                  2.21.0                   py37_0    defaults
ruamel_yaml               0.15.46          py37hfa6e2cd_0    defaults
setuptools                40.6.3                   py37_0    defaults
six                       1.12.0                   py37_0    defaults
sqlite                    3.26.0               he774522_0    defaults
urllib3                   1.24.1                   py37_0    defaults
vc                        14.1                 h0510ff6_4    defaults
vs2015_runtime            14.15.26706          h3a45250_0    defaults
wheel                     0.32.3                   py37_0    defaults
win_inet_pton             1.0.1                    py37_1    defaults
wincertstore              0.2                      py37_0    defaults
yaml                      0.1.7                hc54c509_2    defaults

[dmsul]

I've confirmed this bug exists after conda install python=3.6. On an older machine with Anaconda Python 3.6.5 (compiled March 2018), I compiled the latest version of Vim against the old python36.dll from last March and Vim was able to use it just fine.

If I drop in the old python36.dll into the new Anaconda environment, it breaks on all version of Vim 8.1. So it's probably more than just that one DLL file.

[mingwandroid]

3\. Install 64-bit Vim 8.1, e.g., from [here](https://github.com/vim/vim-win32-installer/releases). However, I have also compiled Vim myself from source and the problem remains.

This is not a URL to a file. Which exact file did you use? If it doesn't matter which is used then please mention that.

[mingwandroid]

This is an interaction between two different bits of software, one of which we don't provide. It is not our software that causes this (for we have thousands of packages that run just fine under our python interpreter).

If you need this fixed you are going to have to solve it yourself (or ask the Vim developers to lend a hand) I am afraid.

Having said that, I am in no way surprised that software compiled against one header and DLL fails to run when that DLL is replaced with another compiled entirely differently (we build nearly everything from source). Also you should not expect copying DLLs around to work. Our Python packages are complete packages with interrelated bits that absolutely must be located together, either in the same directory or relatively (this is not a choice for you to make, each file must continue to meet the criteria. by which is expected to be located by the interpreter). That this might have worked in the past is nothing more than an interesting curiosity.

You say it crashes when built from source, do you have a stack trace?

[dmsul]

@mingwandroid I've updated my original post to include the binary I used.

The copying DLLs around is only something I've done to try to pin down the issue. It's not something I've had to resort to in the past. Everything has worked fine, and pointing Vim to a local DLL is SOP for as long as Vim has offered Python support. The only reason I discovered this at all was that I was doing a clean OS install yesterday.

The only stack trace I get is the runtime one copied above. It's the same whether I used a pre-compiled binary from vim.org or one I've compiled myself against the specific Anaconda DLL.

To summarize (each these is a full Python installation after completely deleting all other Python installations, no DLL copy/pasting):

1. Older Anaconda DLLs (python36.dll for v3.6.5 dated 29 March 2018) + Vim 8.1.36 = GOOD. (My setup on multiple machines for a long time.)
2. Older Anaconda DLLs + Vim 8.1.960 (latest Vim version) = GOOD.
3. New python.org DLLs (python37.dll for 3.7.2) + Vim 8.1.960 = GOOD.
4. New Anaconda DLLs (python36.dll, python37.dll, the later for 3.7.1 and 3.7.2) + Vim 8.1.36 or Vim 8.1.960 = BAD.

To test, I opened Vim (both terminal Vim and gVIim) with no vimrc, then typed :py3 print('hello'). In the good cases, Vim prints hello. In the bad case, you get the following error and then the OS kills the process (error only visible with terminal Vim because OS kills the GUI process too fast):

Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000874 (most recent call first):

If you try to use any Python-dependent plugins for Vim (e.g., YouCompleteMe, ~19k stars, ~2k forks) you get the same error.

So it really seems like the bug is somewhere in the new Anaconda DLLs. I'm spinning up some Win 10 VMs to see if something else in my OS installation caused this, something with MSVC maybe. But it sure seems like it's the Anaconda DLLs that break otherwise functioning Python code.

[mingwandroid]

So it really seems like the bug is somewhere in the new Anaconda DLLs

I was hoping to raise in you the potential understanding that there may be no bug at all just incompatibility. Python does not lock down a Binary API or interface, or provide any compliance test-suite to make sure that Python DLLs conform to some specification. Software compiled against one Python header and DLL has little guarantee of running when compiled against another, still there's a reasonable chance it could work within some very controlled limits (exact same version number for example). You classify this as this a bug in our software when all you've shown is that our Python DLLs are not compatible with those that Vim expects.

If you want to figure this out, please debug it with a debugger against a source build. I know nothing about Vim, having barely used it and do not know the source code or how to build it.

If you want to figure this out in a way that helps the ecosystem, you may want to see if you can contribute towards a Vim package with conda-forge. Having that would certainly make debugging these sorts of things more of a team effort.

[dmsul]

Debugging the source build of Vim showed the error is in Py_Initialize which, as I understand it, is the first function called when extending Python through the C-API (see here).

I can now replicate the bug independently of Vim in a minimal working example that just tries to call Py_Initialize:

#include <stdio.h>
#include <Python.h>

int main(){
    puts("Before");
    Py_Initialize();
    puts("After");
    return 0;
}

The following sequence of commands shows that when newer h, lib, and/or dll are used at runtime, you get the error. The first compilation of and call to main.exe uses the older files. The second compilation (in the py36_today environment) uses the new files and hits the error. The third call to main.exe is outside the py36_today environment, so it references the older Python 3.6 files at runtime and doesn't hit the bug, even though it was compiled against the newer Python 3.6 files.

C:\Users\sullivan\Dropbox\projects\conda_vim
λ set LIB=%LIB%;c:\anaconda3\envs\py36_today\libs;c:\anaconda3\libs

C:\Users\sullivan\Dropbox\projects\conda_vim
λ cl main.c /Ic:\anaconda3\include
Microsoft (R) C/C++ Optimizing Compiler Version 19.16.27027.1 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

main.c
Microsoft (R) Incremental Linker Version 14.16.27027.1
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:main.exe
main.obj

C:\Users\sullivan\Dropbox\projects\conda_vim
λ main.exe
Before
After

C:\Users\sullivan\Dropbox\projects\conda_vim
λ del main.exe main.obj

C:\Users\sullivan\Dropbox\projects\conda_vim
λ activate py36_today

C:\Users\sullivan\Dropbox\projects\conda_vim
(py36_today) λ del main.exe main.obj

C:\Users\sullivan\Dropbox\projects\conda_vim
(py36_today) λ cl main.c /Ic:\anaconda3\envs\py36_today\include
Microsoft (R) C/C++ Optimizing Compiler Version 19.16.27027.1 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

main.c
Microsoft (R) Incremental Linker Version 14.16.27027.1
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:main.exe
main.obj

C:\Users\sullivan\Dropbox\projects\conda_vim
(py36_today) λ main.exe
Before
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000c64 (most recent call first):

C:\Users\sullivan\Dropbox\projects\conda_vim
(py36_today) λ deactivate
DeprecationWarning: 'deactivate' is deprecated. Use 'conda deactivate'.

conda.bat deactivate

C:\Users\sullivan\Dropbox\projects\conda_vim
λ main.exe
Before
After

[dmsul]

I tried to dig into this further but I've already hit the limit of my C knowledge.

[mingwandroid]

Great, thank you for your help. I will try to see what's going on here.

[dmsul]

I found the following, based off of this StackOverflow answer.

If I set either of the following environment variables, the error goes away even though c:\Anaconda3 is in PATH.

set PYTHONPATH=c:\Anaconda3\Lib
set PYTHONHOME=c:\Anaconda3

With at least one of these variables set, I get

C:\Users\Daniel\Dropbox\projects\conda_vim
λ main.exe
Before
After

I also tested with Vim and it works there as well, as long as one of those is set. So something in the old DLL or LIB files maybe defaulted to PATH if there was no PYTHONPATH or PYTHONHOME set, and the new ones don't? This is a bit beyond me, but it's something, I guess.

[dmsul]

Unfortunately, setting PYTHONPATH and PYTHONHOME at the system level messes things up when using environments.

[dmsul]

It would be really nice if I could get an update on this. I can't risk losing my old, working environments, but conda is now trying to force me to update Python which will introduce the broken DLLs/headers. If I try to install a new package, update an old one, etc., it also forces an update to the Python version (see picture). I can't even update conda itself because then it will also update Python to 3.7 and then the C API will break.

[msarahan]

This may help you:

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html#preventing-packages-from-updating-pinning

Sorry, but we have more urgent issues right now and cannot spend time on fixing this. We will get to it, and it is helpful to ping us once in a while to remind us.

[dmsul]

@msarahan The pinning worked to prevent auto-update of default Python. Thank you. Re the original problem, I'm happy to help any way I can. It seems like something at compile-time with env vars, but still happy to help.

[dmsul]

The problem persists in the Python 3.7.3 release compiled March 27 2019

C:\Users\sullivan\Dropbox\projects\conda_vim
(py37) λ conda list
# packages in environment at C:\Anaconda3\envs\py37:
#
# Name                    Version                   Build  Channel
ca-certificates           2019.1.23                     0
certifi                   2019.3.9                 py37_0
openssl                   1.1.1b               he774522_1
pip                       19.0.3                   py37_0
python                    3.7.3                h8c8aaf0_0
setuptools                40.8.0                   py37_0
sqlite                    3.27.2               he774522_0
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.15.26706          h3a45250_0
wheel                     0.33.1                   py37_0
wincertstore              0.2                      py37_0

C:\Users\sullivan\Dropbox\projects\conda_vim
(py37) λ python
Python 3.7.3 (default, Mar 27 2019, 17:13:21) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

C:\Users\sullivan\Dropbox\projects\conda_vim
(py37) λ .\configure.cmd
"Configuring Visual Studio..."
**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.7
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

C:\Users\sullivan\Dropbox\projects\conda_vim
(py37) λ .\build.bat

cl main.c /Ic:\Anaconda3\envs\py37\include
Microsoft (R) C/C++ Optimizing Compiler Version 19.16.27027.1 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

main.c
Microsoft (R) Incremental Linker Version 14.16.27027.1
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:main.exe
main.obj

C:\Users\sullivan\Dropbox\projects\conda_vim
(py37) λ main.exe
Before
Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000b9c (most recent call first):

C:\Users\sullivan\Dropbox\projects\conda_vim
(py37) λ set PYTHONHOME=c:\Anaconda3\envs\py37

C:\Users\sullivan\Dropbox\projects\conda_vim
(py37) λ main.exe
Before
After

C:\Users\sullivan\Dropbox\projects\conda_vim
(py37) λ

[dmsul]

I've gotten a little closer, I think. I've changed the file main.c to

#include <Python.h>
#include <stdio.h>

int main(){
    puts("Before");
    printf("Py_GetPythonHome: %ls \n", Py_GetPythonHome());
    printf("Py_GetPrefix: %ls \n", Py_GetPrefix());
    printf("Py_GetPath: %ls \n", Py_GetPath());
    puts("Before 2");
    Py_Initialize();
    puts("After");
    return 0;
}

The functions Py_GetPythonHome() and Py_GetPath() are discussed here and can be called before Py_Initialize().

If I compile against the old binaries, I get this:

C:\Users\sullivan\Dropbox\projects\conda_vim
λ main.exe
Before
Py_GetPythonHome: (null)
Py_GetPrefix: C:\Anaconda3
Py_GetPath: C:\Anaconda3\python36.zip;C:\Anaconda3\Lib;C:\Anaconda3\DLLs;C:\Users\sullivan\Dropbox\projects\conda_vim
Before 2
After

Against new binaries:

C:\Users\sullivan\Dropbox\projects\conda_vim
(py37) λ main.exe
Before
Py_GetPythonHome: (null)
Py_GetPrefix: C:\Anaconda3\envs\py37
Py_GetPath: C:\Anaconda3\envs\py37\python37.zip;.\DLLs;.\lib;C:\Users\sullivan\Dropbox\projects\conda_vim
Before 2
Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

Current thread 0x000026dc (most recent call first):

so it's looking for DLLs and Lib only in the current directory. After setting PYTHONHOME, Py_GetPath looks for this directories where it should instead of falling back to ..

C:\Users\sullivan\Dropbox\projects\conda_vim
(py37) λ set PYTHONHOME=c:\Anaconda3\envs\py37

C:\Users\sullivan\Dropbox\projects\conda_vim
(py37) λ main.exe
Before
Py_GetPythonHome: c:\Anaconda3\envs\py37
Py_GetPrefix: c:\Anaconda3\envs\py37
Py_GetPath: C:\Anaconda3\envs\py37\python37.zip;c:\Anaconda3\envs\py37\DLLs;c:\Anaconda3\envs\py37\lib;C:\Users\sullivan\Dropbox\projects\conda_vim
Before 2
After

[dmsul]

Monthly ping that the C API is broken.

[yatsky]

Hit the same barrier. vim 8.1.1302 (x64) python 3.7.3

[pnorton-usgs]

I have run into the same problem on macos 10.13.6 with anaconda python 3.7.3 and 3.6.8. I am using the python C API in compiled C and Fortran codes. Neither my code nor the example C API programs from python.org work with anaconda python 3.7.3 or 3.6.8 (https://python.readthedocs.io/en/latest/extending/embedding.html).

Programs crash with the following error:

Could not find platform independent libraries Could not find platform dependent libraries Consider setting $PYTHONHOME to [:] Fatal Python error: initfsencoding: unable to load the file system codec ModuleNotFoundError: No module named 'encodings'

All code works with anaconda python 2.7.15.

If I set PYTHONHOME=~/anaconda3 then my code works as expected. Since I make extensive use of environments I don't consider this to be a good long term solution.

[dmsul]

@pnorton-usgs Above I've posted a very basic C program that isolates exactly where the problem is. I don't have access to MacOS, would you be able to compile/run that code and post the results here?

[pnorton-usgs]

@dmsul I compiled and ran the test program for python 3.7.3 and 2.7.15. The results are below.

1) python 3.7.3 and PYTHONHOME not set NOTE: I have no idea where it got the /opt/concourse/... directory from. This directory does not exist on my system.

(1:194)$ ./simple_test
Before
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Py_GetPythonHome: (null)
Py_GetPrefix: /opt/concourse/worker/volumes/live/313743eb-3595-4b53-4518-f77d2fd60714/volume/python_1553722001805/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol
Py_GetPath: /opt/concourse/worker/volumes/live/313743eb-3595-4b53-4518-f77d2fd60714/volume/python_1553722001805/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python37.zip:/opt/concourse/worker/volumes/live/313743eb-3595-4b53-4518-f77d2fd60714/volume/python_1553722001805/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.7:/opt/concourse/worker/volumes/live/313743eb-3595-4b53-4518-f77d2fd60714/volume/python_1553722001805/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/lib-dynload
Before 2
Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007fffb50b1380 (most recent call first):
Abort trap: 6

2) python 3.7.3 and PYTHONHOME=~/anaconda3

(1:192)$ ./simple_test
Before
Py_GetPythonHome: /Users/xxx/anaconda3
Py_GetPrefix: /Users/xxx/anaconda3
Py_GetPath: /Users/xxx/anaconda3/lib/python37.zip:/Users/xxx/anaconda3/lib/python3.7:/Users/xxx/anaconda3/lib/python3.7/lib-dynload
Before 2
After

3) python 2.7.15 and PYTHONHOME not set.

(1:207)$ ./simple_test_py27
Before
Py_GetPythonHome: (null)
Py_GetPrefix: /Users/xxx/anaconda3/envs/bandit
Py_GetPath: /Users/xxx/anaconda3/envs/bandit/lib/python27.zip:/Users/xxx/anaconda3/envs/bandit/lib/python2.7/:/Users/xxx/anaconda3/envs/bandit/lib/python2.7/plat-darwin:/Users/xxx/anaconda3/envs/bandit/lib/python2.7/plat-mac:/Users/xxx/anaconda3/envs/bandit/lib/python2.7/plat-mac/lib-scriptpackages:/Users/xxx/anaconda3/envs/bandit/lib/python2.7/lib-tk:/Users/xxx/anaconda3/envs/bandit/lib/python2.7/lib-old:/Users/xxx/anaconda3/envs/bandit/lib/python2.7/lib-dynload
Before 2
After

[dmsul]

@pnorton-usgs Awesome, thank you.

[tanweihou]

@dmsul Hi, I came across to this issue after updating both Anaconda and Vim. Did you manage to solve the problem ?

[dmsul]

@tanweihou No. I worked around the problem by installing a standard python distribution for Vim to access, and installing Anaconda on the side for my actual research work.

[joelfrederico]

I'm seeing this issue still. It looks like the owner has left Anaconda development. Can we get a dev in this thread?