Error importing built-in python module on Windows after upgrading vim + python

idbrii commented 1 year ago

Steps to reproduce

On gvim 9.0.1205, :py3 import socket produces an error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\idbrii\scoop\apps\python311\current\Lib\socket.py", line 51, in <module>
    import _socket
ModuleNotFoundError: No module named '_socket'

:!python import socket completes successfully.

However, it doesn't error on gvim_9.0.0844_x64 (the last version using python310.dll). The next version is gvim_9.0.0845_x64 which fails just like 9.0.1205. So it seems that something has failed in upgrading python 3.10 -> 3.11.

Setting 'pythonthreehome' fixes the issue, but it seems like something broke since that was previously not necessary.

I don't have PYTHONHOME or PYTHONPATH environment variables set. Before this issue, the registry file that scoop generates was enough for vim to find everything that python needs (see pep-0514).

More Details

_socket is apparently a C module, so I guess it needs to be found in the python dll?

Issue #2840 looked related (except I'm not using miniconda and I'm on Windows), so I tried the suggestions of setting pythonthreedll, clearing the pythonthree settings, and then setting pythonthreehome and only that last one worked:

set pythonthreehome=C:/Users/idbrii/scoop/apps/python311/3.11.1

The default values on my system:

:set pythonthreedll? pythonthreehome? 
  pythonthreedll=python311.dll
  pythonthreehome=

pythonthreehome has existed for a long time (since 94073167e3aa8cbe18380e93a2fc8e8165438cc8), but did something else change?

Expected behaviour

Module imports without error.

Version of Vim

9.0.1205

Environment

gvim 64 bit Windows 10

The shell and vim python versions match and I only have one version of py3.11 installed: :py3 import sys; print(sys.version) 3.11.1 (tags/v3.11.1:a7a450f, Dec 6 2022, 19:58:39) [MSC v.1934 64 bit (AMD64)] :!python import sys; print(sys.version) 3.11.1 (tags/v3.11.1:a7a450f, Dec 6 2022, 19:58:39) [MSC v.1934 64 bit (AMD64)]

I'm using zip builds from vim-win32-installer and official python builds installed by scoop (specifically python311). Ran the .reg so python is hooked up in the registry and lots of python functionality works fine in vim except for this module.

Logs and stack traces

:Verbose version

VIM - Vi IMproved 9.0 (2022 Jun 28, compiled Jan 15 2023 23:28:45)
MS-Windows 64-bit GUI version with OLE support
Included patches: 1-1205
Compiled by appveyor@APPVYR-WIN
Huge version with GUI.  Features included (+) or not (-):
+acl                +byte_offset        +conceal            -dnd                +float              +jumplist           +lua/dyn            +netbeans_intg      +printer            +signs              +tag_binary         -tgetent            +virtualedit        -xfontset
+arabic             +channel            +cryptv             -ebcdic             +folding            +keymap             +menu               +num64              +profile            +smartindent        -tag_old_static     +timers             +visual             -xim
+autocmd            +cindent            +cscope             +emacs_tags         -footer             +lambda             +mksession          +ole                +python/dyn         +sodium/dyn         -tag_any_white      +title              +visualextra        +xpm_w32
+autochdir          +clientserver       +cursorbind         +eval               +gettext/dyn        +langmap            +modify_fname       +packages           +python3/dyn        +sound              -tcl                +toolbar            +vreplace           -xterm_save
+autoservername     +clipboard          +cursorshape        +ex_extra           -hangul_input       +libcall            +mouse              +path_extra         +quickfix           +spell              -termguicolors      +user_commands      -vtp                
+balloon_eval       +cmdline_compl      +dialog_con_gui     +extra_search       +iconv/dyn          +linebreak          +mouseshape         +perl/dyn           +reltime            +startuptime        +terminal           +vartabs            +wildignore         
-balloon_eval_term  +cmdline_hist       +diff               -farsi              +insert_expand      +lispindent         +multi_byte_ime/dyn +persistent_undo    +rightleft          +statusline         -termresponse       +vertsplit          +wildmenu           
+browse             +cmdline_info       +digraphs           +file_in_path       +ipv6               +listcmds           +multi_lang         +popupwin           +ruby/dyn           -sun_workshop       +textobjects        +vim9script         +windows            
++builtin_terms     +comments           +directx            +find_in_path       +job                +localmap           +mzscheme/dyn       -postscript         +scrollbind         +syntax             +textprop           +viminfo            +writebackup        
   system vimrc file: "$VIM\vimrc"
     user vimrc file: "$HOME\_vimrc"
 2nd user vimrc file: "$HOME\vimfiles\vimrc"
 3rd user vimrc file: "$VIM\_vimrc"
      user exrc file: "$HOME\_exrc"
  2nd user exrc file: "$VIM\_exrc"
  system gvimrc file: "$VIM\gvimrc"
    user gvimrc file: "$HOME\_gvimrc"
2nd user gvimrc file: "$HOME\vimfiles\gvimrc"
3rd user gvimrc file: "$VIM\_gvimrc"
       defaults file: "$VIMRUNTIME\defaults.vim"
    system menu file: "$VIMRUNTIME\menu.vim"
Compilation: cl -c /W3 /GF /nologo -I. -Iproto -DHAVE_PATHDEF -DWIN32 -DHAVE_STDINT_H  -DFEAT_CSCOPE -DFEAT_TERMINAL -DFEAT_SOUND -DFEAT_NETBEANS_INTG -DFEAT_JOB_CHANNEL -DFEAT_IPV6 -DHAVE_INET_NTOP   -DFEAT_XPM_W32 -DHAVE_SODIUM -DDYNAMIC_SODIUM -DDYNAMIC_SODIUM_DLL=\"libsodium.dll\" /I "C:\libsodium\include" -DWINVER=0x0601 -D_WIN32_WINNT=0x0601 /source-charset:utf-8 /MP /Ox /GL -DNDEBUG /Zl /MT /D_CRT_SECURE_NO_DEPRECATE /D_CRT_NONSTDC_NO_DEPRECATE -DFEAT_OLE -DFEAT_MBYTE_IME -DDYNAMIC_IME -DFEAT_GUI_MSWIN -DFEAT_DIRECTX -DDYNAMIC_DIRECTX -DFEAT_DIRECTX_COLOR_EMOJI -DDYNAMIC_ICONV -DDYNAMIC_GETTEXT -DFEAT_LUA -DDYNAMIC_LUA -DDYNAMIC_LUA_DLL=\"lua54.dll\" -DFEAT_PYTHON -DDYNAMIC_PYTHON -DDYNAMIC_PYTHON_DLL=\"python27.dll\" -DFEAT_PYTHON3 -DDYNAMIC_PYTHON3 -DDYNAMIC_PYTHON3_DLL=\"python311.dll\" -DFEAT_MZSCHEME -I "C:\racket\include" -DMZ_PRECISE_GC -DDYNAMIC_MZSCHEME -DDYNAMIC_MZSCH_DLL=\"libracket3m_da32rk.dll\" -DDYNAMIC_MZGC_DLL=\"libracket3m_da32rk.dll\" -DFEAT_PERL -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DDYNAMIC_PERL -DDYNAMIC_PERL_DLL=\"perl532.dll\" -DFEAT_RUBY -DDYNAMIC_RUBY -DDYNAMIC_RUBY_DLL=\"x64-msvcrt-ruby300.dll\" -DRUBY_VERSION=30 -DFEAT_HUGE /Fd.\ObjGXOULYHRZAMD64/ /Zi
Linking: link /nologo /opt:ref /LTCG /HIGHENTROPYVA:NO oldnames.lib kernel32.lib advapi32.lib shell32.lib gdi32.lib  comdlg32.lib ole32.lib netapi32.lib uuid.lib user32.lib  /machine:AMD64 version.lib  winspool.lib comctl32.lib libcmt.lib oleaut32.lib  /nodefaultlib:lua54.lib  /STACK:8388608  /nodefaultlib:python27.lib /nodefaultlib:python311.lib    winmm.lib WSock32.lib Ws2_32.lib xpm\x64\lib-vc14\libXpm.lib  /PDB:gvim.pdb -debug

zewpo commented 1 year ago

Ahh, yes, I saw this one too. My mapleader plugin complains about it. I thought I missed something in upgrading my local instal of Python from 3.10 to 3.11.1 But, there was a similar problem going from 3.10 to 3.11.0. (ie. also related to path variables) python partially resolved the issue in Python 3.11.1. but it looks like there are still some related issues in python 3.11.1

I think there is a security footprint that they are trying to minimize with respect to dynamic dlls.

Remiol commented 1 year ago

Same problem here Importing almost all python modules gives an error message in 3.11.1 (ModuleNotFoundError). No problems in 3.11

tried idbrii's solution without success (I'm on win11)

DanielW10001 commented 1 year ago

Same problem for me, and idbrii's solution (pythonthreehome) fixes the issue. This might be worth investigating.

Remiol commented 1 year ago

Same problem for me, and idbrii's solution (pythonthreehome) fixes the issue. This might be worth investigating.

What did you do to make it work? idbrii's solution didn't work.

What are you default values? set pythonthreedll? pythonthreehome?

update: Python 3.11.2 resolves the problem

k-takata commented 1 year ago

update: Python 3.11.2 resolves the problem

Then, I think we can close this.

vim / vim