Could you please revisit the Python-2.x-and-Python-3.x workflow with MacVim

[cryptoeraser]

Hi, This is a great reference and it seems like many people used it in the past: Python-2.x-and-Python-3.x

However, I would say, the solution suggested at the link above is failing with multiple different issues.

Could you (anyone with experience) please update that solution, and maybe in that way we can make sure that this works with the latest Python2.x and Python3.x versions and pyenv, conda etc.

For example, in my case the pyenv installation for Python2.x and Python3.x described in that link was extreemly painful.

Then the homebrew bit did not work.

I also was not sure if the dynamic linking was working at all. My otool -L output is:

otool -L _ctypes.so
_ctypes.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)

And I never managed to get this bit to work:

$ otool -L $HOME/.pyenv/versions/2.7.11/lib/python2.7/lib-dynload/_ctypes.so
/Users/foo/.pyenv/versions/2.7.11/lib/python2.7/lib-dynload/_ctypes.so:
    /Users/foo/.pyenv/versions/2.7.11/lib/libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

Many plugins are now using Python3 and some are still using Python2. A hybrid solutions is becoming critical even more now.

Thank you very much.

[eirnym]

You look into a very old and outdated documentation. Please, visit https://github.com/vim/vim project where build process described much better. MacVim adds only GUI and there's nothing changed with python.

[eirnym]

Additionally, you can download a fresh build, setup a few variables if paths are not standard and try to use it. Recent versions of Python 2.7 and Python 3.7 works well with MacVim (please, consult with a compatibility notes), as well as vitrual environment.

However, I've never tried to use conda and can't say if it's compatible with an implementation from python.org on a native library level.

[cryptoeraser]

Hi @eirnym,

Thank you very much for the information. I have been trying to build MacVim with dual support for both Python2 and Python3. I have always been building my Vim and MacVim from source, however, a couple of days ago, I realized that now I need both Python2 and Python3.

I have been trying to get this working without success so far. Here is what I have:

1) I have my two Pythons: Python 2.7.15 and Python 3.5.6. They both live in their respective .pyenv folders and they are working. The dynamic libs are there:

~/.pyenv/versions/3.5.6/Python.framework/Versions/3.5/lib/libpython3.5.dylib
~/.pyenv/versions/2.7.15/lib/libpython2.7.dylib

2) MacVim is built against these pythons in dynamic mode:

export vi_cv_path_python=/Users/me/.pyenv/versions/2.7.15/bin/python2
export vi_cv_path_python3=/Users/me/.pyenv/versions/3.5.6/Python.framework/Versions/3.5/bin/python3
export vi_cv_dll_name_python=/Users/me/.pyenv/versions/2.7.15/lib/libpython2.7.dylib
export vi_cv_dll_name_python3=/Users/me/.pyenv/versions/3.5.6/Python.framework/Versions/3.5/lib/libpython3.5m.dylib

with the following command:

/configure --with-features=huge --enable-multibyte --enable-terminal --enable-cscope --enable-pythoninterp=dynamic --enable-python3interp=dynamic --with-compiledby=Symbolix --with-properly-linked-python2-python3

So the vim --version|grep python command returns:

+comments          +libcall           +python/dyn        +viminfo
+conceal           +linebreak         +python3/dyn       +vreplace

4) The options are set in a minimal vmirc file, and running :set lists the relevat options:

pythonthreedll=~/.pyenv/versions/3.5.6/Python.framework/Versions/3.5/lib/libpython3.5.dylib
pythonthreehome=~/.pyenv/versions/3.5.6/Python.framework/Versions/3.5
pythondll=~/.pyenv/versions//2.7.15/lib/libpython2.7.dylib
pythonhome=~/.pyenv/versions/2.7.15

Everything seems to be in place, yet I am still getting the nasty crash when trying to mix py2 and py3. This is frustrating :(

Vim: Caught deadly signal SEGV
Vim: Finished.

Has anyone got this mixed Python workflow working?

[cryptoeraser]

Some updates: So with the released MacVim binary, I am getting this:

E837: This Vim cannot execute :py3 after using :python
E263: Sorry, this command is disabled, the Python library could not be loaded.

Is it correct to assume that, the release binary, will run with only one of the Pythons. And it does not support Python2 and Python3 at the same time, even when the options are set?

[cryptoeraser]

I think I have made some progress:

mvim -u NONE -U NONE --noplugin -c 'py print("test")'
--> IS WORKING

mvim -u NONE -U NONE --noplugin -c 'py3 print("test")' 
--> IS WORKING

mvim -u NONE -U NONE --noplugin -c 'py print("test")' -c 'py3 print("test")'
--> IS WORKING

mvim -u NONE -U NONE --noplugin -c 'py3 print("test")' -c 'py print("test")'
--> IS NOT WORKING

Before, only the first two were working. Now only the last one is crashing Vim. Any ideas?

[eirnym]

The story behind enabling only one python version and not both is quite simple (and it's partially described in :help python page)

Both libraries export almost the same API. So statically linking both is impossible from the beginning.

With dynamic linking you either load one and use easy mapping of exported symbols for almost the same code of :py and py3 commands, or have a headache to work with library handles and calling all functions in exotic way all the time. The second method provides you ability to use both dynamically loaded libraries simultatiously, but it's hard to keep implementation quality on a high level. Vim developers have chosen the easiest one and in most cases you don't really need both Python versions in the same Vim process.

[eirnym]

I use in my setup Python3 by default, massively use virtual environments and the latest versions of most libraries from Python2 virtual environments is python3 compatible to have YCM working well.

[eirnym]

The latest build of MacVim (151) from Github doesn't crash for me.

[eirnym]

If you have better implementation of python bridge for Vim, you should propose your patch to Vim upstream.

[cryptoeraser]

@eirnym Thank you very much for your time and for the detailed explaination. I have been building and testing with snapshot 151 (but the released one).

So would you confirm that you have VIM running both versions of Python (Py2 and Py3) as dynamically linked libs and with no issues what-so-ever when moving between :py and py3 commands and the other way around, when moving from :py3 to py? So when you run the little test above, all 4 options are working without a crash?

Also, this is from the docs.

Here Vim's behavior depends on the system in which it was configured. In a system where both versions of Python were configured with --enable-shared, both versions of Python will be activated at the same time. There will still be problems with other third party libraries that were not linked to libPython.

I don't understand that last bit:

There will still be problems with other third party libraries that were not linked to libPython.

Is there an example about this third-party-library incompatibility?

[cryptoeraser]

Despite the fact that I am building my Python2 and my Python3 using the --enable-shared flags and I am doing my best to build those as shared libs, there is one thing that bothers me and it is this:

I am not able to see the link to the dynamic library libpython2.7.dylib on the .so files inside my freshly built Python. For example, any dynamically linked python would look like the following, please note that I am trying this on Linux, but the case is similar on Darwin64.

ldd /usr/lib64/python2.7/lib-dynload/_ctypes.so
    linux-vdso.so.1 =>  (0x00007ffd4a5dd000)
    libffi.so.6 => /lib64/libffi.so.6 (0x00007f443bf1e000)
    libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007f443bb52000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f443b935000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f443b572000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f443b36e000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007f443b16a000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f443ae68000)
    /lib64/ld-linux-x86-64.so.2 (0x0000558a69696000)

So you can clearly see that the libpython2.7.so.1.0 library is requested. However, when I run the similar command on my OSX Python2, that I have dynamically built, I only get this:

otool -L _ctypes.so
_ctypes.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50)

So no references to the libpython2.7 file. This is a bit worrying. However, that is not the case for the actual executable, I can see the python binary is linked to the libpython2.7 file.

Maybe you can share your Python build flags? That would be very helpful. Cheers.

[eirnym]

So when you run the little test above, all 4 options are working without a crash?

yes, all 4 options works for me without a crash, it sais that one version is blocked because another version is activated. It's fine as I use vim plugins built against Python 3. Syntax compatibilities allow me to edit Python 2 code without changing tools like YCM, … Some libraries even started to drop support of Python 2.

Currently, I don't see any reason to have Python2 in ViM at all. July of 2020 is very soon.

Is there an example about this third-party-library incompatibility?

Potential points of incompatibility include all native modules and packages like sciencific packages, could be ctypes/cffi or any libraries with their own JIT. They could depend on a different versions of third-party libaries which could be incompatible with each other. As an alternative example of incompatibility, system version of a library is often differ than a version installed with third-party installation procedure (including manual installation).

So no references to the libpython2.7 file

This is about how Python works, please, look documentation for it.

[cryptoeraser]

Hello Again,

Yes. With that approach, it works for me as well. Actually I like it this way better. So, here is what I have and it works:

(1) I have two miniconda Pythons: Python 2.7.15 and Python 3.7.0
(2) I have the VIM options active in my .vimrc file.
    and these are pointing to their respective conda Python builds:
            pythonthreedll,
            pythonthreehome,
            pythondll,
            pythonhome,
(3) I have downloaded and installed the latest MacVim (snapshot_151)
    and it finds the dynamic python links. The crash protection is there, meaning,
    once the user invokes py2 or py3, it stays in that way. You just get a warning
    in case you try to invoke the other Python.
(4) I also can confirm that I have installed copule of packages with conda
    and I have full access to those through my :py2 or :py3 interfaces.

So it makes sense to keep it safe in that way.

Just couple of last questions, before I close this issue. And once again, thanks for the help!

1. If I still would like to build MacVim from source, what are the flags that will ensure the safety system that is existent in the current binary release (151) of MacVim will be available in my custom build as well? The reason I am asking is that, I would like to tweak my MacVim (build it myself) yet still would like to have that Python safety in place. The one that goes "You have invoked Py2, you can't invokve Py3 at this moment." or the oposite.

2. Is it possible to hardcode the dynamic python links at build time, so we do not need to set the options in the .vimrc file. I thought that is possible through flags such as: -DYNAMIC_PYTHON_DLL and -DYNAMIC_PYTHON3_DLL at build time?

NOTE: I guess the most important thing is that, we should remember that the Python that VIM uses, has nothing to do with our shell Python and our Python development environment. These are two seprate things. Any VIM plugin that needs Python, our :py scripts, etc. all of these are relying on the linked Python (and probably YCM etc. as these have internal :py calls). On the other hand, the Python project we are building, that one relies on the Python environment outside VIM. Including YCM's completion workflow (cause it needs to load and index the modules we are using for our project.

Just writing this down as a side note :) Since I have noticed that most of the people tend to think that they need all of the packages related to their Python development, be existing under VIM's Python. That is NOT the case. The packages that need to be available to VIM's Python are only those needed by your scripts and any plugin specific needs.

For example, your VIM Python can have only one package installed pyyaml and that is it. Only because you have a VIM plugin that needs to read .yaml files. However, on the other hand you might be working on a complex Python project that needs numpy, scipy, beautifulsoup etc. These have nothing to do with your VIM Python.

[eirnym]

Thank you for answers :)

As per ./configure --help you need to add two options listed below for each version of Python. One enables interpreter code and the second allows you to hardcode the default path of Python interpreter.

• for Python 2: --enable-pythoninterp=dynamic and --with-python-command=/path/to/python2.7
• For Python 3: --enable-python3interp=dynamic and --with-python3-command=/path/to/python3.x

this this will both enable Python2 and Python3 as dynamic.

Additionally, if you're user of MacPorts, change for vimrc bundled with MacVim paths were integrated after build 151 and will be rolled out with the next release.

[ychin]

@cryptoeraser do you still have questions on this? Do you still want to compile MacVim for simultaneous Python 2/3 usage?

If not, please close the issue.

1. … I would like to tweak my MacVim (build it myself) yet still would like to have that Python safety in place. The one that goes "You have invoked Py2, you can't invokve Py3 at this moment." or the oposite.

I believe the protection is always there, unless you followed the wiki and configured MacVim using --with-properly-linked-python2-python3. You can always re-configure MacVim to the defaults when building using make distclean && ./configure.

2. Is it possible to hardcode the dynamic python links at build time, so we do not need to set the options in the .vimrc file. I thought that is possible through flags such as: -DYNAMIC_PYTHON_DLL and -DYNAMIC_PYTHON3_DLL at build time?

You could, although as @eirnym said the default MacVim vimrc already does auto-detection for Python so if you installed Python using the normal methods (HomeBrew, MacPorts, the website), MacVim should find it. Otherwise you can use vi_cv_dll_name_python3 when running configure. Check out how we set it up for CI.

[eirnym]

@ychin Probably, we should just remove that page as it's really misleading people, and then close the issue.