venv question

[dtenny]

I have a venv that I need to use with py4cl2-cffi. Up until now I have been firing up emacs/slime from a command line whose process has already done the .env/bin/activate.

That's fairly inconvenient, I'd like to configure from lisp after lisp has started. I see in the readme the following:

However, pyvenv environments require that users set python-executable-path manually, before loading py4cl2-cffi.

I suppose the key word is 'manually'. However there needs to be a way to automate the process. Forgive the ASDF question, but is there a hook I can place on the :py4cl2-cffi/config processing before :py4cl2-cffi is loaded to bind *python-executable-path* at the right time when quickloading my system? It has this declaration:

:depends-on (:alexandria :py4cl2-cffi/config :py4cl2-cffi :float-features)

If there isn't a way to add hooks to the asdf definition, do you have other ideas about how I can select the venv before py4cl2:cffi initializes with the wrong paths?

Thanks

[dtenny]

Answering my own question, I suppose one solution is a dependency whose only purpose is to modify things set up by prior dependencies. I'll try that.

[digikar99]

Yes, one solution would indeed be to set up a "my-system/config" that loads only the configuration systems of other systems and sets them up. The main "my-system" will contain the main code of your system and will also :depends-on "my-system/config".

But this looks ugly because, someone who wants to build upon your system "my-system-user" might also need to maintain another similar config system "my-system-user/config".

Interesting problem, thanks for bringing this up! I will ping here if I run into better ideas.

[dtenny]

I created a package to load so that my asd is:

  :depends-on (:alexandria :py4cl2-cffi/config 
                           :spacy-venv  ;modify :py4cl2-cffi/config:*python-executable-path*
                           :py4cl2-cffi :float-features :jdt-utils)

so that I can change *python-executable-path* as suggested in the readme, however imports still fail because py4cl2-cffi/config:*python-site-packages-path* is evaluated using the python3 in effect before I set it, and so doesn't pick up the correct site packages.

This code in my spacy-venv.lisp fixes things:

(setf py4cl2-cffi/config:*python-executable-path* "/home/dave/spacy-env/bin/python3")
(setf py4cl2-cffi/config::*python-site-packages-path* 
      (py4cl2-cffi/config::return-value-as-list
       (format nil "~a -c 'import sys; print(\"\\n\".join(sys.path))'"
               py4cl2-cffi/config:*python-executable-path*)))

It sets multiple variables in the correct order so that venv packages are found.

This could be avoided by providing a py4cld-cffi/config:init function that sets both config variables (and any others that may be necessary) in the correct order, to be called by my (or other user's) venv ASDF middleman package. However as that's a hack in the first place and you're probably hatching better alternatives I'm just mentioning this FYI, that setting *python-executable-path* alone is necessary but not sufficient.

Meanwhile, I have a solution that works so I can at least run my development with the appropriate venv without needing to start emacs from the venv.

[digikar99]

So, a few solutions come to mind -

1. Load py4cl2-cffi/config and set the config variables as soon as you can after the lisp starts

This way, whenever py4cl2-cffi is loaded later by anyone, it will pick up the correct config variables.

2. Use uiop:getenv to obtain values from the OS environment

The environment variables can be set in your .profile or equivalent. The current ways of finding the variables will be used as a fallback.

3. Provide a reconfigure function

If I understand correctly, this reconfigure function will (i) set the config variables from the values provided as arguments (similar to the init function you are proposing?) and (ii) recompile and reload the py4cl-utils and optionally py4cl-numpy-utils shared libraries.

However, this still assumes that no one (including your dependencies) has called (pystart) before the reconfiguration. (pystart) is called almost anytime one uses any function in the py4cl2-cffi library, including: pycall, pyvalue, import-module, defpymodule (with :cache t the default), defpyfun, raw-py.

Otherwise, if reconfigure does get called after pystart, then the effects will be unpredictable and enough to produce segmentation faults.