Pre-exec support? (related to virtualenvs)

[mgrandi]

Hello,

I am interested in using honcho, but my project uses a virtualenv. I searched around, and couldn't find anything in the documentation, but i did find that you posted this in December , saying that you can just add the "activate" script to your Procfile and it should work

Well, I tried, that, but it seems the problem is that honcho sees that the source command exits (running source activate.sh in bash), and therefore kills everything else. (This is also assuming that the jobs are started in order, which seems to be this case but i feel like it shouldn't be relied on.)

my proc file: (mac os x 10.10 / python 3.4 if that matters)

venv: source <SOURCE_PATH>/venv_dir/bin/activate
ldb_server: python3 server.py config.yaml

and that produces:

Corvidae:daemon_src markgrandi$ honcho start
04:10:44 venv.1    | started with pid 10991
04:10:44 ldb_server.1 | started with pid 10992
04:10:44 venv.1    | process terminated
04:10:44 system    | sending SIGTERM to all processes
04:10:44 system    | sending SIGTERM to pid 10992
04:10:44 ldb_server.1 | process terminated

So since you said in the above linked bug report that you want to keep this language agnostic and therefore not really add 'specific support for virtualenvs', I propose adding something in the Procfile, or maybe a command switch that allows you to specify what should be run before and after honcho starts everything / kills everything.

It would execute it, and when it exits, honcho knows it can start running everything in the Procfile (as it does now) normally, and when those all exit/are terminated, honcho runs whatever is specified for 'post exec' (like running the 'deactivate' command for venvs). And this would also solve a potential race condition where its not specified anywhere that the honcho jobs are started in order, so to not have the possibility of the other jobs starting before they should. I do not know how this would effect the 'official' use case for Procfiles (heroku) and the exporting functionality

I'm guessing theoretically my specific problem (activating the virtualenv) could be solved by manually putting the the ENV variables that virtualenv creates when you run the activate script, but that is cumbersome, and i feel a pre/post exec feature would be useful in other cases too.

[slafs]

Hi @mgrandi. This sounds like an interesting idea. Your above solution won't work since honcho exits (i.e. kills all managed processes) whenever one of them exits. Although from my personal experience I really try to avoid the activate script wherever I can, when using tools like Honcho. You could for example have this in your Procfile:

ldb_server: $VIRTUAL_ENV/bin/python3 server.py config.yaml

and control that variable from your .env file (or pass a different one with -e).

Anyway. We could probably use some details about how would you see this to be implemented. As an extra command line arguments perhaps? Then we can also ask @nickstenning and @msabramo opinion on this.

[mgrandi]

OK! I downloaded the source so i'll look at it and see how I would do it.

Also, I use virtualenv for mostly the having specific versions of libraries installed, and that involves setting various ENV variables that let know python know to find the libraries and python binary and whatnot, so its not just the path to the python binary. Thats also what i meant where i guess i could put those ENV variables that the venv creates (in activate.sh / activate.bat) in the .env file, but it would to just call activate/deactivate with said pre/post hooks

[mgrandi]

I was thinking, wouldn't it be easiest to just have 'magic' .env entries that specify a program to run for pre/post exec? so like

HONCHO_PREXEC = python some_pre_exec_program.py some arguments here 200

HONCHO_POSTEXC = ruby some_post_exc_something.rb

[nickstenning]

Sorry, is there a reason you can't write a Procfile that does:

web: source venv/bin/activate && python run.py

Running the activate script as a separate task won't work at all as the processes don't share an environment.

[slafs]

I think @mgrandi wanted an ability to influence all the processes inside his Procfile and leverage different .env files to activate different virtualenvs. But still I guess that can be done by some wrapper script or something like that. Or maybe I'm missing something. @mgrandi do you wanted to achieve something else with this feature? If not, then I'm not sold with this idea yet.

[mgrandi]

Ideally yes, i wanted to have a venv activate itself automatically when i ran honcho, but @nickstenning is correct in that the way honcho works, it won't influence the rest of the processes. However, upon thinking, i dont think there is any easy way for you to run the activate script within python and easily keep the env that it modifies, as when you run something using subprocess, it gets its own copy of the env variables, and if you modify that, it doesn't go back to the parent process.

so basically my original idea of having a pre-exec thing run the activate script won't work no matter how its implemented. However i could still see a use case for running something before all the other processes start.

Or if that is not acceptable, perhaps a config switch to honcho that specifies a delay between process launches? Because as i said in my original post, i have seen (at least once), that honcho launches all the processes pretty much at the same time. I have two processes, two servers, but one is acting like a database server. So if the 'database server' process doesn't finish launching before the other process, which needs to connect to the 'database server', then it errors out because it cant connect to the second process because it hasn't finished launching yet. A race condition of sorts

A delay of X seconds would solve that, and would also hack around the 'pre-exec' thing, where you can just have the first process do some sanity checking that you would normally do in a pre-exec functionality, and if anything is wrong then it quits, which causes honcho to quit as well. You see this sort of thing in the new fancy init systems, which can be configured to launch something after an event , like process started or network has come up, but this would be much simpler.

[nickstenning]

I'm just not sure that the complexity added to honcho for this use case is justifiable when you can achieve the same thing by either writing wrapper scripts or prepending your tasks with source venv/bin/activate &&. In this circumstance I think I would make the scripts you want to run python console_scripts, install them in the virtualenv, and run them directly: venv/bin/myscript.

As for delaying tasks, again I don't see that this is honcho's problem to solve when you can just prepend your tasks with sleep 10 &&.

Honcho is supposed to work with the existing Unix toolchain, not replace each component of it with a highly-coupled and inflexible implementation of its own.

[mgrandi]

thats fine, i get what you are saying =) If i have time i'll see if i can contribute some documentation to show how things like virtualenv's and solving 'race conditions' are solved using the things talked about here!