Add ability to use ipdb/pdb

[epicserve]

Is it possible to add the ability to use ipdb/pdb. If I setup a tasks file like this:

from invoke import run, task

@task
def test():
    run("./manage.py test")

And then add this inside a test:

import ipdb; ipdb.set_trace()

I'm not able to run commands and I don't get a ipdb prompt. I was using Invoke 0.10.1 and Python 2.7.9.

[epicserve]

The work around I found is to run tests via python rather than with run function. Some sample code is in my work around for issue #264.

[bitprophet]

I feel like I've run into this myself, tho that might have been in my test runner (spec) - then again since I actually run it via Invoke tasks, it's probably Invoke's fault.

My worry is that ipdb & friends are doing something overly clever which our "hi I'm a pty except not really" approach is tripping up, but we'll have to dig. I can't prioritize this but it's definitely a thing so leaving open. Thanks!

[nkantar]

Definitely want to :+1: this so I can use run('python -m pdb manage.py runserver 0.0.0.0:8000', pty=True).

[bitprophet]

@nkantar assuming you're altoid on freenode who was asking about "always default pty=True" - you can just set a local config file to say run = {'pty': True} (or equiv yaml or json), see http://docs.pyinvoke.org/en/0.11.1/concepts/configuration.html#default-values and the rest of that doc for details.

Re: @epicserve's original request, it's not clear to me why neither of us discussed pty=True unless that was out of band - it's the first thing I would try. Haven't actually tested it w/ ipdb myself tho.

[nkantar]

@bitprophet Thanks, I worked that out — appreciate the help.

---

Regarding the issue at hand, though, pty=True doesn't let me interact with the debugger in the manner @epicserve wants to do in his tests, and it would be helpful. It's well out of my comfort zone, but is (better) support for this on the horizon?

For reference: Python 2.7.6, Invoke 0.11.1, Django 1.6.11, zsh 5.0.2, Ubuntu 14.04

[bitprophet]

Support depends entirely on what exactly is causing ipdb and similar things to not work right. I certainly want that to be possible - in an ideal world, a base-case tasks file with nothing but some run calls should "feel" identical to a shell script doing the same.

Issue is just that this is difficult without sacrificing the 'middleware' features that make this software what it is - capturing output, auto-responding to it, etc. But it's the goal.

The IO stream handling got a big overhaul in the stream-response branch (merging as soon as I confirm it works well for Fabric 2 sudo purposes) so that may either have fixed the problem, or made it worse :D EDIT: looks like it just made it break differently. May poke more.

[epicserve]

Random thought ... what about adding an argument to run that allows a person turn off/skip the middleware?

[bitprophet]

@epicserve In this case, the stdin mirroring isn't even present in master and the version you were using when this ticket got filed...so turning it off is unlikely to help.

Turning off the stdout/stderr capturing, tho, so that all 3 streams are hooked up direct - that's worth testing at least. Especially since "capture the stdout for examination" and "do interactive things" are almost certainly orthogonal most of the time.

May not help (as) much with the fully abstract use case (e.g., sub use cases like Fabric's, where there IS no ability to attach directly to the process' pipes) but still.

[bitprophet]

@epicserve Good news! On a whim I tried out ipdb with aforementioned branch, and it seems to actually work now (again as long as pty=True).

Specifically, some other Python app/script that does a few random things and then calls ipdb.set_trace(); then that script is run indirectly via Invoke inside run - i.e. run("hello-yes-this-is-pdb.py", pty=True).

I confirmed that it does not work under master: the IPDB session starts, and I see a prompt, but upon entering anything at the prompt, things just stop happening, i.e.:

[18:27] bahro:~/Code/oss/invoke {invoke27} [master**]
» inv foo
5
lolcats
--Return--
None
> /Users/jforcier/Code/oss/invoke/run-ipdb.py(7)lol()
      6     print "lolcats"
----> 7     ipdb.set_trace()
      8 

ipdb> l

But in stream-response branch (it's been pushed...), I can interact normally, listing and executing Python commands, etc:

[18:29] bahro:~/Code/oss/invoke {invoke27} [stream-response**]
» inv foo
5
lolcats
--Return--
None
> /Users/jforcier/Code/oss/invoke/run-ipdb.py(7)lol()
      6     print "lolcats"
----> 7     ipdb.set_trace()
      8 

ipdb> l
      2 
      3 
      4 def lol():
      5     print 5
      6     print "lolcats"
----> 7     ipdb.set_trace()
      8 
      9 
     10 if __name__ == '__main__':
     11     lol()

ipdb> print "well that's odd huh"
well that's odd huh
ipdb> 

---

I'm actually not sure why this is - why is a branch that interferes more with stdin & friends better at this than a branch that doesn't? (though, given my experiments with vim in the branch's own ticket, it's still much better at vim now than in master, despite still having some hangups...) That may or may not impact the tentative plans to turn off all middleware behavior, I guess.

[epicserve]

That sounds like progress! Any sort of an estimate on when the stream-response will be merged and made a release?

[bitprophet]

As soon as I get some Fabric 2 sudo support nailed down (working on that today) I'll merge stream-response to master, and I'll put out an Invoke release soon, been sitting on stuff aimed at the Fabric 2 alpha but either I'll get that out soon (really want it public before 2016...) or I'll just do an Invoke release for its own sake.

[epicserve]

@bitprophet I'm personally in no hurry, I was more just curious what your plans were. Thanks for all your work!

[bitprophet]

No worries, those were my actual plans :) I really have been sitting on some stuff in Invoke that I was going to release either way, so.

If you have the ability to nab that branch and try it out sometime soon that would be cool too, I'd hate to close this and then find out it actually still doesn't work in your case!

[epicserve]

I did the following to test:

$ cd Desktop
$ mkdir invoke_tesk && cd invoke_tesk
$ virtualenv -p python3 env
$ source env/bin/activate
$ pip install django ipdb https://github.com/pyinvoke/invoke/archive/stream-response.zip
$ django-admin.py startproject config .
cat > tests.py << EOF
from django.test import TestCase

class GenericTest(TestCase):

    def test_math(self):
        import ipdb; ipdb.set_trace()
        self.assertEqual(1 + 1, 2)

EOF
cat > tasks.py << EOF
from invoke import run, task

@task
def test():
    run("./manage.py test")

EOF

Then for a control test I did the following:

$ ./manage.py test

Which outputs:

Creating test database for alias 'default'...
> /Users/brento/Desktop/invoke_tesk/tests.py(8)test_math()
      6     def test_math(self):
      7         import ipdb; ipdb.set_trace()
----> 8         self.assertEqual(1 + 1, 2)

ipdb> print('test')
test

However when I run I get a inv test, I can type but what I type nothing is output to the screen. But if I type "print('test')" without making a mistake it will return test and another ipdb prompt. Example:

$ inv test
Creating test database for alias 'default'...
> /Users/brento/Desktop/invoke_tesk/tests.py(8)test_math()
      6     def test_math(self):
      7         import ipdb; ipdb.set_trace()
----> 8         self.assertEqual(1 + 1, 2)

ipdb> test

[bitprophet]

You forgot pty=True on your run("./manage.py test")! It's a must-have. ipdb and friends simply won't work well without an actual terminal in place.

[epicserve]

Oopps. After I made that change it worked.

[bitprophet]

[bitprophet]

OK noting this in changelog entry for #289 and closing, thanks for the prompt feedback @epicserve :) sorry this took so bloody long to semi-accidentally fix!

[bitprophet]

Also, merged to master just now, for whatever that's worth. May shit out a release today just to clear the pipes, keep an eye out.

[epicserve]

@bitprophet was "shit" out a release a typo or on purpose? :)

[bitprophet]

Self-deprecating sarcasm :) :hankey:

[epicserve]

lol