Why captured subprocess `!()` is not evaluated immediately?

[skirkham]

When capturing a sub-process with !(<command>) the attributes of the storage variable appear empty until the variable itself is evaluated on its own. This is better shown by example:

ls
# one  three  two

x = !(ls)
print(x.output)
# 

x
# CommandPipeline(stdin=<_io.BytesIO object at 0x7fbc5889b650>, stdout=<_io.BytesIO object at 0x7fbc5889b5f0>, stderr=<_io.BytesIO object at 0x7fbc5889b170>, pid=16798, returncode=0, args=['ls'], alias=['ls', '--color=auto', '-v'], stdin_redirect=['<stdin>', 'r'], stdout_redirect=[6, 'wb'], stderr_redirect=[8, 'w'], timestamps=[1574779875.0330954, 1574779879.5113409], executed_cmd=['/bin/ls', '--color=auto', '-v'], input=, output=one
# three
# two
# , errors=None)

print(x.output)
# one
# three
# two

For community

⬇️ Please click the 👍 reaction instead of leaving a +1 or 👍 comment

[gforsyth]

Thanks for reporting, @skirkham ! I can reproduce this. This is definitely a regression -- can you post the output of the xonfig command, please? Thanks!

[gforsyth]

Ahh, weird -- so the output of a CommandPipeline object is lazily evaluated -- depending on whether or not the command has "ended" or not.

This is not the expected result:

(base) gil@badcatredux ~ $ x = !(ls)                                                          
(base) gil@badcatredux ~ $ x.ended                                                            
False

but evaluating the variable is leading it to be marked as ended, so then output is correctly teed up.

[skirkham]

Output of xonfig if it's still helpful

+------------------+-----------------+
| xonsh            | 0.9.8           |
| Python           | 3.7.5           |
| PLY              | 3.11            |
| have readline    | True            |
| prompt toolkit   | 1.0.15          |
| shell type       | prompt_toolkit1 |
| pygments         | 2.3.1           |
| on posix         | True            |
| on linux         | True            |
| distro           | ubuntu          |
| on darwin        | False           |
| on windows       | False           |
| on cygwin        | False           |
| on msys2         | False           |
| is superuser     | False           |
| default encoding | utf-8           |
| xonsh encoding   | utf-8           |
| encoding errors  | surrogateescape |
+------------------+-----------------+

[daniel-shimon]

@shahinism as I understand it, this issue is with .output specifically since it's supposed to be lazy evaluated

Could you try the same with .out?

[scopatz]

Yeah, this should be the difference between .out and .output, one of them forces the process to end

[scopatz]

Maybe this distinction is too subtle for public APIs...

[mehow-ves]

Ok so this issue remains unresolved right? Why is !() lazy evaluating the output whereas it doesn't say so anywhere in the docs? Furthremore none of the other subprocess types does the same, all of them evaluate right away and block the program until the command isn't evaluated.

What's worse is that calling .output doesn't actually force the command to end or even start so that attribute is useless unless you force the command to evaluet somehow else.

I found that doing any of the following will force the command to finish (non-exhaustive):

• Reading any of these atrributes: .err, .out, .raw_err, .raw_out, .returncode, .rtn
  • Accessing any other attribute doesn't trigger completion, especially baffling are the .errors and .output attributes
• Calling any of these functions: __str__(), __iter__(), __bool__(), end(), itercheck()

[mehow-ves]

Sure, I might give it a go. Question is what is the outcome that we want? Do we just want to prevent any lazy behaviour or just improve what triggers the execution? I reckon removing the lazines might mess up performance/execution order of some scripts and hence would be a breaking change. However if that would be the case then the script wasn't utilizing the subprocess correctly as they were using a bug rather than a feature.

[anki-code]

improve what triggers the execution

I prefer this way for now. User just need to have what expected by calling .output. Thanks for working on this!