Unexpected new line in CP.out on xonsh2 fork

[Lassulus]

hi, first of all, thanks a lot for these changes to xonsh. I'm currently trying out xonsh and was very disappointed in the default $() behavior, which needs a lot of stripping.

currently I'm trying out some scripts, but a weird thing is happening which I don't really understand. Not sure if its even related to the changes in this fork.

t1 = $(curl -Ss https://p.krebsco.de/029cwqk).out
t2 = $(curl -Ss https://p.krebsco.de/029cwqk)
t3 = t2.out

t1[32760:32790]
output: ': "http:/\n/rss.r/?action=displ'
t3[32760:32800]
output: ' "http://rss.r/?action=display'

why is there an erronous newline in t1? this sadly breaks the json and makes json.loads error with

json.decoder.JSONDecodeError: Invalid control character at: line 922 column 28 (char 32769)

hopefully you can help me figure this out

EDIT: this problem does not appear in normal xonsh with !()

[anki-code]

Hi @Lassulus! Thank you for trying the XEP-2!

why is there an erronous newline in t1?

I can't reproduce this:

# I tried with Python 3.6 and Python 3.9 with the same result, here's 3.9:
docker run --rm -it python:3.9-slim /bin/bash -c 'apt update; apt install -y git curl; /bin/bash'

pip install -U git+https://github.com/anki-code/xonsh2
xonsh2

t1 = $(curl -Ss https://p.krebsco.de/029cwqk).out
t2 = $(curl -Ss https://p.krebsco.de/029cwqk)
t3 = t2.out

t1[32760:32790]
# ': "http://rss.r/?action=displa'
t3[32760:32800]
# ': "http://rss.r/?action=display&bridge=R'

"/\n/" in t1
# False
"/\n/" in t3
# False

import json
j1 = json.loads(t1)
j3 = json.loads(t3)
# no errors

So your output is weird and maybe xonfig output or the OS version could help.

I'm currently trying out xonsh and was very disappointed in the default $() behavior, which needs a lot of stripping.

Completely understand you :) Please star this repository and thanks to you for the likes in #3924 discussion.

I'm currently trying out xonsh

I recommend to read the xonsh-cheatsheet with copy-pastable examples. It will give you minimal basis to understand xonsh.

[Lassulus]

I now tried upgrading the python version to 3.9, pinning the python deps to the same version but the problem still is appearing. I'm building xonsh2 manually by checking out the master of https://github.com/anki-code/xonsh2. maybe there is a problem with that version?

xonfig from xonsh2 reports:

+------------------+-------------------------+
| xonsh2           | xonsh2 fork from 0.9.24 |
| Python           | 3.9.1                   |
| PLY              | 3.11                    |
| have readline    | True                    |
| prompt toolkit   | 3.0.6                   |
| shell type       | prompt_toolkit          |
| pygments         | 2.6.1                   |
| on posix         | True                    |
| on linux         | True                    |
| distro           | unknown                 |
| on darwin        | False                   |
| on windows       | False                   |
| on cygwin        | False                   |
| on msys2         | False                   |
| is superuser     | False                   |
| default encoding | utf-8                   |
| xonsh encoding   | utf-8                   |
| encoding errors  | surrogateescape         |
| on jupyter       | False                   |
| jupyter kernel   | None                    |
| xontrib2         | []                      |
+------------------+-------------------------+

I'm running NixOS 20.09

[anki-code]

Thanks for the details @Lassulus!

I can repeat the issue using this code in addition to the docker example:

for i in range(0, 100): 
    if "/\n/" in ( out := !(curl -Ss https://p.krebsco.de/029cwqk).out): 
        print(out, file=open('out.txt', 'w'))
        print('e', end='') 
        break
    else: 
        print('.', end='') 
    sleep 0.1
# ...............e

cat out.txt | egrep -A 1 '/$'
#            "feed": "https:/
# /www.wiwo.de/contentexport/feed/rss/schlagzeilen"

What I see:

• I can reproduce this on xonsh2
• I've taken a quick look on the xonsh2 code changes and have no obvious ideas
• Sometimes the issue appears once per 10-20-30 requests
• I tested xonsh 0.9.24 and 0.9.26 and the issue does not appears in this versions so this is not upstream problem

Result. I think the issue is really actual for xonsh2 but why it appears randomly and so strange - it's the question without answer now. The xonsh2 fork is just proof of concept for demo and can contain issues.

It's great catch! Many thanks!

[anki-code]

A bit offtop but

I'm currently trying out xonsh and was very disappointed in the default $() behavior, which needs a lot of stripping.

@Lassulus could you please share the examples where you should do a lot of stripping?

[Lassulus]

sure, basically everything with find: find $(ls /nix/store | fzf) find $(nix-build -A findutils '<nixpkgs>')

or this one, I have a bunch of different hosts in /etc/hosts ping $(cat /etc/hosts | cut -d ' ' -f 2 | fzf)

I will add more when I find something

[anki-code]

@Lassulus Because it's context free pure bash commands I can suggest you xontrib-sh and you just add ! at the beginning to run these commands in bash from xonsh. Example:

pip install xontrib-sh
xontrib load sh
! find $(ls /nix/store | fzf)

[anki-code]

@Lassulus I've found shorter workaround (add pipe to cat):

echo $(ls /tmp | fzf | cat)