mvxt commented 7 years ago

I've been poking at this for a little while and I'm at my wits' end. I would appreciate any direction/pointers. I'll include the relevant tidbits from files.

I have the following setup for a test suite:

package.luxinc [file w/ package-specific macros]
macro-27.luxinc [file w/ test-case specific macros, INCLUDE package.luxinc]
function-27.lux [main test file, INCLUDE macro-27.luxinc]

function-27:

[shell function-27-REST]
  [invoke setup]
  [invoke function27_rest $var1 $var2 $var3 $var4 $var5 $SOME_URL $SOME_FILE]
  !bash compare.sh ../expectedOutput $SOME_FILE
  ??True
  [invoke function27_rest $var1 $var2 $var3 $var4 $var5 $OTHER_URL $OTHER_FILE]
  !grep '201 Created' $OTHER_FILE
  ??HTTP/1.1 201 Created

macro-27.luxinc:

[macro function27_rest var1 var2 var3 var4 var5 URL FILE]
  """!echo '
  <GIANT XML BODY w/ access to above variables>
  </GIANT XML BODY>
  ' | curl -is -X POST -d @- -u admin:admin -H "Content-type: application/vnd.yang.data+xml" $URL >> $FILE
  """
  ?SH-PROMPT:
  !echo ==$?==
  ?==0==
[endmacro]

package.luxinc:

[macro setup]
  [progress 'Setup']
  !mkdir -p lux_logs/latest_run/output_files
  ?SH-PROMPT:
  !echo ==$?==
  ?==0==
[endmacro]

So, the main test script function-27 first calls setup from the package.luxinc, which just creates a directory. This works fine. The place where I encounter trouble is when the main test script calls function27rest from the macro-27.luxinc file, which echoes a large XML body to a cURL command then pipes the resulting return body into a file.

For whatever reason, when I expect the SH-PROMPT, it never returns. The part that really confuses me is that I have the same setup for multiple tests (i.e., function-28, function-29, etc...) and they all work except this one. The biggest difference I can think of between this one and the rest of the tests is that this one has the largest XML body. But I'm not sure exactly what problem that causes.

The following comments on this issue contain the output from the .stdin and .stdout of the test suite. Any help would be appreciated!!

mvxt commented 7 years ago

function27 stdin:

export PS1=SH-PROMPT:
mkdir -p lux_logs/latest_run/output_files
echo '
  <GIANT XML BODY>
' | curl -is -X POST -d @- -u admin:admin -H "Content-type: application/vnd.yang.data+xml" [REDACTED_URL] >> lux_logs/latest_run/output_files/FUNC-027-DRYRUN-CREATE.tmp

function27 stdout:

[?1034hsh-3.2$ export PS1=SH-PROMPT:
SH-PROMPT:mkdir -p lux_logs/latest_run/output_files
SH-PROMPT:

mvxt commented 7 years ago

function28 stdin:

export PS1=SH-PROMPT:
mkdir -p lux_logs/latest_run/output_files
 echo '
  <GIANT XML BODY>
' | curl -is -X POST -d @- -u admin:admin -H "Content-type: application/vnd.yang.data+xml" [REDACTED_URL] >> lux_logs/latest_run/output_files/FUNC-028-DRYRUN-CREATE.tmp
echo ==$?==
bash compare.sh ../expected/native/FUNC-028 lux_logs/latest_run/output_files/FUNC-028-DRYRUN-CREATE.tmp
 echo '
  <GIANT XML BODY>
' | curl -is -X POST -d @- -u admin:admin -H "Content-type: application/vnd.yang.data+xml" [REDACTED_URL] >> lux_logs/latest_run/output_files/FUNC-028-COMMIT-CREATE.tmp
echo ==$?==

function28 stdout:

[?1034hsh-3.2$ export PS1=SH-PROMPT:
SH-PROMPT:mkdir -p lux_logs/latest_run/output_files
SH-PROMPT: echo '
>   <GIANT XML BODY>
> ' | curl -is -X POST -d @- -u admin:admin -H "Content-type: application/vnd.ya 
ng.data+xml" [REDACTED_URL] >> lux_logs/l 
atest_run/output_files/FUNC-028-DRYRUN-CREATE.tmp
SH-PROMPT:echo ==$?==
==0==
SH-PROMPT:bash compare.sh ../expected/native/FUNC-028 lux_logs/latest_run/output 
_files/FUNC-028-DRYRUN-CREATE.tmp
#### File1: ../expected/native/FUNC-028
#### File2: lux_logs/latest_run/output_files/FUNC-028-DRYRUN-CREATE.tmp
Partial True: ../expected/native/FUNC-028 is a subset of lux_logs/latest_run/output_files/FUNC-028-DRYRUN-CREATE.tmp
SH-PROMPT: echo '
>   <GIANT XML BODY>
> ' | curl -is -X POST -d @- -u admin:admin -H "Content-type: application/vnd.ya 
ng.data+xml" [REDACTED_URL] >> lux_logs/latest_run/outp 
ut_files/FUNC-028-COMMIT-CREATE.tmp
SH-PROMPT:echo ==$?==
==0==
grep '201 Created' lux_logs/latest_run/output_files/FUNC-028-COMMIT-CREATE.tmp
SH-PROMPT:grep '201 Created' lux_logs/latest_run/output_files/FUNC-028-COMMIT-CR 
EATE.tmp
HTTP/1.1 201 Created

As you can clearly see, function28 has the exact same makeup as function27, but is working while function27 is not. Once again, the only difference I can think of is that function27 has a much larger XML body.

Thanks again in advance for your time/help.

hawk commented 7 years ago

Does it work sometimes? If not, perhaps it may be some issue with how long string echo can manage (on your host)?

If it works sometimes I would compare the (textual) event.log file from a successful run with a failing run. More specifically I would try to figure out what happened just before the "echo giant | curl" command. Such as has the previous command REALLY finished before the giant command gets executed? And what differs in the stdout stream when the giant command gets executed??

mvxt commented 7 years ago

Yes, it works sometimes, so that's how I came to the conclusion it was some kind of weird race condition, although I could be wrong.

Unfortunately, looking at the .event.log files doesn't help much.... between the runs that worked and the runs that didn't work, everything is the same up until one of them fails...

Successful run:

func-027-rest(19): progress "'Dry-Run'"
func-027-rest(20): invoke_function27_rest "var1=VAR1 var2=VAR2 var3=VAR3 URL=[REDACTED_URL] FILE=lux_logs/latest_run/output_files/FUNC-027-DRYRUN-CREATE.tmp "
file_enter 20 41 68 "~/test/rest/FUNC-027.lux"
func-027-rest(42): fail pattern "(\\\\[error\\\\])|(.*database is locked)|(Aborted)|([F|f]alse)"
func-027-rest(63): send " echo '     <LARGE XML BODY>     ' | curl -is -X POST -d @- -u admin:admin -H "Content-type: application/vnd.yang.data+xml" [REDACTED_URL] >> lux_logs/latest_run/output_files/FUNC-027-DRYRUN-CREATE.tmp\n"
func-027-rest(64): expect "SH-PROMPT:"
func-027-rest(64): timer start (10 seconds * 1.000)
func-027-rest(64): recv " echo '        <LARGE XML BODY>   ...  "
func-027-rest(64): recv "SH-PROMPT:"
func-027-rest(64): timer cancel (0 seconds)

Unsuccessful run:

func-027-rest(19): progress "'Dry-Run'"
func-027-rest(20): invoke_function27_rest "var1=VAR1 var2=VAR2 var3=VAR3 URL=[REDACTED_URL] FILE=lux_logs/latest_run/output_files/FUNC-027-DRYRUN-CREATE.tmp "
file_enter 20 41 68 "~/test/rest/FUNC-027.lux"
func-027-rest(42): fail pattern "(\\\\[error\\\\])|(.*database is locked)|(Aborted)|([F|f]alse)"
func-027-rest(63): send "echo '     <LARGE XML BODY>     ' | curl -is -X POST -d @- -u admin:admin -H "Content-type: application/vnd.yang.data+xml" [REDACTED_URL] >> lux_logs/latest_run/output_files/FUNC-029-DRYRUN-CREATE.tmp\n"
func-027-rest(64): expect "SH-PROMPT:"
func-027-rest(64): timer start (10 seconds * 1.000)
func-027-rest(64): recv "timeout"
func-027-rest(64): timer fail (10 seconds)
func-027-rest(64): skip ""
func-027-rest(64): stop fail
func-027-rest(64): skip ""

Looking at the above output and comparing it, it looks like the failing test doesn't see the 'echo' output. This probably means the echo command doesn't complete, as it's expecting SH-PROMPT afterwards and never gets it.

I'm a little stumped about what could possibly block an echo command, however.

hawk commented 7 years ago

Check the previous "send" item in the event log and verify that it has completed. With this I mean that you have matched on something reliable, such as the SH-PROMPT produced AFTER the command has finished. Do ensure it was not matched on some earlier prompt.

mvxt commented 7 years ago

Yes, that's correct. It looks like the last successful match was on SH-PROMPT in a previous macro, so it doesn't look like we have an expect/output mismatch. I'll paste in the event.log items.

func-027-rest(11): send "mkdir -p lux_logs/latest_run/output_files\n"
func-027-rest(11): recv "mkdir"
func-027-rest(12): expect "SH-PROMPT:"
func-027-rest(12): timer start (10 seconds * 1.000)
func-027-rest(12): recv " -p lux_logs/latest_run/output_files\r\n"
func-027-rest(12): recv "SH-PROMPT:"
func-027-rest(12): timer cancel (0 seconds)
func-027-rest(12): skip "mkdir -p lux_logs/latest_run/output_files\r\n"
func-027-rest(12): match "SH-PROMPT:"
file_exit 15 68 73 "/Users/mvxt_dd/Documents/Data_Ductus/sprintvpn/packages/sprint-l2vpn/test/rest/FUNC-027.lux"
func-027-rest(16): progress "'Function-027"
func-027-rest(19): progress "'Dry-Run'"
func-027-rest(20): invoke_function27_rest "REDACTED VARIABLES"
file_enter 20 55 97 "/Users/mvxt_dd/Documents/Data_Ductus/sprintvpn/packages/sprint-l2vpn/test/rest/FUNC-029.lux"
func-027-rest(42): fail pattern "(\\\\[error\\\\])|(.*database is locked)|(Aborted)|([F|f]alse)"
func-027-rest(63): send "echo '<LARGE XML BODY>\n' | curl -is -X POST -d @- -u admin:admin -H "Content-type: application/vnd.yang.data+xml" [REDACTED_URL] >> lux_logs/latest_run/output_files/FUNC-027-DRYRUN-CREATE.tmp\n"
func-027-rest(64): expect "SH-PROMPT:"
func-027-rest(64): timer start (10 seconds * 1.000)
func-027-rest(64): recv "timeout"
func-027-rest(64): timer fail (10 seconds)
func-027-rest(64): skip ""
func-027-rest(64): stop fail
func-027-rest(64): skip ""

mvxt commented 7 years ago

Hey Hakan, how does invoking a macro work? Does it just spawn a separate shell to run the macro or does it simply include the commands and run it within the sequence? test:

!some command
?some expect
[invoke some macro]

[macro some macro]
!some other command
?some other expect
[endmacro]

Are the above two separate files essentially the equivalent of:

!some command
?some expect
!some other command
?some other expect

? Or does it actually spawn a separate shell or some separate thread to run the macro?

hawk commented 7 years ago

They are essentially equivalent. Shells are started on demand with [shell NAME]. You can this in the event.log. It is very transparent.

hawk commented 7 years ago

Btw, avoid to build in assumptions about where the script will be executed or where its logs will be located. Use "mkdir -p $LUX_EXTRA_LOGS/output_files" instead of using "mkdir -p lux_logs/latest_run/output_files".

hawk commented 7 years ago

Have you tried to copy the "echo giant |curl" command from the stdin log and paste it repeatedly in a (Bourne) shell? It may cause the same problems as you encounter in the lux script.

mvxt commented 7 years ago

Yes, I ran the command manually on the terminal and it works as expected, repeatedly.

I managed to narrow down the problem to the specific line with the echo function. It looks like macro was not the issue, so I've renamed this thread to reflect that.

Here is what I did:

Copied all of the contents inside the macro to run directly inside the test file
Removed the extra | curl command following the multi-line echo.
Then I ran the debugger to step through line-by-line

Results:

Even without piping into curl and then redirecting output to a file, it looks like the multi-line echo command doesn't return the prompt. What could stop an echo command from returning?
It freezes both when I run the test as well as when I step through each line manually in the Lux debugger
As stated above, I can paste the multi-line XML body directly into the Terminal and echo it without problems. Always consistently returns within 0.35 seconds, so I don't think it's the echo command itself.

Conclusions:

I can only conclude at this point that maybe it's the way echo plays with the multi-line command in Lux. I'll paste in my test script so you can see its contents.

[shell func-027-rest]
  [invoke setup_rest]
  [progress 'FUNC-027 - TEST']

  [progress 'Dry-Run']
  """!echo '
  <GIANT XML BODY>
  ...
  ...
  '
  """
  ?SH-PROMPT:
  !echo ==$?==
  ?==0==

  [progress 'Commit']
  """!echo '
  <GIANT XML BODY>
  ...
  ...
  '
  """
  ?SH-PROMPT:
  !echo ==$?==
  ?==0==

Thanks again for your time and help. Much appreciated

mvxt commented 7 years ago

UPDATE 1: I experimented with the XML body I was sending and shortened it to test, and it looks like the shorter body works (consistently, with repeated running), which explains why the rest of my tests work (shorter XML bodies).

UPDATE 2: Replaced echo with printf and it looks like it still encounters the same issues. Sometimes works, sometimes doesn't... :/

Based on UPDATE 1, is there some kind of buffer limit or buffer overflow or some kind of array or variable size limit being placed on multi-line regex and other items for Lux?

hawk commented 7 years ago

I do not think the problem really is lux. The argument string given to 'echo' is seems quite extreme. So I would avoid it as it may indicate that you have run into some system limit related to the shell (/bin/sh), the terminal or what not. Try using 'cat' or something else which avoids the very long argument string. Something like this may work:

"""!
cat > xml.tmp << END
line1
.
.
line10000
END
[invoke eval "cat xml.tmp | curl"]
"""
[cleanup]
[invoke eval "rm -f xml.tmp"]

hawk commented 7 years ago

[macro eval cmd] !$cmd ?SH-PROMPT: !echo ==$$?== ?==0== ?SH-PROMPT: [endmacro]

[macro eval_any cmd] !$cmd ?SH-PROMPT: !echo ==$$?== ?==[0-9]+== ?SH-PROMPT: [endmacro]

mvxt commented 7 years ago

Gave your suggestions a try and it runs into the same problem. The multi-line command with cat > xml.tmp << HEREDOC HEREDOC runs into the same issue: I ran it a multitude of times, and it succeeded once, and failed the other times.

Every single multi-line command tested (i.e., echo, printf, cat w/ heredoc, even directly passing the entire body as an argument to curl) all worked fine when run directly on the shell, but all commands w/ the multi-line XML hang inside Lux.

I did notice that when I shorten the XML body, it then does work in Lux, and I'm not convinced that that is a system limitation, as you say, because once again, it works when run directly on the terminal.

Thanks for your help thus far, I'll keep digging through and experimenting and I'll report back whether I find anything.

hawk commented 7 years ago

Are you using the same shell as lux when you run it outside lux? See the event.log about how lux starts its shell. (It is probably "/bin/sh -i".)

/Håkan

On Jan 12, 2017 00:10, "Michael Thanh" notifications@github.com wrote:

Gave your suggestions a try and it runs into the same problem. The multi-line command with cat > xml.tmp << HEREDOC HEREDOC runs into the same issue: I ran it a multitude of times, and it succeeded once, and failed the other times.

Every single multi-line command tested (i.e., echo, printf, cat w/ heredoc, even directly passing the entire body as an argument to curl) all worked fine when run directly on the shell, but the test hangs inside Lux.

I did notice that when I shorten the XML body, it then does work in Lux, and I'm not convinced that that is a system limitation, as you say, because once again, it works when run directly on the terminal.

Thanks for your help thus far, I'll keep digging through and experimenting and I'll report back whether I find anything.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hawk/lux/issues/17#issuecomment-272025530, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJpsO5PfUur6DIRBpsku8pX4I7suuAFks5rRWF6gaJpZM4Lf6yC .

mvxt commented 7 years ago

Hmmm. No, my terminal runs zsh. So you think it might be a limitation of sh vs. zsh?

I'll try running the lux tests in a different shell tomorrow.

hawk commented 7 years ago

Maybe. It is one possible difference in the run environment. We have had issues with /bin/sh compliance on some platforms.

Another issue may be the terminal settings. If you cannot reproduce the problem with plain "/bin/sh -i", you may try with /bin/sh in conjunction with lux/priv/bin/runpty like lux does. See the event.log for the shell start details with runpty.

/Håkan

On Thu, Jan 12, 2017 at 5:35 AM, Michael Thanh notifications@github.com wrote:

Hmmm. No, my terminal runs zsh. So you think it might be a limitation of sh vs. zsh?

I'll try running the lux tests in a different shell tomorrow.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hawk/lux/issues/17#issuecomment-272074030, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJpsJ8H8oy1TDJuw_3vZWcFKCsdnqBAks5rRa2dgaJpZM4Lf6yC .

lloydroc commented 7 years ago

Hello Håkan,

I'm working with Michael here. Seems like there is a lux limitation on how many characters can be in the send command. Please see my lux script it seems to fail when the number of characters are around 2048 on a send line. I've gotten the same results using printf and echo instead of cat as shown.

/Lloyd [shell test]

Passes

!cat '0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000' ?SH-PROMPT

Passes

!cat '00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000' ?SH-PROMPT

Will timeout

!cat '00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000' ?SH-PROMPT

mvxt commented 7 years ago

Hey @hawk,

I tried your suggestions to reproduce the error on the shell and was unable to. I first logged into the standard Bourne shell using the command as follows: /usr/local/lux/lib/lux-1.15.1/priv/bin/runpty /bin/sh -i, which is what I pulled out of the Lux .event.log file.

Afterwards, I ran the same large command from inside the Lux script manually in the Bourne shell, and it succeeds without any problems repeatedly.

echo  '
  <LARGE XML BODY>
  ...
  ...
  ...
  </LARGE XML BODY>
' | curl -is -X POST -d @- -u admin:admin -H "Content-type: application/vnd.yang.data+xml" -H "Expect:" "[REDACTED_URL]"

and the output I get is as expected:

HTTP/1.1 201 Created
Server:
Location: [REDACTED_URL]
Date: Thu, 12 Jan 2017 18:40:01 GMT
Last-Modified: Thu, 12 Jan 2017 16:59:20 GMT
Cache-Control: private, no-cache, must-revalidate, proxy-revalidate
Etag: 1484-240360-717269
Content-Length: 1047
Content-Type: text/xml
Vary: Accept-Encoding
Pragma: no-cache
<XML RETURN BODY>

I also tried a few other shells like zsh and bash, and those are also working as expected. At this point my tests and experiments don't reinforce that the issue is with the system or with the specific shell environment the command is being run in.

hawk commented 7 years ago

I can confirm that it is a bug in Lux. It was possible to reproduce with:

/usr/local/lux/lib/lux-1.15.1/priv/bin/runpty /bin/sh -i
echo <HERE I PASTED A BIG CHUNK DATA>

then the shell hanged.

Unfortunately I don't know a good temporary workaround. The bug needs to be fixed.

lloydroc commented 7 years ago

Hey @hawk just curious when we might see the fix come through? Being able to send these sends with > 2048 bytes is a bit critical for our project.

hawk commented 7 years ago

If you are brave you may test the latest on the non-blocking-pty branch. It is not fully tested though.

/Håkan

On Mon, Jan 16, 2017 at 5:09 PM, Lloyd Rochester notifications@github.com wrote:

Hey @hawk https://github.com/hawk just curious when we might see the fix come through? Being able to send these sends with > 2048 bytes is a bit critical for our project.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hawk/lux/issues/17#issuecomment-272902390, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJpsGWOOMXi_MoKFbR8iQAlP_o-olshks5rS5ZPgaJpZM4Lf6yC .

lloydroc commented 7 years ago

@hawk Sure thing we will try it out and let you know.

lloydroc commented 7 years ago

Working on testing it. I looked through the code and the 4*BUF was interesting. The runpty is quite complex and cool as well, was looking at how you did that. Currently get 2 issues always when installing on OS X in that my erlc path is wrong so I edit include.mk, but then get an issue in make saying

lux_html_parse.erl:14: can't find include lib "xmerl/include/xmerl.hrl"

Once I fix that I should be able to get back to you.

mvxt commented 7 years ago

@lloydroc

When I fix my include.mk files to reflect the correct erl and erlc locations, my error for xmerl goes away. Maybe try checking that again?

@hawk Pulled the non-blocking-pty branch and tested it with the tests I was previously having problems with. In the stdin.log, I'm seeing the expected SEND command. However, in the stdout.log, I'm seeing an oddly mangled version of what I sent, and it's also multiple times what the original buffer had. Example:

ORIGINAL COMMAND:

!echo '
  <LARGE XML BODY>
  '

STDOUT OUTPUT:

SH-PROMPT: echo '
  <LARGE XML BODY w/ LAST 2 LINES MISSING>
  ...
  <DUPLICATE LARGE XML BODY w/ FIRST 2 LINES MISSING>
  ...
  <DUPLICATE LARGE XML BODY w/ FIRST 6 LINES MISSING>
  ...etc

If I had to make a guess, it looks like the 4*BUF didn't actually expand the size of the buffer to 4 times, but instead duplicated the limited-size buffer 4 times, resulting in the output you see above.

I tested also with a simple lux script where I echo a large body of 0s like so:

[shell test]
  !echo '000000 ... ' # ... replaces many more 0s for brevity
  ?SH-PROMPT:

and I am getting an error as follows:

result            : FAIL at 3
expected
    SH-PROMPT:
actual error
    The command must be executed in context of a shell

successful        : 0
failed            : 1
    temp.lux:3 - The command must be executed in context of a shell
summary           : FAIL

It also looks like the SH-PROMPT: is still not returning

hawk commented 7 years ago

Can you provide a minimal test case which reproduces the (new) problem?

The error message saying something about shell context, probably implies that the shell has died prematurely. Take a look at the event log. Or run lux with the -v flag.

/Håkan

On Jan 17, 2017 6:58 PM, "Michael Thanh" notifications@github.com wrote:

@lloydroc https://github.com/lloydroc

When I fix my include.mk files to reflect the correct erl and erlc locations, my error for xmerl goes away. Maybe try checking that again?

Pulled the non-blocking-pty branch and tested it with the tests I was previously having problems with. In the stdin.log, I'm seeing the expected SEND command. However, in the stdout.log, I'm seeing an oddly mangled version of what I sent, and it's also multiple times what the original buffer had. Example:

SH-PROMPT: echo ' <LARGE XML BODY w/ LAST 2 LINES MISSING> ... <DUPLICATE LARGE XML BODY w/ FIRST 2 LINES MISSING> ... <DUPLICATE LARGE XML BODY w/ FIRST 6 LINES MISSING>

...etc.

I tested also with a simple lux script where I echo a large body of 0s like so:

[shell test] !echo '000000 .... ' ?SH-PROMPT:

and I am getting an error as follows:

result : FAIL at 3 expected SH-PROMPT: actual error The command must be executed in context of a shell

successful : 0 failed : 1 temp.lux:3 - The command must be executed in context of a shell summary : FAIL

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hawk/lux/issues/17#issuecomment-273247067, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJpsPPpLfWN3LkpirG0vwzcr6GSs2lBks5rTQFCgaJpZM4Lf6yC .

mvxt commented 7 years ago

1) Regarding the early-terminated shell

Yes, you're right. I looked through the event log and it looks like the shell is terminated.

test(3): recv "0"
test(3): recv "Child terminated by signal 1\n"
test(3): recv "shell_exit"

For reference, in the test, I echo a string of 6222 zeroes. It looks like the recv is a bit odd though...

test(3): expect "SH-PROMPT:"
test(3): timer started (10 seconds * 1.000 multiplier)
test(3): recv "echo '"
test(3): recv "00000000000000000000"
test(3): recv "000000000000000000"
test(3): recv "00000000000"
test(3): recv "0000000000000"
test(3): recv "00 \r0000000000000000"
test(3): recv "000000000000000000"
test(3): recv "0000000000000"
test(3): recv "00000000000"
test(3): recv "00000000000"
test(3): recv "000000000"
test(3): recv "00 \r000000000"
test(3): recv "000000000000000"
test(3): recv "000000000000"
test(3): recv "00000000"
test(3): recv "00000000000"
test(3): recv "000000"

This continues for about a thousand more lines because the recv eventually dwindles down to receiving a single 0 at a time.... line 974: test(3): recv "0"

The total number of 0s it receives is only around 3000 or so however before the shell is terminated.

2) Regarding Simple Test Case to Replicate New Problem

[shell test]
    """!echo '00000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000
    0000000000000000000000000000000000000000
    '
    """
    ?SH-PROMPT:

That's the whole test. It seems to die because of the new line characters - if I remove the new lines and have it pasted as a single contiguous line, then it works, but I somehow feel like that defeats the purpose of the multi-line send...

mvxt commented 7 years ago

This test works / passes.

[shell test]
    """!echo '000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
    '
    """
    ?SH-PROMPT:

lloydroc commented 7 years ago

@mvxt I modified my include.mk and it has this.

# Erlang
ERL="/opt/local/bin/erl"
ERLC="/opt/local/bin/erlc"

Erlang I installed from brew.

From shell I have:

╭─lloyd@lloyd-dd.local ~/Documents/workspace/sprintvpn  ‹remotes/origin/Refactor/REST*›
╰─➤  echo $ERL_LIBS                                                                                           1 ↵
/usr/local/lux/lib
╭─lloyd@lloyd-dd.local ~/Documents/workspace/sprintvpn  ‹remotes/origin/Refactor/REST*›
╰─➤  ls /usr/local/lux/lib
compiler-5.0.3.ez et-1.5.ez         lux-1.12.2        lux-1.15.1        sasl-2.4.1.ez     xmerl-1.3.7.ez
erts-6.3.ez       kernel-3.1.ez     lux-1.12.2.ez     lux-1.15.1.ez     stdlib-2.3.ez
╭─lloyd@lloyd-dd.local ~/Documents/workspace/sprintvpn  ‹remotes/origin/Refactor/REST*›
╰─➤

Not sure why it can't find xmerl from /usr/local/lux/lib tried other things but not working.

lloydroc commented 7 years ago

Ok found the issue with installation.

If you try to install lux with ERL_LIBS present it will fail saying it is missing

lux_html_parse.erl:14: can't find include lib "xmerl/include/xmerl.hrl"

Solution is to set the ERL_LIBS to empty, but then you can't run lux or you will get:


ADD PATH FAILED: /usr/local/lux/ebin bad_directory

FATAL ERROR : The mandatory application 'lux' is missing: bad_name
ESCRIPT     : /usr/local/lux/bin/lux.escript
VERSION     : unknown
CWD         : /Users/lloyd/git/lux
ROOT DIR    : /opt/local/lib/erlang
CODE PATH   : [".","/opt/local/lib/erlang/lib/kernel-3.1/ebin",```

So then you go ahead and set your `ERL_LIBS` variable back and all is good with running `lux`.

lloydroc commented 7 years ago

@hawk with the non-blocking-pty I can run my echo that was previously failing.

like this:

!cat '<about 2048 bytes>'
?SH-PROMPT

with success. But when I run this (the cat is line 16):

!cat '<about 4096 bytes>'
?SH-PROMPT

This is the minimal test case I see to reproduce the error. We need to be able to send commands that are around 4096 I would assume maybe even bigger. In the grand scheme this isn't that big.

The failure will be:


result            : FAIL at 16
expected
    SH-PROMPT
actual error
    The command must be executed in context of a shell

successful        : 0
failed            : 1
    blah.lux:16 - The command must be executed in context of a shell
summary           : FAIL

hawk commented 7 years ago

Sorry, I have not have the time to investigate the runpty problem. Hopefully I can do it tomorrow.

Regarding your installation problem, you need an Erlang installation containing the xmerl application. Nowdays Erlang/OTP is split into tons av tiny packages in the eco system of many package managers. I don't know what platform you are using, but on Mac there is a package called 'erlang' which contains the works. "sudo port install erlang" installs what you need.

If you have a symbolic link to lux from some bin directory in your path, you may also get that kind of error. If that is the case you need my recently contributed Erlang/OTP patch of escript.c:

https://github.com/erlang/otp/pull/1293

lloydroc commented 7 years ago

Thanks @hawk just please let us know when you have something we can test again. In the meantime we found a workaround.

hawk commented 7 years ago

I think I have fixed it now.

/Håkan

On Tue, Jan 24, 2017 at 12:37 AM, Lloyd Rochester notifications@github.com wrote:

Thanks @hawk https://github.com/hawk just please let us know when you have something we can test again. In the meantime we found a workaround.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hawk/lux/issues/17#issuecomment-274653429, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJpsKbDrKuqHa9FN9h7cZOn5yKfIgLCks5rVTnEgaJpZM4Lf6yC .

mvxt commented 7 years ago

@hawk Thanks, we will test it as soon as we are available. Same branch, non-blocking-pty?

hawk commented 7 years ago

Yes

/Håkan

On Jan 31, 2017 18:01, "Michael Thanh" notifications@github.com wrote:

@hawk https://github.com/hawk Thanks, we will test it as soon as we are available. Same branch, non-blocking-pty?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hawk/lux/issues/17#issuecomment-276424084, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJpsH_Uu2mBIZlJF6BBxJygTTC7hyIWks5rX2jXgaJpZM4Lf6yC .

lloydroc commented 7 years ago

@hawk I was able to verify the fix worked for me. Looked at the diff and that was a tricky bug. Do have one comment though I will comment on your commit.

hawk / lux

Large send commands (!) not returning prompt in lux #17

Passes

Passes

Will timeout