Support runtest.py to work on Windows when no_pty=True

cy20lin commented 1 year ago

Tweak runtest.py script so that it could now partially run on Windows. The pty part is not implemented, and would raise if --no-pty is not specified.

For the implementation, the threading and queue modules are used to read the subprocess.stdout pipe with timeout, as the select module doesn't work on Windows.

OldLiu001 commented 1 year ago

I briefly tested it and it seemed to add some strange token at the end of each line, like \x16 or something.

This caused both my VBScript implementation and the repository's original PowerShell implementation by Joel Martin to fail the test (although it produced the expected results).

cy20lin commented 1 year ago

Thanks for your feedback. The special character \x16 was introduced at commit 18616b105e6872705ab290eec22fcec3661db700 by @kanaka.

As I have no clear Idea about how the string '\x16\r' work at Runner.writeline on posix, I keep that part intact on posix platform and only modify the code to write the normal str with line_break to the stdin of the subprocess on Windows.

I've tested the powershell implementation by Joel Martin from step0 to step9, and the tests passed smoothly. Except for the step5, you might wants to increase --test-timeout to avoid timeout when testing (def! res2 (sum2 10000 0)).

Regarding stepA, there is an hanging issue when testing (readline "mal-user> "), the same issue is also found when testing on Windows WSL Ubuntu. I'm now fixing that issue and will update if I make further progress.

OldLiu001 commented 1 year ago

Thanks for update.

But when I test step0 with PowerShell impl, got this:

Testing basic string
] -> FAIL (line 3): -> ['',abcABC123
    Expected : 'abcABC123\\\r'
    Got      : 'abcABC123'
Testing string containing spaces
] -> FAIL (line 7):rld\r' -> ['',hello mal world
    Expected : 'hello\\ mal\\ world\\\r'
    Got      : 'hello mal world'
Testing string containing symbols
] -> FAIL (line 11):\r' -> ['',[]{}"'* ;:()
    Expected : '\\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\\r'
    Got      : '[]{}"\'* ;:()'
Test long string
TEST: 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r' -> ['',hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() []{}"'] -> FAIL (line 16):
    Expected : 'hello\\ world\\ abcdefghijklmnopqrstuvwxyz\\ ABCDEFGHIJKLMNOPQRSTUVWXYZ\\ 0123456789\\ \\(;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\)\\\r'
    Got      : 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)'
Non alphanumeric characters
] -> FAIL (line 20):
    Expected : '!\\\r'
    Got      : '!'
] -> FAIL (line 22):
    Expected : '\\&\\\r'
    Got      : '&'
] -> FAIL (line 24):
    Expected : '\\+\\\r'
    Got      : '+'
] -> FAIL (line 26):
    Expected : ',\\\r'
    Got      : ','
] -> FAIL (line 28):
    Expected : '\\-\\\r'
    Got      : '-'
] -> FAIL (line 30):
    Expected : '/\\\r'
    Got      : '/'
] -> FAIL (line 32):
    Expected : '<\\\r'
    Got      : '<'
] -> FAIL (line 34):
    Expected : '=\\\r'
    Got      : '='
] -> FAIL (line 36):
    Expected : '>\\\r'
    Got      : '>'
] -> FAIL (line 38):
    Expected : '\\?\\\r'
    Got      : '?'
] -> FAIL (line 40):
    Expected : '@\\\r'
    Got      : '@'
] -> FAIL (line 43):
    Expected : '\\^\\\r'
    Got      : '^'
] -> FAIL (line 45):
    Expected : '_\\\r'
    Got      : '_'
] -> FAIL (line 47):
    Expected : '`\\\r'
    Got      : '`'
] -> FAIL (line 49):
    Expected : '\\~\\\r'
    Got      : '~'
------- Optional Functionality --------------
------- (Not needed for self-hosting) -------
Non alphanumeric characters
] -> SOFT FAIL (line 58):
    Expected : '\\#\\\r'
    Got      : '#'
] -> SOFT FAIL (line 60):
    Expected : '\\$\\\r'
    Got      : '$'
] -> SOFT FAIL (line 62):
    Expected : '%\\\r'
    Got      : '%'
] -> SOFT FAIL (line 64):
    Expected : '\\.\\\r'
    Got      : '.'
] -> SOFT FAIL (line 66):
    Expected : '\\|\\\r'
    Got      : '|'

FAILURES:
]:> ['',abcABC123 3): abcABC123
    Expected : 'abcABC123\\\r'
    Got      : 'abcABC123'
]:> ['',hello mal worldello mal world
    Expected : 'hello\\ mal\\ world\\\r'
    Got      : 'hello mal world'
]:> ['',[]{}"'* ;:()): []{}"'* ;:()
    Expected : '\\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\\r'
    Got      : '[]{}"\'* ;:()'
FAILED TEST (line 16): hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() [ -> ['',hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() []{}"'* ;:() []{]:'*)
    Expected : 'hello\\ world\\ abcdefghijklmnopqrstuvwxyz\\ ABCDEFGHIJKLMNOPQRSTUVWXYZ\\ 0123456789\\ \\(;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\)\\\r'
    Got      : 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)'
]:> ['',!ST (line 20): !
    Expected : '!\\\r'
    Got      : '!'
]:> ['',&ST (line 22): &
    Expected : '\\&\\\r'
    Got      : '&'
]:> ['',+ST (line 24): +
    Expected : '\\+\\\r'
    Got      : '+'
]:> ['',,ST (line 26): ,
    Expected : ',\\\r'
    Got      : ','
]:> ['',-ST (line 28): -
    Expected : '\\-\\\r'
    Got      : '-'
]:> ['',/ST (line 30): /
    Expected : '/\\\r'
    Got      : '/'
]:> ['',<ST (line 32): <
    Expected : '<\\\r'
    Got      : '<'
]:> ['',=ST (line 34): =
    Expected : '=\\\r'
    Got      : '='
]:> ['',>ST (line 36): >
    Expected : '>\\\r'
    Got      : '>'
]:> ['',?ST (line 38): ?
    Expected : '\\?\\\r'
    Got      : '?'
]:> ['',@ST (line 40): @
    Expected : '@\\\r'
    Got      : '@'
]:> ['',^ST (line 43): ^
    Expected : '\\^\\\r'
    Got      : '^'
]:> ['',_ST (line 45): _
    Expected : '_\\\r'
    Got      : '_'
]:> ['',`ST (line 47): `
    Expected : '`\\\r'
    Got      : '`'
]:> ['',~ST (line 49): ~
    Expected : '\\~\\\r'
    Got      : '~'
]:> ['',#ED TEST (line 58): #
    Expected : '\\#\\\r'
    Got      : '#'
]:> ['',$ED TEST (line 60): $
    Expected : '\\$\\\r'
    Got      : '$'
]:> ['',%ED TEST (line 62): %
    Expected : '%\\\r'
    Got      : '%'
]:> ['',.ED TEST (line 64): .
    Expected : '\\.\\\r'
    Got      : '.'
]:> ['',|ED TEST (line 66): |
    Expected : '\\|\\\r'
    Got      : '|'

TEST RESULTS (for D:\@Resources\编程\@Repositories\mal\impls\tests\step0_repl.mal):
    5: soft failing tests
   19: failing tests
    0: passing tests
   24: total tests

I fix it:

elif os.name == 'nt':
        expects = ["%s%s" % (t.out, re.escape(t.ret))]

with this:

elif os.name == 'nt':
        expects = ["%s%s" % (t.out, re.escape(t.ret)),
                   "%s%s" % (t.out, re.escape(t.ret.strip()))]

Then step0 of powershell & vbs can pass smoothly.

And when I test step1,

With powershell I got these 'FAILURES':

FAILURES:
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : "Exception: expected ')', got EOF"
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : "Exception: expected ']', got EOF"
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : 'Exception: expected \'"\', got EOF'
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : 'Exception: expected \'"\', got EOF'
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : 'Exception: expected \'"\', got EOF'
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : 'Exception: expected \'"\', got EOF'
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : 'Exception: expected \'"\', got EOF'
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : "Exception: expected ')', got EOF"
 -> ['{"a([1-3])" \\1 "a(?!\\1)([1-3])" \\2 "a(?!\\1)(?!\\2)([1-3])" \\3}\r',]:
    Expected : '{"a([1-3])" \\1 "a(?!\\1)([1-3])" \\2 "a(?!\\1)(?!\\2)([1-3])" \\3}\r'
    Got      : '{"a2" 2 "a1" 1 "a3" 3}'

TEST RESULTS (for D:\@Resources\编程\@Repositories\mal\impls\tests\step1_read_print.mal):
    0: soft failing tests
    9: failing tests
  108: passing tests
  117: total tests

With my vbs impl, It stuck in ;; whole line comment (not an exception)\r,

] -> SUCCESS  {:b   {  :cde     3   }  }}\r' -> ['',{:a {:b {:cde 3}}}
] -> SUCCESS1}\r' -> ['',{"1" 1}
] -> SUCCESSr' -> ['',({})
Testing read of comments
TEST: ' ;; whole line comment (not an exception)\r' -> ['',] -> TIMEOUT (line 226)

Exception: TestTimeout('TIMEOUT (line 226)')
Output before exception:
user>

So I test my vbs's step1 with pipe:

mal\impls\vbs> .\run_step1_read_print.cmd < ..\tests\step1_read_print.mal

It did not get stuck and generate the expected result.

user> user> 1
user> user> 7
user> user> 7
user> user> -123
user> user> user> user> user> +
user> user> abc
user> user> abc
user> user> abc5
user> user> abc-def
user> user> user> user> -
user> user> -abc
user> user> ->>
user> user> user> user> (+ 1 2)
user> user> ()
user> user> ()
user> user> (nil)
user> user> ((3 4))
user> user> (+ 1 (+ 2 3))
user> user> (+ 1 (+ 2 3))
user> user> (* 1 2)
user> user> (** 1 2)
user> user> (* -3 6)
user> user> (() ())
user> user> user> user> (1 2 3)
user> user> user> user> user> user> user> user> user> user> nil
user> user> true
user> user> false
user> user> user> user> "abc"
user> user> "abc"
user> user> "abc (with parens)"
user> user> "abc\"def"
user> user> ""
user> user> "\\"
user> user> "\\\\\\\\\\\\\\\\\\"
user> user> "&"
user> user> "'"
user> user> "("
user> user> ")"
user> user> "*"
user> user> "+"
user> user> ","
user> user> "-"
user> user> "/"
user> user> ":"
user> user> ";"
user> user> "<"
user> user> "="
user> user> ">"
user> user> "?"
user> user> "@"
user> user> "["
user> user> "]"
user> user> "^"
user> user> "_"
user> user> "`"
user> user> "{"
user> user> "}"
user> user> "~"
user> user> "!"
user> user> user> user> Exception: unbalanced parentheses.
user> user> Exception: unbalanced parentheses.
user> user> user> user> Exception: unterminated string, got EOF.
user> user> Exception: unterminated string, got EOF.
user> user> Exception: unterminated string, got EOF.
user> user> Exception: unterminated string, got EOF.
user> user> Exception: unbalanced parentheses.
user> user> Exception: unbalanced parentheses.
user> user> user> user> (quote 1)
user> user> (quote (1 2 3))
user> user> (quasiquote 1)
user> user> (quasiquote (1 2 3))
user> user> (unquote 1)
user> user> (unquote (1 2 3))
user> user> (quasiquote (1 (unquote a) 3))
user> user> (splice-unquote (1 2 3))
user> user> user> user> user> :kw
user> user> (:kw1 :kw2 :kw3)
user> user> user> user> [+ 1 2]
user> user> []
user> user> []
user> user> [[3 4]]
user> user> [+ 1 [+ 2 3]]
user> user> [+ 1 [+ 2 3]]
user> user> ([])
user> user> user> user> {}
user> user> {}
user> user> {"abc" 1}
user> user> {"a" {"b" 2}}
user> user> {"a" {"b" {"c" 3}}}
user> user> {"a" {"b" {"cde" 3}}}
user> user> user> user> {"a1" 1 "a2" 2 "a3" 3}
user> user> {:a {:b {:cde 3}}}
user> user> {"1" 1}
user> user> ({})
user> user> user> user> user> 1
user> user> 1
user> user> user> user> (deref a)
user> user> user> user> user> user> user> user> user> (with-meta [1 2 3] {"a" 1})
user> user> user> user> user> user> "\n"
user> user> "#"
user> user> "$"
user> user> "%"
user> user> "."
user> user> "\\"
user> user> "|"
user> user> user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> user> 1
user> user>

And then I manually test all tests in step1_read_print.mal with my vbs impl, It also not get stuck.

Here is my test script (Batch File, placed in same folder with runtest.py):

@echo off
pushd "%~dp0"
goto p
for %%a in (
    step0_repl
    step1_read_print
) do (
    echo @pushd "%%~dp0" ^& @cscript -nologo %%a.vbs > .\impls\vbs\run_%%a.cmd
    python runtest.py --no-pty "%~dp0impls\tests\%%a.mal" "%~dp0impls\vbs\run_%%a.cmd"
    pause
)
exit
:p
for %%a in (
    step0_repl
    step1_read_print
) do (
    echo @pushd "%%~dp0" ^& @powershell ".\%%a.ps1" > .\impls\powershell\run_%%a.cmd
    python runtest.py --no-pty "%~dp0impls\tests\%%a.mal" "%~dp0impls\powershell\run_%%a.cmd"
    pause
)

May I ask what your PowerShell testing environment is? I think there may be slight differences in our environment, which may result in your PowerShell testing passing while mine cannot.

cy20lin commented 1 year ago

I fix the hanging issue when testing on stepA. I test from step0 to stepA again for all my environments and the runtest.py script runs smoomthly.

I have two testing environments, which are:

First
- Windows 11
- Python 3.11.4
- PSVersion 5.1.22621.1778
Second
- Ubuntu 22.04 WSL on Windows 11
- Python 3.10.6
- PSVersion 7.3.5

To test in my first environment, I use the following script.

@echo off
@REM test.cmd
@REM Put this script in the same directory as the runtest.py file

@REM # How to run
@REM # O. Start In powershell shell environment
@REM # 1. Change to Mal project directory 
@REM cd path\to\mal
@REM # 2. Run the test and log the result
@REM .\test.cmd | Tee-Object -FilePath test-nt.log

for %%a in (
    step0_repl
    step1_read_print
    step2_eval
    step3_env
    step4_if_fn_do
    step5_tco
    step6_file
    step7_quote
    step8_macros
    step9_try
    stepA_mal
) do (
    python runtest.py --rundir "impls\powershell" --test-timeout 60 --deferrable --optional --no-pty "..\tests\%%a.mal" powershell ".\%%a.ps1"
)

To test in my second environment, I use the following script.

#!/usr/bin/bash
# test.bash

# Put this script in the same directory as the runtest.py file

# How to run
# # 1. Change to Mal project directory 
# cd path/to/mal
# # 2. Run the test and log the result
# bash test.bash | tee test-posix.log

names=(
    step0_repl
    step1_read_print
    step2_eval
    step3_env
    step4_if_fn_do
    step5_tco
    step6_file
    step7_quote
    step8_macros
    step9_try
    stepA_mal
)

for name in "${names[@]}"
do
    python runtest.py --rundir "impls/powershell" --test-timeout 60 --deferrable --optional --no-pty "../tests/${name}.mal" pwsh "${name}.ps1"
done

One could refer to the attached files for the test log.

cy20lin commented 1 year ago

@OldLiu001 couldn't reproduce your issue on my machine though, hope the infomation above could help you located the problem.

OldLiu001 commented 1 year ago

@OldLiu001 couldn't reproduce your issue on my machine though, hope the infomation above could help you located the problem.

I finally discovered the root cause of the problem.

In Windows, git defaults to changing the line feed encoding of the cloned text file to CRLF, while runtest.py only considers the case where the line feed encoding is LF (i.e. split('\n')).

Resulting in an additional \r at the end of the generated regular expression and leading to test failure.

OldLiu001 commented 4 weeks ago

小哥您好，好久不见，别来无恙？我将您的代码一并合并到了我的分支中，若您仍有兴趣和有空闲，也烦请关注 #624 的相关讨论。

Hello, long time no see, how are you? I have merged your code into my branch, if you are still interested and have spare time, please follow the discussion on #624.

kanaka commented 2 weeks ago

@cy20lin Thank you for your work on this. @OldLiu001 and I figured out a way to use WSL on Windows plus the runtest --no-pty option to enable testing implementations on Windows. It doesn't require significant modification to runtest.py to make it work (and my hesitation with anything that makes runtest.py is that it's already complicated and difficult for me to support as it is). I'm going to close this ticket since there is a workaround that I prefer. Again, thanks for your work on this and sorry for the very high latency in replying.

kanaka / mal

Support runtest.py to work on Windows when no_pty=True #640