Closed cy20lin closed 2 weeks ago
I briefly tested it and it seemed to add some strange token at the end of each line, like \x16
or something.
This caused both my VBScript implementation and the repository's original PowerShell implementation by Joel Martin to fail the test (although it produced the expected results).
Thanks for your feedback. The special character \x16
was introduced at commit 18616b105e6872705ab290eec22fcec3661db700 by @kanaka.
As I have no clear Idea about how the string '\x16\r'
work at Runner.writeline
on posix, I keep that part intact on posix platform and only modify the code to write the normal str
with line_break
to the stdin of the subprocess on Windows.
I've tested the powershell implementation by Joel Martin from step0 to step9, and the tests passed smoothly. Except for the step5, you might wants to increase --test-timeout
to avoid timeout when testing (def! res2 (sum2 10000 0))
.
Regarding stepA, there is an hanging issue when testing (readline "mal-user> ")
, the same issue is also found when testing on Windows WSL Ubuntu. I'm now fixing that issue and will update if I make further progress.
Thanks for update.
But when I test step0 with PowerShell impl, got this:
Testing basic string
] -> FAIL (line 3): -> ['',abcABC123
Expected : 'abcABC123\\\r'
Got : 'abcABC123'
Testing string containing spaces
] -> FAIL (line 7):rld\r' -> ['',hello mal world
Expected : 'hello\\ mal\\ world\\\r'
Got : 'hello mal world'
Testing string containing symbols
] -> FAIL (line 11):\r' -> ['',[]{}"'* ;:()
Expected : '\\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\\r'
Got : '[]{}"\'* ;:()'
Test long string
TEST: 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r' -> ['',hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() []{}"'] -> FAIL (line 16):
Expected : 'hello\\ world\\ abcdefghijklmnopqrstuvwxyz\\ ABCDEFGHIJKLMNOPQRSTUVWXYZ\\ 0123456789\\ \\(;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\)\\\r'
Got : 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)'
Non alphanumeric characters
] -> FAIL (line 20):
Expected : '!\\\r'
Got : '!'
] -> FAIL (line 22):
Expected : '\\&\\\r'
Got : '&'
] -> FAIL (line 24):
Expected : '\\+\\\r'
Got : '+'
] -> FAIL (line 26):
Expected : ',\\\r'
Got : ','
] -> FAIL (line 28):
Expected : '\\-\\\r'
Got : '-'
] -> FAIL (line 30):
Expected : '/\\\r'
Got : '/'
] -> FAIL (line 32):
Expected : '<\\\r'
Got : '<'
] -> FAIL (line 34):
Expected : '=\\\r'
Got : '='
] -> FAIL (line 36):
Expected : '>\\\r'
Got : '>'
] -> FAIL (line 38):
Expected : '\\?\\\r'
Got : '?'
] -> FAIL (line 40):
Expected : '@\\\r'
Got : '@'
] -> FAIL (line 43):
Expected : '\\^\\\r'
Got : '^'
] -> FAIL (line 45):
Expected : '_\\\r'
Got : '_'
] -> FAIL (line 47):
Expected : '`\\\r'
Got : '`'
] -> FAIL (line 49):
Expected : '\\~\\\r'
Got : '~'
------- Optional Functionality --------------
------- (Not needed for self-hosting) -------
Non alphanumeric characters
] -> SOFT FAIL (line 58):
Expected : '\\#\\\r'
Got : '#'
] -> SOFT FAIL (line 60):
Expected : '\\$\\\r'
Got : '$'
] -> SOFT FAIL (line 62):
Expected : '%\\\r'
Got : '%'
] -> SOFT FAIL (line 64):
Expected : '\\.\\\r'
Got : '.'
] -> SOFT FAIL (line 66):
Expected : '\\|\\\r'
Got : '|'
FAILURES:
]:> ['',abcABC123 3): abcABC123
Expected : 'abcABC123\\\r'
Got : 'abcABC123'
]:> ['',hello mal worldello mal world
Expected : 'hello\\ mal\\ world\\\r'
Got : 'hello mal world'
]:> ['',[]{}"'* ;:()): []{}"'* ;:()
Expected : '\\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\\r'
Got : '[]{}"\'* ;:()'
FAILED TEST (line 16): hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() [ -> ['',hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() []{}"'* ;:() []{]:'*)
Expected : 'hello\\ world\\ abcdefghijklmnopqrstuvwxyz\\ ABCDEFGHIJKLMNOPQRSTUVWXYZ\\ 0123456789\\ \\(;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\)\\\r'
Got : 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)'
]:> ['',!ST (line 20): !
Expected : '!\\\r'
Got : '!'
]:> ['',&ST (line 22): &
Expected : '\\&\\\r'
Got : '&'
]:> ['',+ST (line 24): +
Expected : '\\+\\\r'
Got : '+'
]:> ['',,ST (line 26): ,
Expected : ',\\\r'
Got : ','
]:> ['',-ST (line 28): -
Expected : '\\-\\\r'
Got : '-'
]:> ['',/ST (line 30): /
Expected : '/\\\r'
Got : '/'
]:> ['',<ST (line 32): <
Expected : '<\\\r'
Got : '<'
]:> ['',=ST (line 34): =
Expected : '=\\\r'
Got : '='
]:> ['',>ST (line 36): >
Expected : '>\\\r'
Got : '>'
]:> ['',?ST (line 38): ?
Expected : '\\?\\\r'
Got : '?'
]:> ['',@ST (line 40): @
Expected : '@\\\r'
Got : '@'
]:> ['',^ST (line 43): ^
Expected : '\\^\\\r'
Got : '^'
]:> ['',_ST (line 45): _
Expected : '_\\\r'
Got : '_'
]:> ['',`ST (line 47): `
Expected : '`\\\r'
Got : '`'
]:> ['',~ST (line 49): ~
Expected : '\\~\\\r'
Got : '~'
]:> ['',#ED TEST (line 58): #
Expected : '\\#\\\r'
Got : '#'
]:> ['',$ED TEST (line 60): $
Expected : '\\$\\\r'
Got : '$'
]:> ['',%ED TEST (line 62): %
Expected : '%\\\r'
Got : '%'
]:> ['',.ED TEST (line 64): .
Expected : '\\.\\\r'
Got : '.'
]:> ['',|ED TEST (line 66): |
Expected : '\\|\\\r'
Got : '|'
TEST RESULTS (for D:\@Resources\编程\@Repositories\mal\impls\tests\step0_repl.mal):
5: soft failing tests
19: failing tests
0: passing tests
24: total tests
I fix it:
elif os.name == 'nt':
expects = ["%s%s" % (t.out, re.escape(t.ret))]
with this:
elif os.name == 'nt':
expects = ["%s%s" % (t.out, re.escape(t.ret)),
"%s%s" % (t.out, re.escape(t.ret.strip()))]
Then step0 of powershell & vbs can pass smoothly.
And when I test step1,
With powershell I got these 'FAILURES':
FAILURES:
-> ['.*(EOF|end of input|unbalanced).*\r',]:
Expected : '.*(EOF|end of input|unbalanced).*\r'
Got : "Exception: expected ')', got EOF"
-> ['.*(EOF|end of input|unbalanced).*\r',]:
Expected : '.*(EOF|end of input|unbalanced).*\r'
Got : "Exception: expected ']', got EOF"
-> ['.*(EOF|end of input|unbalanced).*\r',]:
Expected : '.*(EOF|end of input|unbalanced).*\r'
Got : 'Exception: expected \'"\', got EOF'
-> ['.*(EOF|end of input|unbalanced).*\r',]:
Expected : '.*(EOF|end of input|unbalanced).*\r'
Got : 'Exception: expected \'"\', got EOF'
-> ['.*(EOF|end of input|unbalanced).*\r',]:
Expected : '.*(EOF|end of input|unbalanced).*\r'
Got : 'Exception: expected \'"\', got EOF'
-> ['.*(EOF|end of input|unbalanced).*\r',]:
Expected : '.*(EOF|end of input|unbalanced).*\r'
Got : 'Exception: expected \'"\', got EOF'
-> ['.*(EOF|end of input|unbalanced).*\r',]:
Expected : '.*(EOF|end of input|unbalanced).*\r'
Got : 'Exception: expected \'"\', got EOF'
-> ['.*(EOF|end of input|unbalanced).*\r',]:
Expected : '.*(EOF|end of input|unbalanced).*\r'
Got : "Exception: expected ')', got EOF"
-> ['{"a([1-3])" \\1 "a(?!\\1)([1-3])" \\2 "a(?!\\1)(?!\\2)([1-3])" \\3}\r',]:
Expected : '{"a([1-3])" \\1 "a(?!\\1)([1-3])" \\2 "a(?!\\1)(?!\\2)([1-3])" \\3}\r'
Got : '{"a2" 2 "a1" 1 "a3" 3}'
TEST RESULTS (for D:\@Resources\编程\@Repositories\mal\impls\tests\step1_read_print.mal):
0: soft failing tests
9: failing tests
108: passing tests
117: total tests
With my vbs impl, It stuck in ;; whole line comment (not an exception)\r
,
] -> SUCCESS {:b { :cde 3 } }}\r' -> ['',{:a {:b {:cde 3}}}
] -> SUCCESS1}\r' -> ['',{"1" 1}
] -> SUCCESSr' -> ['',({})
Testing read of comments
TEST: ' ;; whole line comment (not an exception)\r' -> ['',] -> TIMEOUT (line 226)
Exception: TestTimeout('TIMEOUT (line 226)')
Output before exception:
user>
So I test my vbs's step1 with pipe:
mal\impls\vbs> .\run_step1_read_print.cmd < ..\tests\step1_read_print.mal
It did not get stuck and generate the expected result.
user> user> 1
user> user> 7
user> user> 7
user> user> -123
user> user> user> user> user> +
user> user> abc
user> user> abc
user> user> abc5
user> user> abc-def
user> user> user> user> -
user> user> -abc
user> user> ->>
user> user> user> user> (+ 1 2)
user> user> ()
user> user> ()
user> user> (nil)
user> user> ((3 4))
user> user> (+ 1 (+ 2 3))
user> user> (+ 1 (+ 2 3))
user> user> (* 1 2)
user> user> (** 1 2)
user> user> (* -3 6)
user> user> (() ())
user> user> user> user> (1 2 3)
user> user> user> user> user> user> user> user> user> user> nil
user> user> true
user> user> false
user> user> user> user> "abc"
user> user> "abc"
user> user> "abc (with parens)"
user> user> "abc\"def"
user> user> ""
user> user> "\\"
user> user> "\\\\\\\\\\\\\\\\\\"
user> user> "&"
user> user> "'"
user> user> "("
user> user> ")"
user> user> "*"
user> user> "+"
user> user> ","
user> user> "-"
user> user> "/"
user> user> ":"
user> user> ";"
user> user> "<"
user> user> "="
user> user> ">"
user> user> "?"
user> user> "@"
user> user> "["
user> user> "]"
user> user> "^"
user> user> "_"
user> user> "`"
user> user> "{"
user> user> "}"
user> user> "~"
user> user> "!"
user> user> user> user> Exception: unbalanced parentheses.
user> user> Exception: unbalanced parentheses.
user> user> user> user> Exception: unterminated string, got EOF.
user> user> Exception: unterminated string, got EOF.
user> user> Exception: unterminated string, got EOF.
user> user> Exception: unterminated string, got EOF.
user> user> Exception: unbalanced parentheses.
user> user> Exception: unbalanced parentheses.
user> user> user> user> (quote 1)
user> user> (quote (1 2 3))
user> user> (quasiquote 1)
user> user> (quasiquote (1 2 3))
user> user> (unquote 1)
user> user> (unquote (1 2 3))
user> user> (quasiquote (1 (unquote a) 3))
user> user> (splice-unquote (1 2 3))
user> user> user> user> user> :kw
user> user> (:kw1 :kw2 :kw3)
user> user> user> user> [+ 1 2]
user> user> []
user> user> []
user> user> [[3 4]]
user> user> [+ 1 [+ 2 3]]
user> user> [+ 1 [+ 2 3]]
user> user> ([])
user> user> user> user> {}
user> user> {}
user> user> {"abc" 1}
user> user> {"a" {"b" 2}}
user> user> {"a" {"b" {"c" 3}}}
user> user> {"a" {"b" {"cde" 3}}}
user> user> user> user> {"a1" 1 "a2" 2 "a3" 3}
user> user> {:a {:b {:cde 3}}}
user> user> {"1" 1}
user> user> ({})
user> user> user> user> user> 1
user> user> 1
user> user> user> user> (deref a)
user> user> user> user> user> user> user> user> user> (with-meta [1 2 3] {"a" 1})
user> user> user> user> user> user> "\n"
user> user> "#"
user> user> "$"
user> user> "%"
user> user> "."
user> user> "\\"
user> user> "|"
user> user> user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> user> 1
user> user>
And then I manually test all tests in step1_read_print.mal
with my vbs impl, It also not get stuck.
Here is my test script (Batch File, placed in same folder with runtest.py
):
@echo off
pushd "%~dp0"
goto p
for %%a in (
step0_repl
step1_read_print
) do (
echo @pushd "%%~dp0" ^& @cscript -nologo %%a.vbs > .\impls\vbs\run_%%a.cmd
python runtest.py --no-pty "%~dp0impls\tests\%%a.mal" "%~dp0impls\vbs\run_%%a.cmd"
pause
)
exit
:p
for %%a in (
step0_repl
step1_read_print
) do (
echo @pushd "%%~dp0" ^& @powershell ".\%%a.ps1" > .\impls\powershell\run_%%a.cmd
python runtest.py --no-pty "%~dp0impls\tests\%%a.mal" "%~dp0impls\powershell\run_%%a.cmd"
pause
)
May I ask what your PowerShell testing environment is? I think there may be slight differences in our environment, which may result in your PowerShell testing passing while mine cannot.
I fix the hanging issue when testing on stepA. I test from step0 to stepA again for all my environments and the runtest.py script runs smoomthly.
I have two testing environments, which are:
To test in my first environment, I use the following script.
@echo off
@REM test.cmd
@REM Put this script in the same directory as the runtest.py file
@REM # How to run
@REM # O. Start In powershell shell environment
@REM # 1. Change to Mal project directory
@REM cd path\to\mal
@REM # 2. Run the test and log the result
@REM .\test.cmd | Tee-Object -FilePath test-nt.log
for %%a in (
step0_repl
step1_read_print
step2_eval
step3_env
step4_if_fn_do
step5_tco
step6_file
step7_quote
step8_macros
step9_try
stepA_mal
) do (
python runtest.py --rundir "impls\powershell" --test-timeout 60 --deferrable --optional --no-pty "..\tests\%%a.mal" powershell ".\%%a.ps1"
)
To test in my second environment, I use the following script.
#!/usr/bin/bash
# test.bash
# Put this script in the same directory as the runtest.py file
# How to run
# # 1. Change to Mal project directory
# cd path/to/mal
# # 2. Run the test and log the result
# bash test.bash | tee test-posix.log
names=(
step0_repl
step1_read_print
step2_eval
step3_env
step4_if_fn_do
step5_tco
step6_file
step7_quote
step8_macros
step9_try
stepA_mal
)
for name in "${names[@]}"
do
python runtest.py --rundir "impls/powershell" --test-timeout 60 --deferrable --optional --no-pty "../tests/${name}.mal" pwsh "${name}.ps1"
done
One could refer to the attached files for the test log.
@OldLiu001 couldn't reproduce your issue on my machine though, hope the infomation above could help you located the problem.
@OldLiu001 couldn't reproduce your issue on my machine though, hope the infomation above could help you located the problem.
I finally discovered the root cause of the problem.
In Windows, git defaults to changing the line feed encoding of the cloned text file to CRLF, while runtest.py
only considers the case where the line feed encoding is LF (i.e. split('\n')
).
Resulting in an additional \r
at the end of the generated regular expression and leading to test failure.
小哥您好,好久不见,别来无恙? 我将您的代码一并合并到了我的分支中,若您仍有兴趣和有空闲,也烦请关注 #624 的相关讨论。
Hello, long time no see, how are you? I have merged your code into my branch, if you are still interested and have spare time, please follow the discussion on #624.
@cy20lin Thank you for your work on this. @OldLiu001 and I figured out a way to use WSL on Windows plus the runtest --no-pty
option to enable testing implementations on Windows. It doesn't require significant modification to runtest.py
to make it work (and my hesitation with anything that makes runtest.py
is that it's already complicated and difficult for me to support as it is). I'm going to close this ticket since there is a workaround that I prefer. Again, thanks for your work on this and sorry for the very high latency in replying.
Tweak runtest.py script so that it could now partially run on Windows. The pty part is not implemented, and would raise if --no-pty is not specified.
For the implementation, the threading and queue modules are used to read the subprocess.stdout pipe with timeout, as the select module doesn't work on Windows.