Issues with wxExecuteStdoutStderr or wxDir and none-ascii output

[jjvbsag]

Triggered by my Subversion Plugin I have problems with wxExecuteStdoutStderr, if stdout has utf8 (or more precise, none-ascii) characters within. To verify it, I wrote the following script:

#!/usr/bin/lua
require"wx"

--
-- help functions to construct the error
--
local sbyte=string.byte
local schar=string.char
local sprintf=string.format
local function touch(path)
    local fd=assert(io.open(path,"w"))
    fd:close()
end
local function printf(...)
    io.write(sprintf(...))
    io.flush()
end
local function vis(val)
    if type(val)~="string" then return tostring(val) end
    return '"'..val:gsub("[^\x20-\x7E]",function(c) return sprintf("\\x%02x",sbyte(c))end)..'"'
end

--
-- create some file names
--
touch("ascii-abc.txt")
touch("iso-\xa0\xa8\xb2.txt")
touch("utf-\xc3\xa0\xc3\xa8\xc3\xb2.txt")

--
-- now try
--
local sts,output,errors=wx.wxExecuteStdoutStderr("ls -1")
for _l,line in ipairs(output) do
    printf("output.line=%s\n",vis(line))
end
for _l,line in ipairs(errors) do
    printf("errors.line=%s\n",vis(line))
end

The output I get is:

output.line="ascii-abc.txt"
output.line="iso-"
output.line="est-wx-execute.lua"
output.line="utf-"
output.line="t"

The output of ls | cat on the command line is

ascii-abc.txt
iso-���.txt
test-wx-execute.lua
utf-àèò.txt

So you can see, the output returned in stdout of wxExecuteStdoutStderr has issues.

• lines with none-ascii chars are truncated
• following lines miss characters
• additional lines show up

---

all of this with ZBS-1.80 on xubuntu 18.04.1 LTS

[jjvbsag]

The same is true for wxDir. Change the end of the script above to

--
-- now try
--
local d=wx.wxDir(".")
local ok,entry=d:GetFirst('*',wx.wxDIR_FILES)
while ok do
    printf("FILES entry=%s\n",vis(entry))
    ok,entry=d:GetNext()
end

the output is

FILES entry="test-wx-execute.lua"
FILES entry="iso-\xf4\x80\x82\xa0\xf4\x80\x82\xa8\xf4\x80\x82\xb2.txt"
FILES entry="test-wx-dir.lua"
FILES entry="ascii-abc.txt"
FILES entry="utf-\xc3\xa0\xc3\xa8\xc3\xb2.txt"

but I expected to get

FILES entry="test-wx-execute.lua"
FILES entry="iso-\xa0\xa8\xb2.txt"
FILES entry="test-wx-dir.lua"
FILES entry="ascii-abc.txt"
FILES entry="utf-\xc3\xa0\xc3\xa8\xc3\xb2.txt"

This is an issue of wxWidgets, not lua. Because, when I change the script ending to

--
-- now try
--
require"lfs"
for entry in lfs.dir(".") do
    printf("FILES entry=%s\n",vis(entry))
end

I get the expected:

FILES entry="test-wx-execute.lua"
FILES entry="iso-\xa0\xa8\xb2.txt"
FILES entry="test-wx-dir.lua"
FILES entry="test-lfs-dir.lua"
FILES entry="."
FILES entry="ascii-abc.txt"
FILES entry="utf-\xc3\xa0\xc3\xa8\xc3\xb2.txt"
FILES entry=".."

[pkulchenko]

@jjvbsag, I'm not sure what's going on, but I can't reproduce these results on Windows and everything looks good to me there.

I created a file using Russian unicode characters (привет.lua). This is how it's reported by dir command on my machine:

12/08/2019  22:12             1,431 ascii-stdout.lua
12/08/2019  20:30                11 привет.lua

When I run this script that is using wxDir and lfs, I get the following results:

wxdir=ascii-stdout.lua
wxdir=привет.lua
lfs=ascii-stdout.lua
lfs=??????.lua

Here is the script:

require "wx"
local function getDir(path)
  local dir = wx.wxDir()
  dir:Open(path)
  local found, file = dir:GetFirst("*", wx.wxDIR_FILES)
  while found do
    printf("wxdir=%s\n",file)
    found, file = dir:GetNext()
  end
  dir:Close()
end

getDir('.')

require"lfs"
for entry in lfs.dir(".") do
    printf("lfs=%s\n",entry)
end

As you can see, the results returned by wxDir are the correct ones. I'm not sure if this is somehow related to the OS differences and will try to test on Linux as well, but so far I don't see an issue with the wxwidgets code.

[pkulchenko]

Closing this one, as I don't see what can be done about this one. Please update if there is new information.