mingw 4.5.2 + libiconv + asian character filename = broken unicode support

[GoogleCodeExporter]

Even though the OS (Win XP sp2) supports and displays correctly the 
asian-character filename of the .torrent file libtorrent cannot handle it. I 
tested it with client_test and passed the torrent name but it couldn't "find" 
the file.

Also unicode support is semi-broken. It can handle filenames with greek 
characters just fine(my OS is in greek) but not asian characters and maybe not 
other "exotic" characters from european languages.

Bulding libotorrent with vs2008(and thus not using the libiconv code) works 
fine with asian characters.

Original issue reported on code.google.com by hammered...@gmail.com on 24 Sep 2011 at 10:53

[GoogleCodeExporter]

My understanding is that mingw doesn't support std::wstring, which means the 
wide character file functions in windows cannot be used. I suppose I could 
implement a custom wide string implementation to use, but my impression so far 
is that the use case for mingw specifically is not strong enough. The visual 
studio compiler is free and provides better c++ support.

You could try defining TORRENT_USE_WPATH 1 and see what happens.

When using the narrow string interface, the local code page is used (in your 
case, greek), which means greek characters can be represented while asian 
characters probably cannot be represented.

Original comment by arvid.no...@gmail.com on 25 Sep 2011 at 3:14

[GoogleCodeExporter]

I am not familiar with charsets etc... but how does Windows manage to display 
the asian characters(eg on the desktop) when my local page is greek?

In the end is this a limitation of using libiconv instead of wide strings?

Original comment by hammered...@gmail.com on 25 Sep 2011 at 6:13

[GoogleCodeExporter]

>My understanding is that mingw doesn't support std::wstring, which means the 
wide character file functions in windows cannot be used. I suppose I could 
implement a custom wide string implementation to use, but my impression so far 
is that the use case for mingw specifically is not strong enough. The visual 
studio compiler is free and provides better c++ support.

Maybe utf8-cpp->http://utfcpp.sourceforge.net/ is what you need. It seems to be 
a header-only library and appropriately licensed.

Original comment by hammered...@gmail.com on 25 Sep 2011 at 6:21

[GoogleCodeExporter]

You were right. Defining TORRENT_USE_WPATH 1 makes it work.
Please make the necessary changes to config.hpp because now, when it detects 
that mingw32 is used it defines TORRENT_USE_WPATH 0

Original comment by hammered...@gmail.com on 25 Sep 2011 at 10:17

[GoogleCodeExporter]

> In the end is this a limitation of using libiconv instead of wide strings?

Sort of. The codepage of your system determines how windows interprets narrow 
strings. wide strings can always represent all characters, so it's only a 
problem when converting from a format that also can represent any character (in 
this case utf-8, in the .torrent file) to a representation that is limited 
(narrow string).

> Defining TORRENT_USE_WPATH 1 makes it work.

so, does mingw support wide strings now? I'm pretty sure setting that define 
will rely on std::wstring.

Original comment by arvid.no...@gmail.com on 25 Sep 2011 at 5:44

[GoogleCodeExporter]

>so, does mingw support wide strings now? I'm pretty sure setting that define 
will rely on std::wstring.

Well, I explicitly avoided that by putting a "#define TORRENT_USE_WPATH 1" in 
the of config.hpp(inside the header guard though). This makes libtorrent(as far 
as I can tell) use libiconv for charset conversion but also use the "wide" 
version of the WINAPI calls. Does this make sense?

However if you can come up with sample code that tests std::wstring 
availability/functionality I'll be happy to compile it on 4.5.2 and report back.

Original comment by hammered...@gmail.com on 25 Sep 2011 at 6:47

[GoogleCodeExporter]

I'm assuming this is the 0.15 branch you're building. When I try to build in 
mingw using wide paths, I get this error:

../include/libtorrent/file_storage.hpp:110: error: `libtorrent::fs::wpath' has 
not been declared

I believe this is because boost.filesystem doesn't support wide paths where 
there are no wide strings.
Which version of boost are you using? I'm building with 1.46.0

Original comment by arvid.no...@gmail.com on 25 Sep 2011 at 9:25

[GoogleCodeExporter]

I am using libtorrent 0.15.8(svn trunk) and boost 1.47.0. I am also using 
libiconv 1.13.1(built it myself)

Command line:
Boost: bjam -q --with-system --with-filesystem --with-thread --toolset=gcc 
variant=release link=static runtime-link=shared

Libtorrent: bjam -q --without-python --toolset=gcc variant=release link=static 
runtime-link=shared zlib=system openssl=pe logging=none geoip=static 
dht-support=on boost=source

I am actually testing with qbittorrent now,  since I am the windows 
maintainer(for now using vs2008 though).

Original comment by hammered...@gmail.com on 25 Sep 2011 at 9:31

[GoogleCodeExporter]

well, I don't know what to do about this. my version of mingw still doesn't 
work with wide paths. Do you think it's new in boost-1.47 or in a newer version 
of mingw?

Original comment by arvid.no...@gmail.com on 26 Sep 2011 at 4:04

[GoogleCodeExporter]

Well, I am baffled too. Today I decided to do a simple test. Enable 
TORRENT_USE_WPATH for mingw too in config.hpp. Then in escape_string.cpp I 
commented out the libtorrent version of convert_to_native(). I compiled 
libtorrent and qbt. I confirmed, through the dependency walker, that it doesn't 
link to the libiconv dll and then launch the program. I selected a torrent with 
japanese characters in the filename and it worked flawlessly. This is really 
weird, a few months ago when I first proposed the use of libiconv it didn't. 
Since then I didn't update mingw. 

I am on Windows XP sp2 32bit. Using boost 1.47 and mingw 4.5.2 (from mingw.org 
not from mingw-w64)

So, maybe it is a good idea to reenable TORRENT_USE_WPATH for mingw?

Original comment by hammered...@gmail.com on 29 Sep 2011 at 3:43

[GoogleCodeExporter]

So? I use latest build of mingw and libtorrent and I got the problem with 
convert_to_native and escape_string.cpp. I comment line 167 in config.hpp (&& 
!defined TORRENT_MINGW \) and libtorrent had built successfully

Original comment by Maledict...@gmail.com on 18 Jan 2012 at 6:55