Add support for approving strings with Unicode characters

[claremacrae]

The was described by a user, who needed to be able to print wchar.

They were able to add wostream overloads to write out wide-strings

http://www.cplusplus.com/reference/ostream/wostream/

It would be good to build that support in.

[OmerYa]

Thanks Clare. My initial fix was to add two functions to StringMaker class:

#include <codecvt>

static std::string StringMaker::toString(const std::wstring wstr)
{
    static std::wstring_convert<std::codecvt_utf8<wchar_t>, wchar_t> utf8_converter;
    return utf8_converter.to_bytes(wstr);
}
static std::string StringMaker::toString(const wchar_t* wstr)
{
    static std::wstring_convert<std::codecvt_utf8<wchar_t>, wchar_t> utf8_converter;
    return utf8_converter.to_bytes(wstr);
}

Though since I'm not familiar enough with the framework I preferred not to modify it and resorted to overload operator<<.

Another issue is that <codecvt>was deprecated in C++17 so I'm not sure what is the recommended method to convert wchar_t to utf-8 strings.

[claremacrae]

Super - thanks Omer...

This may possible help on the deprecation front...

https://stackoverflow.com/questions/42946335/deprecated-header-codecvt-replacement

[claremacrae]

This is failing to build on both Cygwin and mingw:

https://ci.appveyor.com/project/isidore/approvaltests-cpp/builds/35837411

[claremacrae]

Also several GitHub Actions jobs are failing too: https://github.com/approvals/ApprovalTests.cpp/actions/runs/316162086

(Window with MSVC compiler and both Ubuntu)

[claremacrae]

And GCC and Clang on Ubuntu in Travis: https://travis-ci.org/github/approvals/ApprovalTests.cpp/builds/737181531

[claremacrae]

Adding this fixed my Ubuntu builds - and maybe the other failures too:

#include <locale>

[claremacrae]

The code builds on all platforms now - and there are just two test failures - with Visual Studio using its native compilers:

[claremacrae]

The code we added for this causes the pprovalTests.cpp.StarterProject builds to fail on all GitHub Actions CI builds, where warnings are treated as errors - in the v.10.5.0 release:

https://github.com/approvals/ApprovalTests.cpp.StarterProject/runs/1350196515?check_suite_focus=true

D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\lib\ApprovalTests.v.10.5.0.hpp(5528,42): error C2220: the following warning is treated as an error [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
15
D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\lib\ApprovalTests.v.10.5.0.hpp(5528,42): warning C4996: 'std::codecvt_utf8<wchar_t,1114111,(std::codecvt_mode)0>': warning STL4017: std::wbuffer_convert, std::wstring_convert, and the <codecvt> header (containing std::codecvt_mode, std::codecvt_utf8, std::codecvt_utf16, and std::codecvt_utf8_utf16) are deprecated in C++17. (The std::codecvt class template is NOT deprecated.) The C++ Standard doesn't provide equivalent non-deprecated functionality; consider using MultiByteToWideChar() and WideCharToMultiByte() from <Windows.h> instead. You can define _SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING or _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to acknowledge that you have received this warning. [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
16
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\include\codecvt(34,52): message : see declaration of 'std::codecvt_utf8' [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
17
D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\lib\ApprovalTests.v.10.5.0.hpp(5528,21): warning C4996: 'std::wstring_convert<std::codecvt_utf8<wchar_t,1114111,(std::codecvt_mode)0>,wchar_t,std::allocator<wchar_t>,std::allocator<char>>': warning STL4017: std::wbuffer_convert, std::wstring_convert, and the <codecvt> header (containing std::codecvt_mode, std::codecvt_utf8, std::codecvt_utf16, and std::codecvt_utf8_utf16) are deprecated in C++17. (The std::codecvt class template is NOT deprecated.) The C++ Standard doesn't provide equivalent non-deprecated functionality; consider using MultiByteToWideChar() and WideCharToMultiByte() from <Windows.h> instead. You can define _SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING or _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to acknowledge that you have received this warning. [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
18
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\include\xlocbuf(301,55): message : see declaration of 'std::wstring_convert' [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
19
D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\lib\ApprovalTests.v.10.5.0.hpp(5528,9): warning C4996: 'std::wstring_convert<std::codecvt_utf8<wchar_t,1114111,(std::codecvt_mode)0>,wchar_t,std::allocator<wchar_t>,std::allocator<char>>': warning STL4017: std::wbuffer_convert, std::wstring_convert, and the <codecvt> header (containing std::codecvt_mode, std::codecvt_utf8, std::codecvt_utf16, and std::codecvt_utf8_utf16) are deprecated in C++17. (The std::codecvt class template is NOT deprecated.) The C++ Standard doesn't provide equivalent non-deprecated functionality; consider using MultiByteToWideChar() and WideCharToMultiByte() from <Windows.h> instead. You can define _SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING or _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to acknowledge that you have received this warning. [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
20
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\include\xlocbuf(301,55): message : see declaration of 'std::wstring_convert' [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
21
D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\lib\ApprovalTests.v.10.5.0.hpp(5528,74): warning C4996: 'std::wstring_convert<std::codecvt_utf8<wchar_t,1114111,(std::codecvt_mode)0>,wchar_t,std::allocator<wchar_t>,std::allocator<char>>::wstring_convert': warning STL4017: std::wbuffer_convert, std::wstring_convert, and the <codecvt> header (containing std::codecvt_mode, std::codecvt_utf8, std::codecvt_utf16, and std::codecvt_utf8_utf16) are deprecated in C++17. (The std::codecvt class template is NOT deprecated.) The C++ Standard doesn't provide equivalent non-deprecated functionality; consider using MultiByteToWideChar() and WideCharToMultiByte() from <Windows.h> instead. You can define _SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING or _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to acknowledge that you have received this warning. [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
22
D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\lib\ApprovalTests.v.10.5.0.hpp(5529,30): warning C4996: 'std::wstring_convert<std::codecvt_utf8<wchar_t,1114111,(std::codecvt_mode)0>,wchar_t,std::allocator<wchar_t>,std::allocator<char>>::to_bytes': warning STL4017: std::wbuffer_convert, std::wstring_convert, and the <codecvt> header (containing std::codecvt_mode, std::codecvt_utf8, std::codecvt_utf16, and std::codecvt_utf8_utf16) are deprecated in C++17. (The std::codecvt class template is NOT deprecated.) The C++ Standard doesn't provide equivalent non-deprecated functionality; consider using MultiByteToWideChar() and WideCharToMultiByte() from <Windows.h> instead. You can define _SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING or _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to acknowledge that you have received this warning. [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
23
D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\lib\ApprovalTests.v.10.5.0.hpp(5528,74): warning C4996: 'std::wstring_convert<std::codecvt_utf8<wchar_t,1114111,(std::codecvt_mode)0>,wchar_t,std::allocator<wchar_t>,std::allocator<char>>::~wstring_convert': warning STL4017: std::wbuffer_convert, std::wstring_convert, and the <codecvt> header (containing std::codecvt_mode, std::codecvt_utf8, std::codecvt_utf16, and std::codecvt_utf8_utf16) are deprecated in C++17. (The std::codecvt class template is NOT deprecated.) The C++ Standard doesn't provide equivalent non-deprecated functionality; consider using MultiByteToWideChar() and WideCharToMultiByte() from <Windows.h> instead. You can define _SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING or _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to acknowledge that you have received this warning. [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
24
D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\lib\ApprovalTests.v.10.5.0.hpp(5534,42): warning C4996: 'std::codecvt_utf8<wchar_t,1114111,(std::codecvt_mode)0>': warning STL4017: std::wbuffer_convert, std::wstring_convert, and the <codecvt> header (containing std::codecvt_mode, std::codecvt_utf8, std::codecvt_utf16, and std::codecvt_utf8_utf16) are deprecated in C++17. (The std::codecvt class template is NOT deprecated.) The C++ Standard doesn't provide equivalent non-deprecated functionality; consider using MultiByteToWideChar() and WideCharToMultiByte() from <Windows.h> instead. You can define _SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING or _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to acknowledge that you have received this warning. [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
25
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\include\codecvt(34,52): message : see declaration of 'std::codecvt_utf8' [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
26
D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\lib\ApprovalTests.v.10.5.0.hpp(5534,21): warning C4996: 'std::wstring_convert<std::codecvt_utf8<wchar_t,1114111,(std::codecvt_mode)0>,wchar_t,std::allocator<wchar_t>,std::allocator<char>>': warning STL4017: std::wbuffer_convert, std::wstring_convert, and the <codecvt> header (containing std::codecvt_mode, std::codecvt_utf8, std::codecvt_utf16, and std::codecvt_utf8_utf16) are deprecated in C++17. (The std::codecvt class template is NOT deprecated.) The C++ Standard doesn't provide equivalent non-deprecated functionality; consider using MultiByteToWideChar() and WideCharToMultiByte() from <Windows.h> instead. You can define _SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING or _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to acknowledge that you have received this warning. [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
27
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\include\xlocbuf(301,55): message : see declaration of 'std::wstring_convert' [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
28
D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\lib\ApprovalTests.v.10.5.0.hpp(5534,9): warning C4996: 'std::wstring_convert<std::codecvt_utf8<wchar_t,1114111,(std::codecvt_mode)0>,wchar_t,std::allocator<wchar_t>,std::allocator<char>>': warning STL4017: std::wbuffer_convert, std::wstring_convert, and the <codecvt> header (containing std::codecvt_mode, std::codecvt_utf8, std::codecvt_utf16, and std::codecvt_utf8_utf16) are deprecated in C++17. (The std::codecvt class template is NOT deprecated.) The C++ Standard doesn't provide equivalent non-deprecated functionality; consider using MultiByteToWideChar() and WideCharToMultiByte() from <Windows.h> instead. You can define _SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING or _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to acknowledge that you have received this warning. [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
29
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\include\xlocbuf(301,55): message : see declaration of 'std::wstring_convert' [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
30
D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\lib\ApprovalTests.v.10.5.0.hpp(5534,74): warning C4996: 'std::wstring_convert<std::codecvt_utf8<wchar_t,1114111,(std::codecvt_mode)0>,wchar_t,std::allocator<wchar_t>,std::allocator<char>>::wstring_convert': warning STL4017: std::wbuffer_convert, std::wstring_convert, and the <codecvt> header (containing std::codecvt_mode, std::codecvt_utf8, std::codecvt_utf16, and std::codecvt_utf8_utf16) are deprecated in C++17. (The std::codecvt class template is NOT deprecated.) The C++ Standard doesn't provide equivalent non-deprecated functionality; consider using MultiByteToWideChar() and WideCharToMultiByte() from <Windows.h> instead. You can define _SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING or _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to acknowledge that you have received this warning. [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
31
D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\lib\ApprovalTests.v.10.5.0.hpp(5535,30): warning C4996: 'std::wstring_convert<std::codecvt_utf8<wchar_t,1114111,(std::codecvt_mode)0>,wchar_t,std::allocator<wchar_t>,std::allocator<char>>::to_bytes': warning STL4017: std::wbuffer_convert, std::wstring_convert, and the <codecvt> header (containing std::codecvt_mode, std::codecvt_utf8, std::codecvt_utf16, and std::codecvt_utf8_utf16) are deprecated in C++17. (The std::codecvt class template is NOT deprecated.) The C++ Standard doesn't provide equivalent non-deprecated functionality; consider using MultiByteToWideChar() and WideCharToMultiByte() from <Windows.h> instead. You can define _SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING or _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to acknowledge that you have received this warning. [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]
32
D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\lib\ApprovalTests.v.10.5.0.hpp(5534,74): warning C4996: 'std::wstring_convert<std::codecvt_utf8<wchar_t,1114111,(std::codecvt_mode)0>,wchar_t,std::allocator<wchar_t>,std::allocator<char>>::~wstring_convert': warning STL4017: std::wbuffer_convert, std::wstring_convert, and the <codecvt> header (containing std::codecvt_mode, std::codecvt_utf8, std::codecvt_utf16, and std::codecvt_utf8_utf16) are deprecated in C++17. (The std::codecvt class template is NOT deprecated.) The C++ Standard doesn't provide equivalent non-deprecated functionality; consider using MultiByteToWideChar() and WideCharToMultiByte() from <Windows.h> instead. You can define _SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING or _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to acknowledge that you have received this warning. [D:\a\ApprovalTests.cpp.StarterProject\ApprovalTests.cpp.StarterProject\build_space\tests\StarterProject.tests.vcxproj]

[OmerYa]

It is the deprecation issue of . I've looked for manual wchar_t to UTF-8 conversion options and it is a major headache considering each compiler can interpret wchar_t differently. There seems to be good implementation at: https://github.com/Alexhuszagh/UTFPP But I don't have time to test it (is it possible to use external implementations on ApprovalTests.cpp?).

[claremacrae]

Unfortunately we are going to remove the std::wstring implementations we added, as they use <codecvt> which has been deprecated, and the tests don't pass on Windows.

In case it helps anyone else out, the code was:

namespace ApprovalTests
{
    std::string StringMaker::toString(const std::wstring& wstr)
    {
        static std::wstring_convert<std::codecvt_utf8<wchar_t>, wchar_t> utf8_converter;
        return utf8_converter.to_bytes(wstr);
    }

    std::string StringMaker::toString(const wchar_t* wstr)
    {
        static std::wstring_convert<std::codecvt_utf8<wchar_t>, wchar_t> utf8_converter;
        return utf8_converter.to_bytes(wstr);
    }
}

[claremacrae]

We're closing this, as no further action is intended until C++ properly supports unicode, on a language version that we can use.

In the meantime we recommend using a 3rd-party library if you need to support unicode.