search

microsoft / cpprestsdk

The C++ REST SDK is a Microsoft project for cloud-based client-server communication in native code using a modern asynchronous C++ API design. This project aims to help C++ developers connect to and interact with services.
Other
7.97k stars 1.65k forks source link

crash on conversions::to_string_t #1142

Open xicor333 opened 5 years ago

xicor333 commented 5 years ago

I'm using the version from vcpkg on visual studio 2019. Every time i call to_string_t, it crashes with a read access violation or some sort of other random exception.

Some times i get 'UTF-8 string has invalid unicode code point'. sometimes i get 'UTF-8 string character can never start with 10xxxxxx'

one thing is for sure though... the code doesn't work for me.

mohamedAlaaK commented 5 years ago

can you provide the string you are trying to convert so that i can try it on linux to see if the problem still exists.

xicor333 commented 5 years ago

It doesn't matter what the string is. i just used "hello world" and it still crashed in the same spot. this one gave me an error saying "UTF-8 string character can never start with 10xxxxxx". It wants to conver from utf8 to utf16, but it seems to have a problem with that.

mohamedAlaaK commented 5 years ago

I tried this line of code on ubunu16.04 and it works fine utility::string_t x = conversions::to_string_t("hello world"); i think you are doing something wrong. could you provide the code you are running.

xicor333 commented 5 years ago

unfortunately, as i posted at the start, i'm pretty sure this is an issue with visual studio 2019 incompatibility or something going wrong when building with it. as for the code, i'm doing this:

std::string m_accessToken = "hello world"; utility::string_t testStr = utility::conversions::to_string_t(m_accessToken);

mcneish1 commented 5 years ago

I'm having this issue too. It looks like strings getting passed into utility::conversions::utf8_to_utf16 are corrupted somehow. windbg shows the input string's size as 0xcccccccc, and that its contents are garbage.

sheryl-leduc commented 5 years ago

I am seeing this issue as well with Visual Studio 2019. Was there a resolution to it?

xicor333 commented 5 years ago

i havent seen one yet. i just gave up on my project until i get notice of a fix.

sheryl-leduc commented 5 years ago

Darn. I think that I will try to compile it using a different IDE and maybe OS. I would really like to use this.

nickyang4github commented 5 years ago

I just met this problem too.

here is the data.

I use to_string_t() to convert std::string to string_t

image

My OS is Window10 and use VS2019.

jason-ha commented 5 years ago

I added these lines to the strings.cpp in utils_test, built with VS2019 and it passes as desired:

        std::string utf8_single_bytes("dir-1.0.2.1");
        auto utf16_from_single_bytes = utility::conversions::to_string_t(utf8_single_bytes);
        VERIFY_ARE_EQUAL(UTF16("dir-1.0.2.1"), utf16_from_single_bytes);

Can you send code sample? Perhaps step into ut8_to_utf16 to confirm the data.

nickyang4github commented 5 years ago

@jason-ha Thanks for the test. Which is the Character set in your VS project, I use Multi-bytes, maybe you use Unicode?

My string is dir-1.0.2.1, so your code is right.

jason-ha commented 5 years ago

cpprestsdk assumes Unicode is in use for Windows, though I don't think it actually uses the conditional TCHAR or non XxxXxxW APIs that should make a difference internally. Windows MBCS is not UTF-8 though both are single byte character based. See https://stackoverflow.com/questions/3298569/difference-between-mbcs-and-utf-8-on-windows for some more discussion. Seems like there is something more going on with string data before it gets to the count_utf8_to_utf16 function for everyone. Please have a look under the debugger at the string when passed to count_utf8_to_utf16, utf8_to_utf16, and to_string_t and confirm that it is unchanged from what was sent in. Also is everyone building their own cpprestsdk library or using a package? What is your compiler version? And what architecture are you targeting? I am building my own cpprestsdk with MSVC version 14.21.27619 targeting x86.

xicor333 commented 5 years ago

I used the build command found on the github (vcpkg). As far as I know i didnt do anything special. Visual Studio was in defaults

mcneish1 commented 5 years ago

cl version 19.16.27027.1, also with vcpkg in 64-bit mode

nickyang4github commented 5 years ago

@jason-ha I built cpprestsdk with vcpkg x64, and use the release library in my project, I try to use the debug library.

Thanks.

nickyang4github commented 5 years ago

I test it again and use cpprest_2_10d.dll in debug mode, everything is fine.

No idea why use release dll have these exceptions.

@xicor333 You can try it.

jason-ha commented 5 years ago

Really think it can be narrowed down by debugging the release build. Watch the local string data at the points I mentioned above. My expectation is that it goes off the rails in there.

sheryl-leduc commented 5 years ago

I am having the same experience. Everything works fine if I use the debug build.

MSVC version 14.21.27619, cpprestsdk with vcpkg x64

jason-ha commented 5 years ago

Has anyone been able to debug in and see what is going wrong? I have not been able to reproduce the issue. Below are my steps. If you haven't been able to debug, but can make a small sample that reproduce the trouble, please share.

(Note using the vcpkg installed per cpprest)

D:\cpprestsdk\vcpkg>vcpkg.exe install cpprestsdk cpprestsdk:x64-windows --triplet x64-windows

The following packages will be built and installed:
    cpprestsdk[brotli,core,default-features,websockets]:x64-windows
Starting package 1/1: cpprestsdk:x64-windows
Building package cpprestsdk[brotli,core,default-features,websockets]:x64-windows...
-- Downloading https://github.com/Microsoft/cpprestsdk/archive/v2.10.13.tar.gz...
-- Extracting source D:/cpprestsdk/vcpkg/downloads/Microsoft-cpprestsdk-v2.10.13.tar.gz
-- Using source at D:/cpprestsdk/vcpkg/buildtrees/cpprestsdk/src/v2.10.13-fe1b687772
-- Configuring x64-windows
-- Building x64-windows-dbg
-- Building x64-windows-rel
-- Installing: D:/cpprestsdk/vcpkg/packages/cpprestsdk_x64-windows/share/cpprestsdk/copyright
-- Performing post-build validation
-- Performing post-build validation done
Building package cpprestsdk[brotli,core,default-features,websockets]:x64-windows... done
Installing package cpprestsdk[brotli,core,default-features,websockets]:x64-windows...
Installing package cpprestsdk[brotli,core,default-features,websockets]:x64-windows... done
Elapsed time for package cpprestsdk:x64-windows: 1.194 min

d:\cpprestsdk\vcpkg>vcpkg integrate project

Created nupkg: d:\cpprestsdk\vcpkg\scripts\buildsystems\vcpkg.d.cpprestsdk.vcpkg.1.0.0.nupkg

With a project open, go to Tools->NuGet Package Manager->Package Manager Console and paste:
    Install-Package vcpkg.d.cpprestsdk.vcpkg -Source "d:\cpprestsdk\vcpkg\scripts\buildsystems"

Start VS 2019. Create new C++ Windows Console project. Installed vcpkg.d.cpprestsdk.vcpkg as listed above. Selected Release x64 config and built with this source:

#include <iostream>
#include <cpprest/json.h>

int main()
{
    std::string utf8_single_bytes("dir-1.0.2.1");
    auto utf16_from_single_bytes = utility::conversions::to_string_t(utf8_single_bytes);
    std::cout << "Hello World (survived conversion and compare)!\n";
}

Debugged in and confirmed conversion was okay.

Further per @NickYang1988 I changed project to use MBCS. But that results in a link error.

ioannis-e commented 5 years ago

A couple of things:

  1. post the Runtime Library setting for debug and release configurations for your project and cpprestsdk library if you build it from source
  2. post the full callstack at the crash to help the readers identify what function is crashing and why
Buggynours commented 5 years ago

The problem is that some cpprest functions have a std::string parameter. Then, to_string_t can be used to convert something to a unicode std::string. The problem is that this class has a different layout in Debug mode and in Release mode. The std::string returned by to_string_t in the calling program can have the wrong format for the called program if they are in different dll's compiled with different flags. I had the same crashes when I was calling cpprest_2_10d.dll, that is apparently compiled this the /MDd flag, from my dll that was compiled with the /MT flag. Recompiling my dll with the /MDd flag was the fix.

This shows that no function should have such parameters.

idofilus commented 4 years ago

Make sure in debug more you use cpprestsdk_x64-windows\debug\lib\cpprest_2_10d.lib with the dll in cpprestsdk_x64-windows\debug\bin\cpprest_2_10d.dll In release cpprestsdk_x64-windows\lib\cpprest_2_10.lib and cpprestsdk_x64-windows\bin\cpprest_2_10.dll

And link the lib:

#ifdef _DEBUG
#pragma comment(lib, "cpprest_2_10d.lib")
#else
#pragma comment(lib, "cpprest_2_10.lib")
#endif