HTTPS configuration issue, Windows specific.

[VaultBoy13]

Hi,

I started working with azure-storage-cpp and everything was fine until I started testing my project on Azure VM. My problem is that any call to the Azure Storage server takes an unacceptable amount of time to execute. Each server request takes ~100 seconds. After some analysis, I found that the winhttp API, in the background, is trying to connect to different Microsoft servers several times, and as far as I understand, this is somehow related to the validation of the SSL certificates. Important note: My Azure VM has some Internet restrictions, it's allowed to connect only to my Azure Storage Account. So, each http request (by azure-storage-cpp) to Azure Storage server leads winhttp API to connect to these Microsoft IPs (unrelated to Azure Storage server), and hang (SYN SENT socket status) on every call to them because these IPs are blocked by security setting on the VM. Another important note: Microsoft Azure Storage Explorer works fine on my VM, and as far as I understand it doesn't use azure-storage-cpp but uses cpprestSDK directly and probably configures winhttp API correctly.

Code:

#include <iostream>

#include <was/storage_account.h>
#include <was/blob.h>

static const utility::string_t storage_connection_string(U("DefaultEndpointsProtocol=https;AccountName=..."));

int main() {

    azure::storage::cloud_storage_account storage_account = azure::storage::cloud_storage_account::parse(storage_connection_string);

    if (!storage_account.is_initialized()) {
        return -1;
    }

    azure::storage::cloud_blob_client blob_client = storage_account.create_cloud_blob_client();
    azure::storage::cloud_blob_container blob_container = blob_client.get_container_reference(U("container-test"));

    if (blob_container.exists()) {
        std::cout << "container-test exists\n";
    }

    return 0;
}

blob_container.exists() takes ~100 seconds, and then prints "container-test exists"

Hang stack:

ntdll.dll!NtWaitForSingleObject() Unknown KERNELBASE.dll!WaitForSingleObjectEx() Unknown winhttp.dll!HTTP_USER_REQUEST::_HandleSyncPending() Unknown winhttp.dll!HTTP_USER_REQUEST::SendRequest() Unknown winhttp.dll!WinHttpSendRequest() Unknown cryptnet.dll!InetSendAuthenticatedRequestAndReceiveResponse() Unknown cryptnet.dll!InetSendReceiveUrlRequest() Unknown cryptnet.dll!CInetSynchronousRetriever::RetrieveObjectByUrl() Unknown cryptnet.dll!InetRetrieveEncodedObject() Unknown cryptnet.dll!CryptRetrieveObjectByUrlW() Unknown crypt32.dll!ChainRetrieveObjectByUrlW(unsigned short const ,char const ,unsigned long,unsigned long,void ,void ,struct _CRYPT_CREDENTIALS ,void ,struct _CRYPT_RETRIEVE_AUX_INFO ) Unknown crypt32.dll!_SerialChainWireRetrieveDisallowedAutoUpdateCab() Unknown crypt32.dll!_SerialChainGetTimeValidDisallowedAutoUpdateCtl() Unknown crypt32.dll!I_CertGetCertificateSerialChain() Unknown crypt32.dll!CertGetCertificateChain() Unknown webio.dll!WapSslVerifyServerCert() Unknown webio.dll!WapSslVerifyNewServerCert() Unknown webio.dll!WapSslQueryAndVerifySslServerCertNoCommit() Unknown webio.dll!WapSslQueryAndVerifySslConnectionInfo() Unknown webio.dll!WapSslPerformHandshake() Unknown webio.dll!WaSslProcessReceiver() Unknown webio.dll!WapSslReceiveCompletionCallback() Unknown webio.dll!WapTcpReceiveCompletionRoutine() Unknown webio.dll!WapTcpThreadPoolCompletionRoutine() Unknown webio.dll!WapTpIoCompletionRoutine() Unknown KERNELBASE.dll!BasepTpIoCallback() Unknown ntdll.dll!TppIopExecuteCallback() Unknown ntdll.dll!TppWorkerThread() Unknown kernel32.dll!BaseThreadInitThunk() Unknown ntdll.dll!RtlUserThreadStart() Unknown

Attached two screenshots of Process Explorer where winhttp is trying to connect to some unrelated to Azure Storage IPs.

Could you please take a look at this blocking issue?

I see here two possible solutions: First and preferred, azure-storage-cpp has to be fixed to properly configure cpprestSDK or winhttp API to work only with Azure Storage server IP as it does in the Microsoft Azure Storage Explorer. Second, provide a new API that allows us to set cpprestSDK settings.

P.S. Removing Internet security settings from Azure Vm is not a workaround in my organization.

[katmsft]

Thank you for raising the issue and detailed explaining on the issue. We will look into this issue further. Also, Microsoft Azure Storage Explorer is using NodeJS SDK on the back-end, I am not sure why you are certain it uses CPPRest SDK. Although it might be a day 0 issue, can you kindly provide the version of this SDK that you are using? Also, in order to verify the below comments you made, can you test the scenario using HTTP and see if the issue persist?

Microsoft servers several times, and as far as I understand, this is somehow related to the validation of the SSL certificates.

[VaultBoy13]

Thank you for looking into this issue.

Reproduced with version 6.1.0, I have not tried 7.0.0 yet.

Microsoft Azure Storage Explorer is using NodeJS SDK on the back-end, I am not sure why you are certain it uses CPPRest SDK.

This is my wrong conclusion. I am not familiar with all SDKs, so I suggested that CPPRestSDK is used.

What do you mean by

can you test the scenario using HTTP and see if the issue persist?

open Azure Storage Account in some Internet browser?

[Jinming-Hu]

@VaultBoy13 Hi, could you provide below information so that I can further look into it?

In the screenshot you posted, 52.239.141.68 is Azure Storage server IP, which is connected. 93.184.220.29 seems to be certificate verification server(ref), which is blocked. Are you using custom domain with DigiCert SSL certificate? Did you override default endpoint(core.windows.net) when using azure-storage-cpp sdk? And when you tested with Microsoft Azure Storage Explorer, which kind of credential did you use?

Thanks.

[VaultBoy13]

Hi,

I am using the default Azure Windows Server virtual machine with default settings. On this machine, nothing has changed, and I do not change anything in my project or any third-party programs. All by default. I just copy my binary (code above) with dependencies into this VM and run it.

[Jinming-Hu]

Hi, I mean the storage_connection_string, is it something like this DefaultEndpointsProtocol=https;AccountName=YOUR_ACCOUNT_NAME;AccountKey=A_LONG_STRING;EndpointSuffix=core.windows.net? Especially the EndpointSuffix.

And have you configured a custom domain or Azure CDN in your storage account?

[VaultBoy13]

Hi,

I didn’t change the connection string, just copied it from my Storage Account.

Cannot comment regarding a custom domain and Azure CDN, I have to ask IT who created my account. If you can explain me where to look for this information in Storage Account, I can try to check this myself.

[Jinming-Hu]

You can check it on Azure Portal

If some private information not appropriate for public discussion is involved, you can send it to my corporation email(see in my GitHub profile).

[VaultBoy13]

Hi,

I sent the screenshots to your email.

Thanks!

[Jinming-Hu]

@VaultBoy13 I received your email, looks like you aren't using custom domain or Azure CDN.

In that case, this problem is kind of tricky. I need to take more time to investigate, will get back to you later.

[Jinming-Hu]

@VaultBoy13 Hi, I finally figured it out. It's connecting digicert.com to check SSL certificate revocation. Therefore, to some degree, this isn't a performance issue of azure-storage-cpp sdk, it's just the way HTTPS works.

I have three workarounds to bypass this issue.

1. Strongly recommended, add digicert.com IPs to your whitelist.

2. Use HTTP instead of HTTPS. This is not recommended because you lost all benefits of HTTPS.

3. Disable certificate revocation checking of cpprestsdk, this also brings about a little security risk. Currently we don't expose such an API directly and are still evaluating whether or not to add it in future release. You have to modify source code and compile the library by yourself. In here, add this line of code config.set_validate_certificates(false). I tested with v7.0.0 and cpprestsdk 2.10.14, it works.

[VaultBoy13]

Thank you very much!

1. Could you please share a list of these IP addresses? There are many different server lists I’m not sure what exactly I should send to IT.
2. This is about changing the connection string, right? DefaultEndpointsProtocol=http;
3. This can be a workaround in different cases. Please consider to expose this API.

BTW, if Azure Storage Explorer application Azure Portal in Internet Explorer work fine without any issue, it means that these applications (ASE and IE) disable certificates validates, right?

Thanks again.

[Jinming-Hu]

1. I don't have too much expertise in HTTPS stuff either, maybe you can try these links 1, 2.

2. Yes, and also change settings here.

3. We'll consider.

BTW, if Azure Storage Explorer application Azure Portal in Internet Explorer work fine without any issue, it means that these applications (ASE and IE) disable certificates validates, right?

Yes, as far as I've tested, different applications have different settings. For IE, it's enabled by default, you can disable it in options. For PowerShell Invoke-WebRequest, it's disabled by default.