Access Violation with download_range_to_stream_async

[jamwhy]

At around line 529 in cloud_file.cpp , the following stack occurs with an access violation in free. Sometimes it happens around line 530 so there is some problem that will cause this particular symptom so perhaps a timing issue. This seems to be a async count issue in that the more async requests there are, the easier it is to reproduce. I found 45 (random number when testing) threads to be a problem intermittently though this sample code with 640 makes it very easy to reproduce.

The file size is 31457280 (30 1024 1024) . I left out all the code to get a valid azure::storage::cloud_file _file

#define COUNT 640
#define READ_COUNT 49152

char* the_bufs[COUNT];
std::shared_ptr<concurrency::streams::rawptr_buffer<uint8_t>> the_rawbuf[COUNT];
std::shared_ptr<concurrency::streams::ostream> the_ostream[COUNT];

for (int i = 0; i < COUNT; i++)
{
    the_bufs[i] = new char[READ_COUNT];
    the_rawbuf[i] = std::make_shared<concurrency::streams::rawptr_buffer<uint8_t>>((uint8_t*)the_bufs[i], READ_COUNT, std::ios::out);
    the_ostream[i] = std::make_shared<concurrency::streams::ostream>(the_rawbuf[i]->create_ostream());
}

size_t offset = 0;
pplx::task<void> ignore_task[COUNT];
for (int i = 0; i < COUNT; i++)
{
    std::shared_ptr<concurrency::streams::ostream> l_ostream = the_ostream[i];
    std::shared_ptr<concurrency::streams::rawptr_buffer<uint8_t>> l_rawbuf = the_rawbuf[i];
    ignore_task[i] =
        _file.download_range_to_stream_async(*l_ostream, offset, READ_COUNT).then([l_ostream, l_rawbuf, offset]
    {
        l_ostream->close().wait();
        l_rawbuf->close().wait();

        printf("offset: %d\n", (int)offset);

    });

    offset += READ_COUNT;
}

while (1) {
    Sleep(100);
}

[katmsft]

Hi, are you referring to this line? https://github.com/Azure/azure-storage-cpp/blob/master/Microsoft.WindowsAzure.Storage/src/cloud_file.cpp#L529

[katmsft]

Also, you mentioned the following stack, can you please also share the stack trace?

[jamwhy]

Yes, that is the line, 529 in cloud_file.cpp.

ntdll.dll!00007ffd8f48a1c2()    Unknown
    ntdll.dll!00007ffd8f44e158()    Unknown
    ntdll.dll!00007ffd8f3f4575()    Unknown
    KernelBase.dll!00007ffd8cc3d53b()   Unknown
    ucrtbased.dll!00007ffd46eb2fd1()    Unknown
    ucrtbased.dll!00007ffd46eb1385()    Unknown
    ucrtbased.dll!00007ffd46eb49c5()    Unknown
>   wastoraged.dll!operator delete(void * block) Line 21    C++
    wastoraged.dll!std::_Deallocate(void * _Ptr, unsigned __int64 _Count, unsigned __int64 _Sz) Line 133    C++
    wastoraged.dll!std::allocator<wchar_t>::deallocate(wchar_t * _Ptr, unsigned __int64 _Count) Line 721    C++
    wastoraged.dll!std::_Wrap_alloc<std::allocator<wchar_t> >::deallocate(wchar_t * _Ptr, unsigned __int64 _Count) Line 988 C++
    wastoraged.dll!std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >::_Tidy(bool _Built, unsigned __int64 _Newsize) Line 2260   C++
    wastoraged.dll!std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >::operator=(std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > && _Right) Line 931   C++
    wastoraged.dll!azure::storage::cloud_file_properties::operator=(azure::storage::cloud_file_properties && __that)    C++
    wastoraged.dll!azure::storage::cloud_file::download_single_range_to_stream_async::__l2::<lambda>(const web::http::http_response & response, const azure::storage::request_result & result, azure::storage::operation_context context) Line 529  C++
    [External Code] 
    wastoraged.dll!azure::storage::core::storage_command<void>::preprocess_response(const web::http::http_response & response, const azure::storage::request_result & result, azure::storage::operation_context context) Line 380   C++
    wastoraged.dll!azure::storage::core::executor_impl::execute_async::__l2::Concurrency::task<bool> <lambda>(void)::__l2::<lambda>(Concurrency::task<web::http::http_response> get_headers_task) Line 196  C++
    [External Code] 
    cpprest_2_10d.dll!Concurrency::details::_TaskProcHandle::_RunChoreBridge(void * _Parameter) Line 160    C++
    cpprest_2_10d.dll!Concurrency::details::_DefaultPPLTaskScheduler::_PPLTaskChore::_Callback(void * _Args) Line 51    C++
    [External Code] 
    [Async Call]    
    wastoraged.dll!azure::storage::core::executor_impl::execute_async::__l2::<lambda>() Line 163    C++
    wastoraged.dll!Concurrency::details::_do_while<Concurrency::task<bool> <lambda>(void),bool>(azure::storage::core::executor_impl::execute_async::__l2::Concurrency::task<bool> <lambda>(void) func) Line 36  C++
    wastoraged.dll!azure::storage::core::executor_impl::execute_async(std::shared_ptr<azure::storage::core::storage_command_base> command, const azure::storage::request_options & options, azure::storage::operation_context context) Line 33  C++
    wastoraged.dll!azure::storage::core::executor<void>::execute_async(std::shared_ptr<azure::storage::core::storage_command<void> > command, const azure::storage::request_options & options, azure::storage::operation_context context) Line 597  C++
    wastoraged.dll!azure::storage::cloud_file::download_single_range_to_stream_async(Concurrency::streams::basic_ostream<unsigned char> target, unsigned __int64 offset, unsigned __int64 length, const azure::storage::file_access_condition & condition, const azure::storage::file_request_options & options, azure::storage::operation_context context, bool update_properties, bool validate_last_modify) Line 571 C++
    wastoraged.dll!azure::storage::cloud_file::download_range_to_stream_async(Concurrency::streams::basic_ostream<unsigned char> target, unsigned __int64 offset, unsigned __int64 length, const azure::storage::file_access_condition & condition, const azure::storage::file_request_options & options, azure::storage::operation_context context) Line 748   C++
    wastoresample.exe!azure::storage::cloud_file::download_range_to_stream_async(Concurrency::streams::basic_ostream<unsigned char> target, __int64 offset, __int64 length) Line 4306   C++
    wastoresample.exe!azure::storage::samples::files_getting_started_sample() Line 70   C++

[Jinming-Hu]

@jamwhy This is caused by updating the properties of the same cloud_file_client from multiple threads, this is a race condition.

I suggest you create a client_file_client for each async operation to avoid the race condition.