search

Azure / azure-storage-cpplite

Lite version of C++ Client Library for Microsoft Azure Storage
MIT License
25 stars 43 forks source link

How to handle exception with blob_clent_wrapper #114

Open Gaurav-Karu opened 3 years ago

Gaurav-Karu commented 3 years ago

Blob_clent_wrapper return void ....and If I go with blob_clent the volume of download I cannot achieve concurrency....like parallel upload.

Please suggest we need to integrate cpplite as azure-iot-sdk do not have a support for proxy.

Please share some sample code as sample2.cpp doesn't seem to be enough

Jinming-Hu commented 3 years ago

@Gaurav-Karu hi, cpplite is going to be deprecated in favor of Track2 storage SDK.

Gaurav-Karu commented 3 years ago

@Jinming-Hu : Yes I am aware of that as it has been conveyed to us by Swadi.Shraddha@microsoft.com , But Does the Track2 storage is in production ?

Note: we have already given the dates to client ....so what I can do and we cannot experiments until and unless the new SDK is under production ?

we are switching from azure-iot-sdk-c to azure-cpplite because cpplite has a support for proxy

Jinming-Hu commented 3 years ago

blob_client_wrapper doesn't throw exception, you'll need to check errno for error information.

Jinming-Hu commented 3 years ago

and blob_client does have concurrent upload function https://github.com/Azure/azure-storage-cpplite/blob/be490edaf413dc113d8182cbc7a29140d3a63481/include/blob/blob_client.h#L154

Jinming-Hu commented 3 years ago

@Jinming-Hu : Yes I am aware of that as it has been conveyed to us by Swadi.Shraddha@microsoft.com , But Does the Track2 storage is in production ?

Not yet, but it's the latest beta8 is considered RC release. So there won't be too many breaking changes between this version and production release (GA).

Gaurav-Karu commented 3 years ago

@Jinming-Hu ...Is there a way I can recompile cpplite and return errno in another wrapper class which I can use in my code to check the errno....because using assert is dangerous in the production code.

Can u please suggest ?

Jinming-Hu commented 3 years ago

You don't have to use asserts.

Just check errno like this

client.create_container(your_container_name);
int ret = errno;
if (ret == 0) {
    // succeeded
} else if (ret == container_already_exists) {
    // container already exists
}

there's a list of error numbers here

or if you want to wrap code above into a function

int create_container_wrapper(const std::string& container_name) {
    client.create_container(container_name);
    return errno;
}
Gaurav-Karu commented 3 years ago

Hi Thanks for the update...I have one more query if the files are larger size (Let say 800MB) it will tgake some time to upload in blob storage....Now if the user suspend the upload and resume back. (Do such handling is there ?)

Gaurav-Karu commented 3 years ago

why blob_client_wrapper obj is not made accessible directly ? And do we need to use only one object blob_client_wrapper to handle upload, Download and delete or we can create different blob_client_wrapper object for handling ? How to make use of it ?

Jinming-Hu commented 3 years ago

I have one more query if the files are larger size (Let say 800MB) it will tgake some time to upload in blob storage....Now if the user suspend the upload and resume back. (Do such handling is there ?)

Nope, there's no way to suspend or resume for now. But there is a workaround. We have Put Block and Put Block List API. You can split that 800MB data into small chunks, and upload one or a few chunks at a time.

why blob_client_wrapper obj is not made accessible directly ? And do we need to use only one object blob_client_wrapper to handle upload, Download and delete or we can create different blob_client_wrapper object for handling ? How to make use of it ?

blob_client_wrapper is not for public use, it was designed for a specific customer. So the API wasn't well-designed. That's why I 've been suggesting you take a look at our new Track2 SDK. There's already an RC-release. The API interface is quite stable.

You can use single client_wrapper for multiple operations, even perform these operations from multiple threads.

Gaurav-Karu commented 3 years ago

How to get the size of committed and uncommitted block if I am using blob_client_wrapper function upload_file_to_blob

Jinming-Hu commented 3 years ago

@Gaurav-Karu take a look at this API

Gaurav-Karu commented 3 years ago

what is the alternative for this api in cpplite ? ///

/// Returns an enumerable collection of the blob's blocks, using the specified block list filter. /// /// One of the enumeration values that indicates whether to return /// committed blocks, uncommitted blocks, or both. /// An object that represents the access condition for the operation. /// An object that specifies additional options for the request. /// An object that represents the context for the current operation. /// An enumerable collection of objects implementing . std::vector download_block_list(block_listing_filter listing_filter, const access_condition& condition, const blob_request_options& options, operation_context context) const {

Jinming-Hu commented 3 years ago

get_block_list

Gaurav-Karu commented 3 years ago 
    /// <summary>
    /// Uploads a list of blocks to a new or existing blob. 
    /// </summary>
    /// <param name="block_list">An enumerable collection of block IDs, as Base64-encoded strings.</param>
    /// <param name="condition">An <see cref="azure::storage::access_condition" /> object that represents the access condition for the operation.</param>
    /// <param name="options">An <see cref="azure::storage::blob_request_options" /> object that specifies additional options for the request.</param>
    /// <param name="context">An <see cref="azure::storage::operation_context" /> object that represents the context for the current operation.</param>
    void upload_block_list(const std::vector<block_list_item>& block_list, const access_condition& condition, const blob_request_options& options, operation_context context)
    {
        **upload_block_list_async**(block_list, condition, options, context).wait();
    }

Do I need to call upload again or any specific api in cpplite who does similar ?

Jinming-Hu commented 3 years ago

Do I need to call upload again or any specific api in cpplite who does similar ?

Sorry I didn't get what you mean

Gaurav-Karu commented 3 years ago

alternative for the below in cpplite?

void upload_block_list(const std::vector& block_list, const access_condition& condition, const blob_request_options& options, operation_context context) { upload_block_list_async(block_list, condition, options, context).wait(); }

Jinming-Hu commented 3 years ago

yeah, put_block_list

Gaurav-Karu commented 3 years ago

@Jinming-Hu can I use cpprestsdk with azure0cpplite, Because I remember old azure-sdk library supports cpprestsdk , Because cpprestsdk internally uses casablanca

So can I use cpprest sdk feature api like...

Jinming-Hu commented 3 years ago

I think so, you can use both sdk side-by-side

Gaurav-Karu commented 3 years ago

@Jinming-Hu I need a small confirmation from you...I am using the below api for Download and Upload.

  1. Download :-> using api of blob client wrapper (download_blob_to_file)...This seems to be working fine but downloading almost 1800 files the Gateway become so slow. Does it because I am creating a object of blob client as global object to use throught a file.
  2. Upload:-> For upload I am using the blob_client api upload_block_blob_from_buffer because to achieve suspend and resume feature.... but when we tested it allows only small files to upload ...Larger files more than 50 MB is failing.

Note: Please suggest some points to improve the performance

Gaurav-Karu commented 3 years ago

@Jinming-Hu Observed crashed when tried to download 8000 Package ...after downloading 1000 Package the Gateway became slow and observed multiple crashes.....I have used void blob_client_wrapper::download_blob_to_file(const std::string &container, const std::string &blob, const std::string &destPath, time_t &returned_last_modified, size_t parallel)

BT TRACE

0 0x00007fb9ae8e0ad9 in tinyxml2::XMLNode::FirstChildElement(char const*) const () from /home/gwa/GWA/lib/libazure-storage-lite.so

1 0x00007fb9ae8e7cc5 in azure::storage_lite::tinyxml2_parser::parse_text(tinyxml2::XMLElement*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const () from /home/gwa/GWA/lib/libazure-storage-lite.so

2 0x00007fb9ae8ea5ec in azure::storage_lite::tinyxml2_parser::parse_storage_error(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const () from /home/gwa/GWA/lib/libazure-storage-lite.so

3 0x00007fb9ae9483db in azure::storage_lite::async_executor::submit_helper(std::shared_ptr<std::promise<azure::storage_lite::storage_outcome > >, std::shared_ptr<azure::storage_lite::storage_outcome >, std::shared_ptr, std::shared_ptr, std::shared_ptr, std::shared_ptr, std::shared_ptr)::{lambda(int, azure::storage_lite::storage_istream, CURLcode)#1}::operator()(int, azure::storage_lite::storage_istream, CURLcode) const () from /home/gwa/GWA/lib/libazure-storage-lite.so

4 0x00007fb9ae948a0d in std::_Function_handler<void (int, azure::storage_lite::storage_istream, CURLcode), azure::storage_lite::async_executor::submit_helper(std::shared_ptr<std::promise<azure::storage_lite::storage_outcome > >, std::shared_ptr<azure::storage_lite::storage_outcome >, std::shared_ptr, std::shared_ptr, std::shared_ptr, std::shared_ptr, std::shared_ptr)::{lambda(int, azure::storage_lite::storage_istream, CURLcode)#1}>::_M_invoke(std::_Any_data const&, int&&, azure::storage_lite::storage_istream&&, CURLcode&&) ()

from /home/gwa/GWA/lib/libazure-storage-lite.so

5 0x00007fb9ae92faf9 in azure::storage_lite::CurlEasyRequest::submit(std::function<void (int, azure::storage_lite::storage_istream, CURLcode)>, std::chrono::duration<long, std::ratio<1l, 1l> >) () from /home/gwa/GWA/lib/libazure-storage-lite.so

6 0x00007fb9ae944f93 in azure::storage_lite::async_executor::submit_helper(std::shared_ptr<std::promise<azure::storage_lite::storage_outcome > >, std::shared_ptr<azure::storage_lite::storage_outcome >, std::shared_ptr, std::shared_ptr, std::shared_ptr, std::shared_ptr, std::shared_ptr) () from /home/gwa/GWA/lib/libazure-storage-lite.so

7 0x00007fb9ae94d2be in azure::storage_lite::async_executor::submit(std::shared_ptr, std::shared_ptr, std::shared_ptr, std::shared_ptr)

() from /home/gwa/GWA/lib/libazure-storage-lite.so

8 0x00007fb9ae9376d1 in azure::storage_lite::blob_client::get_chunk_to_stream_sync(std::cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::cxx11::basic_string<char, std::char_traits, std::allocator > const&, unsigned long long, unsigned long long, std::ostream&) () from /home/gwa/GWA/lib/libazure-storage-lite.so

9 0x00007fb9ae953660 in azure::storage_lite::blob_client_wrapper::download_blob_to_file(std::cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, long&, unsigned long) () from /home/gwa/GWA/lib/libazure-storage-lite.so

10 0x00007fb9b328687c in ftc::ftcLiteDownload::downloadBlocksFromBlob (this=this@entry=0x7fb98d45c470, str_msgid=...)

at //home/gwauser/opt/jenkins/jobs/GWA-test/258/LinuxEnv/RSPBackendConnectivity/FileTransfer/src/FTCLiteDownload.cpp:181

11 0x00007fb9b3287206 in ftc::ftcLiteDownload::downloadBlockBlob (this=, lBlockSize=, primarysas=..., sas_token=...,