Failing blob upload in azs_flush with input path /test.txt because of an error from upload_file_to_blob(). Errno = 35.

DerekRenew commented 8 months ago

Which version of blobfuse was used?

1.3.6

Which OS distribution and version are you using?

Red Hat Enterprise Linux Server release 7.9 (Maipo)

If relevant, please share your mount command.

blobfuse \ --tmp-path= \ --config-file= \ --log-level=LOG_DEBUG \ -o allow_other -o attr_timeout=240 -o entry_timeout=240

What was the issue encountered?

Can list and read a file. Unable to upload a file.

Have you found a mitigation/solution?

No

Please share logs if available.

Command used after the container is mounted: copy large2.txt large3.txt System Logs: Feb 9 15:10:12 gdclappn0017 blobfuse[42923]: Successfully downloaded blob large2.txt into file cache as /path-to-file/large2.txt. Feb 9 15:10:12 gdclappn0017 blobfuse[42923]: Successfully created file /large3.txt in file cache. Feb 9 15:10:12 gdclappn0017 blobfuse[42923]: ==> REQUEST/RESPONSE :: PUT https://REDACTED?sv=2022-11-02&ss=bfqt&srt=sco&sp=rwdlacupyx&se=2025-10-26T13:08:47Z&st=2023-10-26T05:08:47Z&spr=https&sig=REDACTED?&Content-Length=35&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.3.6&x-ms-date=Fri, 09 Feb 2024 04:10:12 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 200 :: Omitted Feb 9 15:17:26 gdclappn0017 blobfuse[42923]: ==> REQUEST/RESPONSE :: PUT https://REDACTED?sv=2022-11-02&ss=bfqt&srt=sco&sp=rwdlacupyx&se=2025-10-26T13:08:47Z&st=2023-10-26T05:08:47Z&spr=https&sig=REDACTED?&Content-Length=35&x-ms-blob-type=BlockBlob&User-Agent=azure-storage-fuse/1.3.6&x-ms-date=Fri, 09 Feb 2024 04:16:13 GMT&x-ms-version=2018-11-09&Transfer-Encoding=--------------------------------------------------------------------------------RESPONSE Status :: 200 :: Feb 9 15:17:26 gdclappn0017 blobfuse[42923]: Failing blob upload in azs_flush with input path /large3.txt because of an error from upload_file_to_blob(). Errno = 35. Feb 9 15:17:26 gdclappn0017 blobfuse[42923]: Failed to map storage error code 35 to a proper errno. Returning EIO = 5 instead.

DerekRenew commented 8 months ago

Where the error was logged: https://github.com/Azure/azure-storage-fuse/blob/971280911b0c39821ceaefc779208f56707113e8/blobfuse/fileapis.cpp#L344

    errno = 0;
    std::vector<std::pair<std::string, std::string>> metadata;
    storage_client->UpdateBlobProperty(blob_name, "", "", &metadata);
    storage_client->UploadFromFile(mntPath, metadata);
    if (errno != 0)
    {
        int storage_errno = errno;
        syslog(LOG_ERR, "Failing blob upload in azs_flush with input path %s because of an error from upload_file_to_blob().  Errno = %d.\n", path, storage_errno);
        free(path_buffer);
        return 0 - map_errno(storage_errno);
    }

It shoud be the below function invoked:

    void BlockBlobBfsClient::UploadFromFile(const std::string sourcePath, METADATA &metadata)
    {
        std::string blobName = sourcePath.substr(configurations.tmpPath.size() + 6 /* there are six characters in "/root/" */);
        m_blob_client->upload_file_to_blob(sourcePath, configurations.containerName, blobName, metadata);
        // upload_file_to_blob does not return a status or success if the blob succeeded
        // it does syslog if there was an exception and changes the errno.
    }

This furhter brings us to: https://github.com/Azure/azure-storage-fuse/blob/971280911b0c39821ceaefc779208f56707113e8/cpplite/src/blob/blob_client_wrapper.cpp

    void blob_client_wrapper::upload_file_to_blob(const std::string &sourcePath, const std::string &container, const std::string blob, const std::vector<std::pair<std::string, std::string>> &metadata, size_t parallel)
    {
        ...omitted...

        if(fileSize <= 64*1024*1024)
        {
            put_blob(sourcePath, container, blob, metadata);
            // put_blob sets errno
    return;
        }

Then to:

    void blob_client_wrapper::put_blob(const std::string &sourcePath, const std::string &container, const std::string blob, const std::vector<std::pair<std::string, std::string>> &metadata)
    {
        ...omitted...

        int error_code = 0;
        try
        {
            auto task = m_blobClient->upload_block_blob_from_stream(container, blob, ifs, metadata);
            auto result = task.get();
            if(!result.success())
            {
                error_code = std::stoi(result.error().code);
            }
        }
        catch(std::exception& ex)
        {
            logger::log(log_level::error, "Failure to upload the blob in put_blob.  ex.what() = %s, container = %s, blob = %s, sourcePath = %s.", ex.what(), container.c_str(), blob.c_str(), sourcePath.c_str());
            error_code = unknown_error;
        }

This should be the source of errno 35: https://github.com/Azure/azure-storage-fuse/blob/971280911b0c39821ceaefc779208f56707113e8/cpplite/src/blob/blob_client.cpp#L187

    std::future<storage_outcome<void>> blob_client::upload_block_blob_from_stream(const std::string &container, const std::string &blob, std::istream &is, const std::vector<std::pair<std::string, std::string>> &metadata)
    {
        auto http = m_client->get_handle();

        auto request = std::make_shared<create_block_blob_request>(container, blob);

        auto cur = is.tellg();
        is.seekg(0, std::ios_base::end);
        auto end = is.tellg();
        is.seekg(cur);
        request->set_content_length(static_cast<unsigned int>(end - cur));
        if (metadata.size() > 0)
        {
            request->set_metadata(metadata);
        }

        http->set_input_stream(storage_istream(is));

        return async_executor<void>::submit(m_account, request, http, m_context);
    }

vibhansa-msft commented 8 months ago

Kindly upgrade to Blobfuse2 and verify the issue persist. All active development in Blobfuse 1.x has stopped.

DerekRenew commented 8 months ago

It should be this code gave the errno 35: https://github.com/Azure/azure-storage-fuse/blob/971280911b0c39821ceaefc779208f56707113e8/cpplite/include/executor.h#L152

        template<>
        class async_executor<void> {
        public:
            static void submit_helper(
                std::shared_ptr<std::promise<storage_outcome<void>>> promise,
                std::shared_ptr<storage_outcome<void>> outcome,
                std::shared_ptr<storage_account> account,
                std::shared_ptr<storage_request_base> request,
                std::shared_ptr<http_base> http,
                std::shared_ptr<executor_context> context,
                std::shared_ptr<retry_context> retry)
            {
                http->reset();
                http->set_error_stream(unsuccessful, storage_iostream::create_storage_stream());
                request->build_request(*account, *http);

                retry_info info = retry->numbers() == 0 ? retry_info(true, std::chrono::seconds(0)) : context->retry_policy()->evaluate(*retry);
                if (info.should_retry())
                {
                    http->submit([promise, outcome, account, request, http, context, retry](http_base::http_code result, storage_istream s, CURLcode code)
                    {
                        if (code != CURLE_OK || unsuccessful(result))
                        {
                            storage_error error;
                            if (code != CURLE_OK)
                            {
                                error.code = std::to_string(code);
                                error.code_name = curl_easy_strerror(code);
                            }
                            else
                            {
                                std::string str(std::istreambuf_iterator<char>(s.istream()), std::istreambuf_iterator<char>());
                                error = context->xml_parser()->parse_storage_error(str);
                                error.code = std::to_string(result);
                            }

                            *outcome = storage_outcome<void>(error);
                            retry->add_result(code == CURLE_OK ? result: 503);
                            http->reset_input_stream();
                            http->reset_output_stream();
                            async_executor<void>::submit_helper(promise, outcome, account, request, http, context, retry);
                        }
                        else
                        {
                            *outcome = storage_outcome<void>();
                            promise->set_value(*outcome);
                        }
                    }, info.interval());
                }
                else
                {
                    promise->set_value(*outcome);
                }
            }

Thus, error code 35 should be libcurl error. https://curl.se/libcurl/c/libcurl-errors.html CURLE_SSL_CONNECT_ERROR (35) A problem occurred somewhere in the SSL/TLS handshake. You really want the error buffer and read the message there as it pinpoints the problem slightly more. Could be certificates (file formats, paths, permissions), passwords, and others.

In the setup there's HTTP proxy server. Network packet was captured, and the HTTP did not respond properly.

Thank you for your reply. Closing the issue.

vibhansa-msft commented 8 months ago

Thanks for confirming. Blobfuse2 does not have any dependency on curl as well. Kindly upgrade to our latest version of Blobfuse2.

Azure / azure-storage-fuse