[BlockBlobClient] class - Invalid URL error message

[malyalavenu]

I am trying to do a multi-part upload to azure blob store from a ruby on rails app that uses dropzone.js, stimulus.js and DirectUpload from @rails/activestorage. Below are the version and package/gem information. Active storage currently doesn't support multi-part upload for large files (>256 MB) to azure blob store. So I am trying to replace the @rails/activestorage functionality with @azure/storage-blob

I have an import as below in my dropzone_controller.js

import { BlobServiceClient } from "@azure/storage-blob";

Below is the code to do the multipart upload

 async uploadWithAzureStorage() {
    try {
      const response = await fetch('/posts/generate_sas_token', {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'X-CSRF-Token': getMetaValue('csrf-token')
        },
        body: JSON.stringify({ blob_name: this.file.name })
      });

      const data = await response.json();
      let sasToken = data.sas_token;

      // Sanitize the SAS token if necessary
      sasToken = sasToken.startsWith('?') ? sasToken.substring(1) : sasToken;

      // Construct the BlobServiceClient
      const blobServiceClient = new BlobServiceClient(
        `https://fileuploadblobstore.blob.core.windows.net/?${sasToken}`
      );

      // Encode the file name if it contains special characters
      const encodedFileName = encodeURIComponent(this.file.name);

      // Get the container client and block blob client
      const containerClient = blobServiceClient.getContainerClient('development');
      const blockBlobClient = containerClient.getBlockBlobClient(encodedFileName);

      // Upload the file using the block blob client
      const uploadResponse = await blockBlobClient.uploadData(this.file, {
        blockSize: 4 * 1024 * 1024, // 4MB block size
        concurrency: 20, // 20 concurrent uploads
        onProgress: (ev) => this.uploadRequestDidProgress(ev)
      });

      // Set the hidden input value and emit success event
      this.hiddenInput.value = blockBlobClient.url; // Adjust based on your backend requirements
      this.emitDropzoneSuccess();
    } catch (uploadError) {
      console.log('upload error is ', uploadError);
      this.emitDropzoneError(uploadError);
    }
  }

The URL formed is as follows. I removed the sensitive information https://<azure-storage-account-name>.blob.core.windows.net/<azure-container-name>/large_image1.jpg?sv=2018-11-09&sr=b&sp=rw&se=2024-07-05T06%3A41%3A23Z&st=2024-07-05T05%3A11%3A23Z&spr=https&sig=<sas_token_signature>

This is throwing an error of Invalid URL and I could trace it as per the console message in the following method blockBlobClient.uploadData(); I was able to trace it, but not able to figure why this is happening

  upload error is  TypeError: Failed to construct 'URL': Invalid URL
    at appendQueryParams (urlHelpers.js:192:1)
    at getRequestUrl (urlHelpers.js:49:1)
    at StorageContextClient.sendOperationRequest (serviceClient.js:65:1)
    at StorageContextClient.sendOperationRequest (extendedClient.js:43:1)
    at StorageContextClient.sendOperationRequest (StorageContextClient.js:13:1)
    at BlockBlobImpl.stageBlock (blockBlob.js:66:1)
    at Clients.js:1540:1
    at tracingClient.js:43:1
    at Object.withContext (instrumenter.js:42:1)
    at withContext (tracingClient.js:62:1)

I am requesting help on two things: 1) Why is this error happening ? 2) Can there be a more descriptive error message, like invalid SAS token if it is, or as per the error

Gemfile

source 'https://rubygems.org'
git_source(:github) { |repo| "https://github.com/#{repo}.git" }

ruby '3.2.2'

# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem "rails", "~> 6.1"
# Use sqlite3 as the database for Active Record
gem 'pg'
# Use Puma as the app server
gem 'puma', '~> 3.12'
# Use SCSS for stylesheets
gem 'sass-rails', '>= 6'
# Transpile app-like JavaScript. Read more: https://github.com/rails/webpacker
gem 'webpacker', '~> 5.x'
# Turbolinks makes navigating your web application faster. Read more: https://github.com/turbolinks/turbolinks
gem 'turbolinks', '~> 5'
# Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder
gem 'jbuilder', '~> 2.7'
gem "cocoon"
gem 'pry'
# Use Redis adapter to run Action Cable in production
# gem 'redis', '~> 4.0'
# Use Active Model has_secure_password
# gem 'bcrypt', '~> 3.1.7'

# Use Active Storage variant
# gem 'image_processing', '~> 1.2'

# Reduces boot times through caching; required in config/boot.rb
gem 'bootsnap', '>= 1.4.2', require: false
# gem 'azure-storage', '~> 0.15.0.preview', require: false
gem 'azure-storage-blob', '~> 2.0'
gem 'active_storage_validations'
group :development, :test do
  # Call 'byebug' anywhere in the code to stop execution and get a debugger console
  gem 'byebug', platforms: [:mri, :mingw, :x64_mingw]
  gem 'dotenv-rails'
end

group :development do
  # Access an interactive console on exception pages or by calling 'console' anywhere in the code.
  gem 'web-console', '>= 3.3.0'
  gem 'listen', '>= 3.0.5', '< 3.2'
  # Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring
  gem 'spring'
  gem 'spring-watcher-listen', '~> 2.0.0'
end

group :test do
  # Adds support for Capybara system testing and selenium driver
  gem 'capybara', '>= 2.15'
  gem 'selenium-webdriver'
  # Easy installation and use of web drivers to run system tests with browsers
  gem 'webdrivers'
end

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby]

gem 'devise', '~> 4.7'
gem 'friendly_id', '~> 5.2', '>= 5.2.5'
gem 'sidekiq', '~> 5.2', '>= 5.2.7'

package.json

{
  "name": "active_storage_drag_and_drop_demo",
  "private": true,
  "dependencies": {
    "@azure/storage-blob": "^12.23.0",
    "@nathanvda/cocoon": "^1.2.14",
    "@rails/actioncable": "^6.0.0-alpha",
    "@rails/activestorage": "^6.0.0-alpha",
    "@rails/ujs": "^6.0.0-alpha",
    "@rails/webpacker": "^5.4.4",
    "axios": "^1.7.2",
    "dropzone": "^5.5.1",
    "jquery": "^3.7.1",
    "stimulus": "^1.1.1",
    "tailwindcss": "^1.1.2",
    "turbolinks": "^5.2.0"
  },
  "version": "0.1.0",
  "devDependencies": {
    "@babel/plugin-transform-for-of": "^7.24.7",
    "webpack-dev-server": "^3.8.0"
  }
}

I am generating a SAS token using the following ruby controller code

class PostsController < ApplicationController
  require 'azure/storage/blob'
  before_action :authenticate_user!

  def generate_sas_token
    begin
      storage_account_name = '<account_name>'
      storage_access_key = '<account_access_key>'
      container_name = '<container_name>'
      blob_name = params[:blob_name]

      token = generate_blob_sas_token(storage_account_name, storage_access_key, container_name, blob_name)
      render json: { sas_token: token }
    rescue => e
      render json: { error: "Error generating SAS token: #{e.message}" }, status: :internal_server_error
    end
  end

  def new
    @post = Post.new
  end

  private

  def generate_blob_sas_token(account_name, account_key, container_name, blob_name)
    require 'base64'
    require 'openssl'
    require 'uri'

    permissions = "rw" # Read and write permissions
    start_time = Time.now.utc - 30 * 60 # 5 minutes ago to account for clock skew
    expiry_time = (Time.now.utc + 1.hour).iso8601

    string_to_sign = [
      permissions,
      start_time.iso8601,
      expiry_time,
      "/blob/#{account_name}/#{container_name}/#{blob_name}",
      "",
      "https",
      "2018-11-09"
    ].join("\n")

    decoded_key = Base64.strict_decode64(account_key)
    signature = Base64.strict_encode64(OpenSSL::HMAC.digest(OpenSSL::Digest.new('sha256'), decoded_key, string_to_sign))

    sas_token = URI.encode_www_form({
      "sv" => "2018-11-09",
      "sr" => "b",
      "sp" => permissions,
      "se" => expiry_time,
      "st" => start_time.iso8601,
      "spr" => "https",
      "sig" => signature
    })

    sas_token
  end
end

Here is the routes.rb code for generate_sas_token

require 'sidekiq/web'

Rails.application.routes.draw do
  resources :posts do
    collection do
     post 'generate_sas_token', to: 'posts#generate_sas_token'
    end
  end

  authenticate :user, lambda { |u| u.admin? } do
    mount Sidekiq::Web => '/sidekiq'
  end
end

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[xirzec]

The error message is coming from the platform, not something the SDK is throwing:

I'm guessing it happens here: https://github.com/Azure/azure-sdk-for-js/blob/d133eaf30319a2bca14661c2a5bca2c995b6b9ef/sdk/core/core-client/src/urlHelpers.ts#L266

The URL you have provided doesn't appear to be invalid, at least not unless the invalid part is coming from one of the pieces you redacted. Does the URL constructor throw for you when you provide the full unredacted URL?

[malyalavenu]

I think the issue is with the SAS token on the ruby side. I tried the same with a node app and it works fine

[xirzec]

I notice you are using URI.encode_www_form which reminded me that there were some subtleties in the ways that formdata is encoded that is different from how we expect URL query parameters to be encoded, specifically around encoding the + character as a space, which causes trouble with base64 data: https://developer.mozilla.org/en-US/docs/Web/API/URLSearchParams#preserving_plus_signs

So if I had to guess, the sig parameter isn't being encoded properly by URI.encode_www_form which is why it is producing an invalid SAS URL.

[github-actions[bot]]

Hi @malyalavenu. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions[bot]]

Hi @malyalavenu, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!