Azure storage service dependency `azure-storage-blob` deprecated

[ahukkanen]

Steps to reproduce

1. Create a rails app
2. Within the Gemfile, add the following:

   gem "azure-storage-blob", "~> 2.0", require: false
   gem "faraday", "~> 2.0"

3. Run bundle --verbose
4. See error

Expected behavior

I would expect Rails applications to support newer versions of the Faraday gem because it is a popular dependency for many other gems that deal with HTTP requests.

Example gems that have this dependency:

• json-jwt
• oauth2 (also works with Faraday version 1)
• openid_connect
• rack-oauth2
• swd
• webfinger

Actual behavior

With any other dependency that relies on Faraday version 2 or above, the Azure storage cannot be used as instructed in the Active Storage documentation.

The Azure storage's dependency azure-storage-blob is deprecated as pointed out in the README of the project: https://github.com/Azure/azure-storage-ruby

This means that fixes that would solve the issue are not getting merged in a timely manner and the gem has to be forked in order to apply the necessary changes making it harder to maintain the dependencies.

System configuration

Rails version: 6.1 (applies also to version 7)

Ruby version: 3.0.6

[ahukkanen]

Just to be perfectly clear (as I'm not sure how the tagging system works here), I also want to add that azire-storage-blob is set as a hard dependency within the Active Storage gem's AzureStorageService as defined here: https://github.com/rails/rails/blob/4cc1de7910dee1dbb0f4abe509a7ca8995ea83ea/activestorage/lib/active_storage/service/azure_storage_service.rb#L3

This is why I created the issue at the Rails repository in the first place.

So the solutions from Rails repository's perspective I would see for this issue are:

1. Change the azure-storage-blob dependency to something else or create a custom integration for the necessary APIs (as suggested by the azure-storage-blob gem's deprecation note, call the APIs directly)
2. Fork the azure-storage-blob gem and do the necessary changes there to keep the same dependency
3. Remove the Azure storage support from the Rails repository

[zzak]

If azure-storage-blob is deprecated, then I think it's worth investigating a replacement or removal.

[rails-bot[bot]]

This issue has been automatically marked as stale because it has not been commented on for at least three months. The resources of the Rails team are limited, and so we are asking for your help. If you can still reproduce this error on the 7-1-stable branch or on main, please reply with all of the information you have about it in order to keep the issue open. Thank you for all your contributions.

[ahukkanen]

This problem still exists for both the main and 7-1-stable branches.

main: https://github.com/rails/rails/blob/85c58ffa364414d74ab1f442218959818225d708/activestorage/lib/active_storage/service/azure_storage_service.rb#L3

7-1-stable: https://github.com/rails/rails/blob/ed84c0d8b0016fd8387d34453803e21c177bcb2b/activestorage/lib/active_storage/service/azure_storage_service.rb#L3

[tomfast]

:fire:

[Michoels]

FWIW, Microsoft deprecated the Azure Ruby SDK back in 2021:

As of Februrary 2021, Azure SDK for Ruby - Resource Management Libraries are officially retired. The libraries will be in maintenance mode until December 31, 2021. However, there will be no further releases based on feature requests. In the future, we recommend that you interface with the Azure REST APIs directly from Ruby; instructions are provided in this doc.

I've been using this fork of the azure-storage gem by @honeyankit, which seems to work pretty well.

[JoeDupuis]

I've been working for a couple of weeks on a simpler gem to replace azure-storage-blob in Active Storage. I am hoping to be done with it sometime in June.

azure-storage-blob is huge, and Active Storage only uses a small subset of the features. This new gem will depend only on default or bundled gems.

[Cosmo]

@JoeDupuis doing gods work! Any task where you need help?

[JoeDupuis]

Thank you! I don't think so for now. Soon™ though. The code is still pretty rough, and I plan a big refactor at some point, so it would not make sense to review anything now.

I was re-implementing the functions Active Storage depends on one by one, keeping a close eye on both the current version from azure-storage-blob and the Azure docs. Once I get most things working, I'll rework some abstractions and the general structure. I don't like the current state, but I needed something to explore the Azure API and start from there.

I have the download/upload (through single and multiple requests) working. Here's the repo if you want to take a look: https://github.com/testdouble/azure-blob. This is still very much a WIP.

[JoeDupuis]

I got the Active Storage tests working on my branch.

I'll try to open the PR this week.

[zzak]

So one possible (maybe most likely) outcome is that ActiveStorage doesn't ship with an Azure adapter, we deprecate this one, and this behavior is provided by a gem. My reasoning is that we can't make the same guarantees with a first-party supported library, and any gem that re-implements their API will not be an atomic change, in other words it's probably safer to offer a migration path than to make the change implicitly.

We also haven't been running the azure integration tests in CI for some time, and I'm not sure if anyone on core is using it.

[JoeDupuis]

I figured this was a likely outcome before I started on this. Though, I think it's still worth discussing. It's a major provider after all.

I'll open the PR and we can discuss it with folks on core.

Worst case I'll close it and write a migration guide.

[zzak]

Ok, I just wanted to make sure expectations were clear.

Thank you for working on this!

[rafaelfranca]

We decided to remove Azure support from Rails on 8.0 given Microsoft isn't interested in the Ruby community. We aren't working on this right now, but if you are Azure users, we encourage the community to extract the Active Store support to a gem.

[JoeDupuis]

Thank you for the fast response, I'll get started on the extraction.

[JoeDupuis]

First working release is ready! https://github.com/testdouble/azure-blob or 0.4.1 on RubyGems: https://rubygems.org/gems/azure-blob

If anyone is looking to help, you can switch your app to use this extracted adapter and tell me if you hit any issues.

The migration instructions are here. Thank you!

[leshik]

@JoeDupuis does it support authentication using managed identities? If not, how hard would be to implement it?

[JoeDupuis]

If the existing adapter supports it, this one might as I mostly kept the same API. I haven't tested it though. I plan to take a look this week.

[leshik]

If the existing adapter supports it, this one might as I mostly kept the same API. I haven't tested it though. I plan to take a look this week.

I wasn't able to make it work with the existing adapter, and the information about the support is quite contradictory, please see here and here. Maybe I'm doing something wrong.

[jasoncodes]

Unfortunately there’s a fair bit of work to get Managed Identities working even with the deprecated Azure libraries. It’s not supported by the current Active Storage adapters.

Using a Managed Identity requires first calling the appropriate MSI endpoint to retrieve an access token. This varies depending on what managed environment you are running in (e.g. App Service). This part is not handled at all by the deprecated Azure libraries. You have to work out how to get and refresh access tokens externally.

Once you have the access token and a way to refresh it periodically, one needs to create a token signer and use that to sign requests. https://github.com/Azure/azure-storage-ruby/blob/master/blob/README.md#access-token shows one way to do this. Things start to get complicated though with shared access signatures (i.e. signed URLs) as you need to retrieve a user delegation key instead of using the storage access key. This delegation key must have an expiry at least as long as the URL expiry and needs to be periodically refreshed to extend the expiry. The deprecated Azure libraries have all the primitives for this but none of this is supported by the current Active Storage adapter.

The main things that need to happen in the new Azure Blob Storage library is to support creation with a token credential object which wraps the access token to support updating the token on expiry, using “Bearer” token authentication instead of “SharedKey” authentication, and doing the whole delegation key dance when generating signed URLs. The deprecated library does not handle any of the delegation key stuff for you but it certainly could be abstracted away in a reimplementation.

[JoeDupuis]

I had discussed it with a few folks internally already and it was on my to-do list for after I was done supporting the existing feature set of the adapter. It was not a priority since the existing adapter didn't seem to support it. I just went through a bunch of documentation. I am hoping to work on it sometime in the next couple of weeks.

[Michoels]

The Managed Identity infrastructure would also be super useful for the ActiveRecord SQL Server adaptor, where Managed Identity support has been requested many times (issue here).

[dan-corneanu]

I've started some work on Managed Identity. Have a look at this PR https://github.com/testdouble/azure-blob/pull/1