search

Azure / azure-webjobs-sdk

Azure WebJobs SDK
MIT License
739 stars 358 forks source link

Update Microsoft.Azure.WebJobs.Extensions.Storage to use Microsoft.Azure.Storage.Blob 9.4 #2065

Open erick-thompson opened 5 years ago

erick-thompson commented 5 years ago

I am updating some WebJobs to use the RTM version of v3 of the packages. This all works ok, except that the latest version of Microsoft.Azure.WebJobs.Extensions.Storage (3.0.2) has a reference to WindowsAzure.Storage (9.3.1).

I need to reference Microsoft.Azure.Storage.Blob 9.4 to use BlobEncryptionPolicy (configuring containers as encrypted). The problem is that if I reference Microsoft.Azure.Storage.Blob 9.4 I get errors because there are identical types in both WindowsAzure.Storage and Microsoft.Azure.Storage.Blob. For example, CloudStorageAccount is defined in both. As a result, I can't reference both Microsoft.Azure.WebJobs.Extensions.Storage and Microsoft.Azure.Storage.Blob.

When will the Web Jobs SDK get bumped to use Microsoft.Azure.Storage.Blob 9.4 (and remove WindowsAzure.Storage)? This is a blocker for me. I'm happy to provide more information if needed.

erick-thompson commented 5 years ago

Has anyone had a chance to look at this? It is currently blocking our ability to move to the RTM WebJob library.

aragacalledpat commented 5 years ago

This is also an issue for me - still optimistically looking for a workaround.

tonycoelho commented 5 years ago

Can someone from the PG please provide a timeline for when this issue will be resolved? @fabiocav

tonycoelho commented 5 years ago

@erick-thompson @aragacalledpat I was able to work around this by using the C# extern alias feature. It's not the best solution because it requires you to create aliases for the conflicting dlls, but will allow you to use the new Microsoft.Azure.Storage.* NuGet packages with WebJobs or Azure Functions until the PG has a chance to upgrade the reference. Take a look at https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/extern-alias for more information.

Also, if you are using .NET Core, then keep in mind that aliases are not built-in, but you can still use them by defining a custom target in your csproj. Take a look at https://stackoverflow.com/a/52630022 for more information. Hope that helps...

StephenCleary commented 5 years ago

Given the sporadic progress at https://github.com/Azure/azure-cosmos-dotnet-v2/issues/344, I'm not surprised Microsoft.Azure.WebJobs.Extensions.Storage depends on WindowsAzure.Storage.

From what I can make out:

In summary, there is no supported Azure Storage Tables client for netstandard right now.

StephenCleary commented 5 years ago

Update: Microsoft.Azure.Cosmos.Table is now GA. I expect now it's just a matter of time before Microsoft.Azure.WebJobs.Extensions.Storage is updated.

However, now that Microsoft.Azure.Storage.* and Microsoft.Azure.Cosmos.Table packages are all split up by storage type, perhaps it would make sense to split up Microsoft.Azure.WebJobs.Extensions.Storage, too.

StephenCleary commented 5 years ago

@mattchenderson This is the update/splitting work I talked with you about in person last week. The way I see it, there are a couple of possible work items:

Can you bring this to the team and let us know:

1) What the direction is - whether to move forward with either or both of these. 2) If the community can take on this work.

twentytwokhz commented 5 years ago

What's the current status? Version 3.0.5 of Microsoft.Azure.WebJobs.Extensions.Storage still references the WindowsAzure.Storage (>= 9.3.1)

electricessence commented 5 years ago

Could someone explain why there is still no option to use the latest packages? Has anyone provided an alternative version/fork of the code that uses Microsoft.Azure instead of Microsoft.WindowsAzure?

andrew-vdb commented 5 years ago

Calling @jeffhollan

electricessence commented 5 years ago

https://github.com/Azure/app-service-announcements/issues/200 Is there any communication between the teams here to be certain that Functions V3 will have the latest packages? It seems that if v3 leaves behind Microsoft.WindowsAzure then this problem might go away?

electricessence commented 5 years ago

@fabiocav, can you comment on the above?

fabiocav commented 5 years ago

@electricessence the extensions will be revved. There will be new major versions of the storage extension, as well as the storage based host services, using the latest storage packages.

Once those are out, you'll be able to use those newer versions to consume the latest SDKs.

electricessence commented 5 years ago

@fabiocav That is great news! Thank you. Last question: if this is a 'preview' how will I be able to get it working with Visual Studio, and will Azure itself support it right away?

electricessence commented 5 years ago

@fabiocav Is there any way I can get early access to the preview. I'm kind of in a bind, and it would be helpful if I could get ahead of the curve.

electricessence commented 5 years ago

Ok just checking in... After some trials I've discovered AFv3 (preview) is still targeting 9.3.1 :( Is my only hope to wait for this to change? I'm in an serious bind that will cost a lot of time if I have to revert and remain in AF limbo. Where all other updated projects can consume 9.4 and use new STJ packages, I'm stuck fighting with this.

fabiocav commented 5 years ago

@electricessence the host reference isn’t important. What matters here is the version of the extension (storage bindings). The work to update that is in progress and will be applicable to both, V2 and V3.

electricessence commented 5 years ago

@fabiocav Ok that's great. Would I be wasting my time at preemptively setting up a binding redirect? Or is that simply not going to work in AF?

ghost commented 4 years ago

Is there an update on this? It seems that the Functions SDK went to non-preview 3.0 today, which seems to suggest that it's production ready (soon). Yet it probably still means that it references the old 9.3.1, while 11.1.1 is already available nowadays. I'd like to mock instances of various interfaces/classes of this library for unit tests, but it seems to be supported in 10+ only.

erick-thompson commented 4 years ago

I'd like to know about the update as well. We have some very nasty workarounds in place that I would love to be rid of. For a service a high profile as Azure Functions, I'm not sure how it can reference the old version of the one assembly/package that every Azure project needs.

paulbatum commented 4 years ago

Hey folks, I've been working on a set of updates that move to storage 11.x. I was hoping to have it ready by now but it was delayed while we focused on getting functions V3 into a production ready state.

Here are the in-progress PRs: https://github.com/Azure/azure-webjobs-sdk/pull/2356 https://github.com/Azure/azure-webjobs-sdk-extensions/pull/602 https://github.com/Azure/azure-functions-host/pull/5218

My best guess for an ETA is towards the end of Jan.

fabiocav commented 4 years ago

Is there an update on this? It seems that the Functions SDK went to non-preview 3.0 today, which seems to suggest that it's production ready (soon). Yet it probably still means that it references the old 9.3.1, while 11.1.1 is already available nowadays. I'd like to mock instances of various interfaces/classes of this library for unit tests, but it seems to be supported in 10+ only.

@b10-dslappendel Just want to clarify, the Functions SDK does not have a dependency on the storage SDK. That dependency is completely isolated as part of the storage extension, which is what @paulbatum is working on moving to a newer release. Once that is done, you'll be able to use it in V3 and V2

ghost commented 4 years ago

@fabiocav Ah yes, you're correct. That's some bad wording on my part.

dersia commented 4 years ago

Hey folks, I've been working on a set of updates that move to storage 11.x. I was hoping to have it ready by now but it was delayed while we focused on getting functions V3 into a production ready state.

Here are the in-progress PRs:

2356

Azure/azure-webjobs-sdk-extensions#602 Azure/azure-functions-host#5218

My best guess for an ETA is towards the end of Jan.

I started 3 Days ago an update myself and didn't know about your work on this. Anyhow I did the update too, but I guess that I don't need to create a PR.

Anyhow, I was also thinking about if I should update to v12. I came to the conclusion, that v12 has so many breaking changes, that it is worth to update to v11 and then update to v12 with the next major version of webjobs sdk.

If you'd like to see my changes: https://github.com/Azure/azure-webjobs-sdk/compare/dev...dersia:dev

paulbatum commented 4 years ago

@dersia Thanks for sharing! Sorry I didn't update this issue earlier regarding my progress.

dersia commented 4 years ago

No worries 😊

chrismimosil commented 4 years ago

Do we know where this is? Looking at those 3 in-progress PRs to bring the referenced version of storage up to v11 or v12, they seem to be stopped since Jan. I'd really like to see the storage version updated as the current v9 version is over 90% of a deployed minimal functions app, and the split of storage libs will presumably reduce the size of referenced libs?

paulbatum commented 4 years ago

@chrismimosil sorry for not getting back to you sooner, this got lost in my sea of notifications.

We released version 4.0.0-preview1 of the storage extension. It references storage 11.x. However we did not split our storage extension itself, so the binary size probably won't change much.

I think its safe to assume that the next major version of the functions/webjobs storage extension will be split into multiple packages to be consistent with the split that storage did. But I can't really comment on an ETA for that because its a big set of changes (dramatic surface area changes in storage 12.x).

andrew-vdb commented 4 years ago

In v12, they rename BlobClient to BlobServiceClient and make another different type of BlobClient They also put the project in giant mega jumbo repo

This issue also obsolete as I don't see the point of aiming v11 anymore, but its depend on your product manager i guess

paulbatum commented 4 years ago

The point of targeting v11 is twofold:

  1. It was more achievable for us - the required updates to our storage extension and functions host were relatively small because not a whole lot changed in terms of API surface area.
  2. Thousands of customers have code that is written against the pre v12 API surface. Those customers need to be able to get reliability and performance fixes without requiring a rewrite and the storage team continues to deliver those as hotfixes for v11.
BrunoZell commented 4 years ago

@paulbatum Is the difference between v11 and v12 synonymous with the different packages Microsft.Azure.Storage.* vs Azure.Storage.*? And will there be any solution for adopters of Azure.Storage.*?

paulbatum commented 4 years ago

Yes, that is what I'm referring to. If you look at the Azure.Storage.* packages they start at 12.x

We will update our storage extension to support 12.x but I don't have a ETA yet, sorry.