Fix Azure Storage HNS Support

Azure / durabletask

Durable Task Framework allows users to write long running persistent workflows in C# using the async/await capabilities.

Apache License 2.0

1.53k stars 296 forks source link

Fix Azure Storage HNS Support #1123

Closed wsugarman closed 4 months ago

wsugarman commented 5 months ago

This PR fixes a bug when enumerating the lease blobs. When hierarchical namespaces (HNS) are enabled for a storage account, the "List Blobs" API returns the folder itself as a "blob" in the response (in addition to the blobs themselves), which results in a NullReferenceException. This PR skips those blobs by checking for the internal hdi_isfolder metadata property.

This PR also includes a refactor of how projects/filters are done on top of paginated data.

wsugarman commented 4 months ago

Thanks. I really like this change as well, but I have some suggestions here for readability. I also suspect it would be good have caught this during testing. I'm curious - is Azurite even able to simulate HNS storage configurations? If so - it might be worth adding a test for that (we can take care of that if you don't have the cycles)

Unfortunately, Azurite does not appear to support ADLS Gen 2, so we would need a proper integration test connecting to a real Azure Storage account with HNS enabled ☹️

davidmrdavid commented 4 months ago

Let's wait until GitHub test action is executed on this branch