Question: forward compatibiity with Azure Data Lake Storage Gen2?

Azure / azure-storage-cpp

Microsoft Azure Storage Client Library for C++

http://azure.github.io/azure-storage-cpp

Apache License 2.0

132 stars 148 forks source link

Question: forward compatibiity with Azure Data Lake Storage Gen2? #270

Open yxiang92128 opened 5 years ago

yxiang92128 commented 5 years ago

Can somebody please comment on the SDK's forward compatibility with Azure Data Lake Storage Gen2? Will the SDK be expanded to support Gen2 especially with the new authentication schemes and/or ACL methods?

Also according to the article here: https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-known-issues

"Blob storage APIs are disabled to prevent feature operability issues that could arise because Blob Storage APIs aren't yet interoperable with Azure Data Lake Gen2 APIs."

Can somebody please clarify the above statement in the article?

Thanks,

Yang

katmsft commented 5 years ago

Yes, there will be forward compatibility for ADLS Gen2 support in the SDK, it is already in the backlog and the priority is high. Although I cannot share the exact date, but it is promising to be released in the next couple of months.
There are several phases of ADLS Gen2's support, and Blob storage API enablement, which allows you to use ADLS Gen2 object as blob is in the later phases that isn't officially supported yet. With this said the SDK has a dependency of the service feature light up schedule.

Can you drop us a direct email stating your company's user scenario, potential usage volume, and timeline for ADLS Gen2 service so we can plan better accordingly? Thanks!

kummishra commented 4 years ago

hi ,

Any update on the feature release?

We are moving to adls gen2, our team is looking for native technologies to access blobs in storage account. we tested with c++ blob api, but it seems cpu intensive. we also tested with r-jni based hdfs call to abfs, which seems fast but has memory issues with the jar.

currently azure-storage-sdk-cpp acheives better cohesiveness with our product. would be much better with adls gen2 feature support.

Our potential usage volume : read and write 5+ TB in files every day. our timeline is : within 3-4 months.

thanks.

katmsft commented 4 years ago

Hi, thanks for raising this issue. In fact we have a lite version SDK for Azure Storage that already supports ADLS Gen 2, with better CPU utilization and performance. The ADLS Gen2 support is added in 0.2.0 version: https://github.com/Azure/azure-storage-cpplite. If this however not ideal for you, since you have maybe already been using this SDK as a dependency and migration is not easy, please do let us know so we will have more ammunition of prioritizing this feature.

Best regards.