Description

Weight the pros and cons of ADLS versus blob, and select which one to implement.

ADLS versus Blob

https://medium.com/awesome-azure/azure-difference-between-azure-blob-storage-and-azure-data-lake-storage-comparison-azure-blob-vs-adls-gen2-81af5ef2a6e1#:~:text=Key%20Differences&text=Blob%3A%20General%20purpose%20object%20store,for%20big%20data%20analytics%20workloads.&text=Blob%3A%20Good%20storage%20retrieval%20performance,ADLS%3A%20Better%20storage%20retrieval%20performance.

ADLS

Hierarchical permissions
Files actually exist in the container in the structure they represent
More performant for big data tasks
Matches the recommendations from FDI, and in general the cloud teams backing container storage solutions

Blob

Still have hierarchal folder structure for the user via the blob-csi driver
Less performant for big data tasks
Older technology

Recommendation

Use ADLS, as a migration will likely need to happen in the future and this is the time to do it (no one is using the system yet, so its the easiest time to migrate)

Estimated Steps for Creating ADLS Containers

In https://github.com/statcan/aaw-kubeflow-profiles-controller, we want to update the controller to provision ADLS containers.

[ ] Add hierarchical namespace option that defaults to false within:
- [ ] Update https://gitlab.k8s.cloud.statcan.ca/cloudnative/terraform/modules/terraform-azurerm-storage-account/-/tree/main/ to include reference to https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account#is_hns_enabled as an output
- [ ] Update reference here https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/minio_gateway.tf#L60 to new version once above is complete.
- [ ] Add the new option for hierarchical name-spacing
- [ ] DOUBLE CHECK WHAT HAPPENS WHEN THIS IS IMPLEMENTED TO ENSURE NO DATA LOSS
[ ] Provisioning of new containers is done here: https://github.com/StatCan/aaw-kubeflow-profiles-controller/blob/ddcdb6708f31be927c83507023d22cf655a7c5af/cmd/blob-csi.go#L524
[ ] PV mount options will need updated to use ADLS AAW containers:
- Take this line here: https://github.com/StatCan/aaw-kubeflow-profiles-controller/blob/main/cmd/blob-csi.go#L329, and add here: https://github.com/StatCan/aaw-kubeflow-profiles-controller/blob/main/cmd/blob-csi.go#L309
[ ] Replace endpoint https://github.com/StatCan/aaw-kubeflow-profiles-controller/blob/main/cmd/blob-csi.go#L309 to be dfs instead of blob (see: https://docs.microsoft.com/en-us/azure/storage/common/storage-account-overview#standard-endpoints)
[ ] Delete one container, observe it get re-created and confirm that it is ADLS

StatCan / aaw