Integration with Azure blob storage

[shyamal890]

Will NextCloud support Azure blob storage as external storage in near future?

[MariusBluem]

Hm ... Let's put this as an enhancement. However - I do not see this in the near future 😁 ... Maybe you want to step in - or know somebody who wants so 😅

We have some external storage backends in Nextcloud, (Amazon) S3 seems to be finished for (primary) storage for Nextcloud 11: https://github.com/nextcloud/server/pull/768

This may be an alternative ... Sharepoint is also on the roadmap for 11, if you rely on Microsoft 😁

cc @icewind1991 @blizzz

[pascalsimi]

I find it rather puzzling that blob is not supported. If you look at the gartner quadrant then there is Amazon and Microsoft as the big players. Supporting only one of them cuts down potential user base by approximately 50%. Take into account how many personal computers actually run on Microsoft it's even more puzzling to me why Microsoft blob is not on the roadmap. But hey its just my thoughts. Thanks for all the hard work you put into next cloud. All you guys are awesome!

[gjonespf]

Would love to see this also. May be worth mentioning that (and I haven't tested this) but Azure File Store allows SMB access, so you could potentially run Nextcloud on an Azure VM with external storage mapped to the Azure File Store, as long as the VM and storage were in the same region (otherwise you need SMB3 encryption support, which isn't there yet in Linux). Something I will have to investigate.

[pascalsimi]

I have a few projects with Microsoft azure for processing and nextcloud for storage and sync right now and with their new push for openness on azure I think there is a good chance their evangelists might be interested in helping to implement that feature. I'll mention it in the next meeting.

[pilcherd]

Support for Azure Blob would be good. Interesting about Azure File Store and SMB. Will look into that.

[MariusBluem]

https://github.com/icewind1991/files_external_azure

Digging for old code 😜😜 What was the state there @icewind1991 🤔🙈

[icewind1991]

Not working atm, will see if I can fix it when I get some time

[strategywise]

We're also interested in this ability. If the SMB mapping on the host box works would it in theory at least be able to read/write to the blob? If I have to get a blob mounted on the VM and it works well I don't think that would be a very hard work around for the time being.

[SpiraMirabilis]

I've been tinkering with this, using the S3 backend as a template. It's a bit of a work in progress. However, as a temporary measure you can use a Windows server (2008 or greater), with WebDAV authoring role for IIS and essentially act a proxy in between your users and azure blob storage, see this how-to: http://fabriccontroller.net/mounting-your-azure-file-shares-on-premises-through-webdav/ Edit: Actually, it looks like that only supports the Azure Files which is much more expensive than blob.

Should it support serving from the verizon/akamai CDN if it is linked to that storage account?

Also, either nextcloud will need dedicated access to the blob container, or maybe we can use a Service Bus subscription and just mandate that anyone adding/changing/removing a file triggers a message sent, that way we can confidently cache the metadata/contents and there wouldnt be polling which increases cost a lot.

[MorrisJobke]

This should be definitely be implemented as an app - see the files_external_dropbox external storage for example: https://github.com/icewind1991/files_external_dropbox

[CyrilHupor]

+1 on the feature which I believe is very important.

I can't however understand why this support should be an app when out of the box support is already included for AWS and OpenStack. Keep in mind Azure is leader together with AWS in the cloud space. I think this matter should be re-visited.

Btw, the reason as to why Blow Storage might be considered as opposed to Azure Files or additional storage on the VM is simply cost. Blob storage is much cheaper.

On another note, congrats to dev's on nextcloud, would just very much like support for Azure Blob.

[jbraes]

I am always amazed how people use software is such different ways. running your "personal" cloud on-top of "public" cloud infrastructure. Isn't the idea of having your own cloud infrastructure , to be independent of azure, Amazon , google etc , to have full controle over your hardware

[MourIdri]

+1

[wvh-shanee]

+1

@jbraes with e2e encryption the public cloud just ends up as a big storage location to stash encrypted files. Some prefer Azure to Amazon so it would be nice to support multiple storage options. I personally like the public cloud because of automatic backups, geographic redundancy, unlimited storage, etc. Running my own box at the house (or a VPS in the cloud) just adds overhead, because I then I have to deal with redundancy and backup myself. My father-in-law is tired of me swapping hard drives out of his gun safe. ;)

[SpiraMirabilis]

It adds additional flexibility. If you are running out of space and don’t want to upgrade your server, you can easily leverage cloud platforms where storage is cheap. If you are running into capacity issues with regards to your servers bandwidth you can segregate highly popular files into a cloud service so they are served without utilizing your servers bandwidth, utilizing a cloud service can ensure important files are backed up offsite based on the files taxonomy, or even backed up using AWS glacier so that it’s impossible to change (for compliance/legal/preservation of document reasons) based on taxonomy.

Hybrid solutions integrating a baseline of owned hardware with a backup of cloud services for either load balancing, backup or reliability is in my opinion a way to very efficiently leverage cloud services very cheaply.

On Thursday, March 15, 2018, jbraes notifications@github.com wrote:

I am always amazed how people use software is such different ways. running your "personal" cloud on-top of "public" cloud infrastructure. Isn't the idea of having your own cloud infrastructure , to be independent of azure, Amazon , google etc ...

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/nextcloud/server/issues/2027#issuecomment-373536747, or mute the thread https://github.com/notifications/unsubscribe-auth/ABf2gOLiAqEqvL8GRrMmxHIB1FoyLCznks5teuSegaJpZM4KqhH3 .

[ohthehugemanatee]

@MorrisJobke Can you give some insight into your decision process here? Nextcloud supports AWS and Openstack out of the box. Why not the number 2 biggest cloud provider?

[MorrisJobke]

@MorrisJobke Can you give some insight into your decision process here? Nextcloud supports AWS and Openstack out of the box. Why not the number 2 biggest cloud provider?

As this sounds like a nice feature, there are currently no plans to implement such a feature. This does not mean we don't want this feature, but it is simply not on our roadmap for the near future. If somebody wants to implement this feature nevertheless we are happy to assist and help out. I hope that explains a bit why it isn't there yet.

If you wish to have this feature implemented by the Nextcloud GmbH there is the option for consulting work on top of your Nextcloud Enterprise subscription to get your features implemented.

[powareverb]

Thanks for the clarification @MorrisJobke is what I figured. You guys only have so much time in the day. I'll be looking into Minio on blob storage, as the s3 backend for Nextcloud seems more stable now. Might be worth looking at for others. https://azuremarketplace.microsoft.com/en-us/marketplace/apps/minio.minioblob

[ohthehugemanatee]

Thanks for the clarification.

For those who are curious, there are ways to mount Azure Blob storage directly on your VM or in k8s. That's what I ended up doing. Honestly if you're going to use it as primary storage, this is a better option than implementing in the application layer IMO.

@see

• BlobFUSE driver howto
• BlobFUSE driver for Kubernetes
• Dysk project "Attach Azure disks in < 1 second. Attach as many as you want. Attach them where ever you want. dysk mounts Azure disks as Linux block devices directly on VMs without dependency on the host."

Hope this unblocks someone else in the future!

[z2z]

Maybe in future, use of https://flysystem.thephpleague.com/docs/ it already has a lot of adapters.

[mstenz]

also instead of BlobFUSE you could potentially map Azure Files with SMB directoy to the host vm. But as you can mount it only with one access rights you need to fix some error.

Please change the permissions to 0770 so that the directory cannot be listed by other users.

f(\OC::$server->getConfig()->getSystemValue('check_data_directory_permissions', true) === false) {

I have not checked further but i think you can change the falue to false in the configuration which will fix this issue, but as i have not yet done it (still in the process) there may be issues afterwards.

[MourIdri]

It seems that it is possible to mount Blob natively via NFS now. Cheaper and easier than object storage (Blob, S3 or other)

[SpiraMirabilis]

Interesting. I wonder how efficient it is with regards to unnecessary metadata requests which Azure charges for. Ideally if you’re using blob you would cache the metadata aggressively for that reason.

On Wed, Jan 8, 2020 at 3:44 AM chapeau notifications@github.com wrote:

[image: AZUREBLOBNFS3] https://user-images.githubusercontent.com/19348103/71972009-1ca82780-320c-11ea-99aa-ab571c21f699.JPG

It seems that it is possible to mount Blob natively via NFS now. Cheaper and easier than object storage (Blob, S3 or other)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/nextcloud/server/issues/2027?email_source=notifications&email_token=AAL7NADARSB62Z2HAHAUHQ3Q4WVATA5CNFSM4CVKCH32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIL6ZZQ#issuecomment-571993318, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAL7NAEVI6GUV7I4BDLQXBLQ4WVATANCNFSM4CVKCH3Q .

[MourIdri]

Interesting. I wonder how efficient it is with regards to unnecessary metadata requests which Azure charges for. Ideally if you’re using blob you would cache the metadata aggressively for that reason. … On Wed, Jan 8, 2020 at 3:44 AM chapeau @.***> wrote: [image: AZUREBLOBNFS3] https://user-images.githubusercontent.com/19348103/71972009-1ca82780-320c-11ea-99aa-ab571c21f699.JPG It seems that it is possible to mount Blob natively via NFS now. Cheaper and easier than object storage (Blob, S3 or other) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2027?email_source=notifications&email_token=AAL7NADARSB62Z2HAHAUHQ3Q4WVATA5CNFSM4CVKCH32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIL6ZZQ#issuecomment-571993318>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAL7NAEVI6GUV7I4BDLQXBLQ4WVATANCNFSM4CVKCH3Q .

It is all about $$$ I guess, I use to have a 28 Millions requests per week on my S3 bucket for around 1000 files. In that case you just get the data and the metadata will be limited to files's GID/UID and embedded in the offering. Let's see .

[ThumbGen]

Did anyone tried the Azure Files NFS v3 as primary storage for NextCloud? Any hints/tips?

[ajdinzutic]

would also be intersted. +1 Azure Blob with features like Soft Delete and a WORM Solution with Azure Storage Explorer, it would be very good product for the clients.

[sudipmandal]

For Anyone who might be interested in getting this working, here are the steps with the Nextcloud docker container, adjust accordingly if you are running on bare metal

My setup: Host : Debian 11, Docker 20.11 Nextcloud : Docker image 23.0.3

On Host:

1. Install Blob fuse As of this writing there is no blobfuse package available in the MS repo for debian 11, do the following to get this installed if using debian 11 (for other distros follow Blob fuse documentation) curl https://packages.microsoft.com/debian/10/prod/pool/main/b/blobfuse/blobfuse-1.4.3-Debian10.0-x86_64.deb --output blobfuse-1.4.3-Debian10.0-x86_64.deb sudo apt-get update sudo apt-get install fuse sudo dpkg -i blobfuse-1.4.3-Debian10.0-x86_64.deb
2. Create a mount folder for your nextcloud "/var/www/html" folder on host say /mnt/nextcloud
3. Create an empty folder say media inside this to be used as volume mounting point in container later, /mnt/nextcloud/media
4. Create a mount folder for your azure blob under this like/mnt/blob
5. Create a cache folder for blob fuse say /mnt/cache
6. Create a connection.cfg file say in /connection.cfg and add the following line to it (without the curly brackets) accountName {Azure Storage Acc Name} accountKey {Azure Storage Access Key} authType Key containerName {Azure Blob Container Name}
7. Run the following command to mount blob to dir sudo blobfuse /mnt/blob --tmp-path=/mnt/cache --config-file=/connection.cfg -o allow_other
8. Add the above command to fstab or a new systemd service to ensure this directory is mounted automatically after server reboots

On Docker: Create the nextcloud container as documented in docker hub, just add a new volume to the above folder as follows docker run ... -v /mnt/nextcloud:/var/www/html -v /mnt/blob:/var/www/html/media ....

For Existing running containers, on portainer duplicate/recreate and add the volumes

Run Container, add the External Storage app to Nextcloud Then mount the local folder /var/www/html/media to Nextcloud as external storage.

Enjoy!

P.S : Also tried to use Azure S3 gateway, Minio Azure Gateway etc, long story short, they dont work with Nextcloud External storage very well, this is the only method which worked for me 100% with good performance.

[brosahay]

I was able to set up and use blobfuse pretty decently, only annoying thing I am unable to figure out yet is the fact that nextcloud goes on telling I am out of space since the blob storage is not reporting the storage space properly. It is showing and calculating the space that is left on the OS drive.

[ThumbGen]

@brosahay Did you find any solution for the wrongly reported size? Does it influence the way Nextcloud works? Does it prevent uploading files to Nextcloud?

[z2z]

@brosahay @ThumbGen report at https://github.com/Azure/azure-storage-fuse/issues

Also jfyi, both sftp and nfsv3 have cost. https://azure.microsoft.com/en-us/pricing/details/storage/blobs/

[brosahay]

@brosahay Did you find any solution for the wrongly reported size? Does it influence the way Nextcloud works? Does it prevent uploading files to Nextcloud?

It does not affect the operations in any way as such. I am working on getting the blobfuse2 working right now.