Azure / azure-webjobs-sdk-extensions

Azure WebJobs SDK Extensions
MIT License
342 stars 206 forks source link

Support Azure Files #14

Open rustd opened 8 years ago

rustd commented 8 years ago

Add a new trigger binding for Azure Files. Here is some more info https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-files/

mathewc commented 7 years ago

We'd want both a trigger as well as input/output bindings.

mliesmons commented 7 years ago

We definitely need this FileTrigger, or a way to mount a drive in a WebJob and use the FileTrigger from WebJob.Extensions. Currently this approach is possible from within an Azure Servcie as it runs as NetworkService, but impossible in a WebJob as it runs as IIS APPPOOL\your_hosting_web_app, which has no network access. Trying to change the identity of the pool using the advanced settings and a applicationHost.xdt with the content shown below does not work either, transform seems to be fully ignored.

This is all really very sad, as such a trigger would make hybrid solutions with customers having no IT personnel trivial: anybody can drop a file in a share...

Hoping to have a solution for this real quick.

`<?xml version="1.0"?>

`
g4mb10r commented 6 years ago

Please do this. This would enable so many lift and ship scenarios.

andersborum commented 6 years ago

+1

fabiocav commented 6 years ago

This work would likely happen after V2 GAs, but this would be a great feature.

/cc @jeffhollan

MisinformedDNA commented 6 years ago

I'd also like to know when a file is deleted. When the researchers remove a file, we need to remove it from the DB.

koolkarni commented 6 years ago

need file-share trigger

jhossy commented 6 years ago

Any updates on this feature? Or is there a plausible workaround while we wait for a feature release?

mathewc commented 6 years ago

No updates - our team doesn't have the time to work on this right now, so the feature request remains in backlog. Note we already have a Files extension that works for the regular file system - this work item is about supporting Azure Files.

MisinformedDNA commented 6 years ago

@mathewc Are you referring to external files, which only work in V1 and is marked as experimental?

MisinformedDNA commented 6 years ago

I created a library that monitors changes in the local file system or on a network drive. Since Azure File Share supports SMB, maybe it can be applied here? You would just set up a recurring job to get the latest changes and then either immediately process them or send them to an Event Grid topic to act as the trigger.

I haven't had time to play with this case, but I think it would work. If anyone wants to give it a go, let me know if it works. Otherwise, I'll try to flesh out the details.

UPDATE: The only way my code would work is if we span up a VM just to host the SMB connection. That's a non-starter for me.

evictor commented 5 years ago

@MisinformedDNA the only missing piece I see with that is you don't know who changed the file.

For audit logging/HIPAA compliance I would like events for Read/Write/Delete. I submitted a feedback item here: https://feedback.azure.com/forums/217298-storage/suggestions/35896822-events-for-azure-files

a-patel commented 5 years ago

Any update on this?

silverl commented 5 years ago

@fabiocav and @jeffhollan - Any hope here? Thanks.

guillaume-kizilian commented 5 years ago

Up?

GFoley83 commented 5 years ago

Nearly 4 years on. Any update whatsoever on this guys?

jeffhollan commented 5 years ago

The path forward to adding new trigger support for Azure Functions is having the service integrate with Azure Event Grid. We have made this request to the Azure Storage team and continue to do so, but the big item here that would enable this would be Azure Files supporting Azure Event Grid for 1st party events around file operations.

glittle commented 5 years ago

Functions can trigger on Blob storage events (https://docs.microsoft.com/en-us/azure/azure-functions/functions-triggers-bindings) and many other bindings. How long until Storage Account/Files can be supported?

MisinformedDNA commented 5 years ago

How can we get the Storage team involved in this discussion? Or is there a better place to have this discussion?

glittle commented 5 years ago

@MisinformedDNA I just added a forum question here: [https://social.msdn.microsoft.com/Forums/azure/en-US/ed7fed62-fcc5-42b1-bae3-0f2850c4cb5e/add-a-trigger-binding-for-azure-files?forum=AzureFunctions](MSDN Forums)

MisinformedDNA commented 5 years ago

@glittle Probably the wrong forum as you chose Azure Functions instead of Azure Storage. (Also, your link is reversed.)

glittle commented 5 years ago

Triggers are defined in Functions, so wouldn't that be where to ask?

MisinformedDNA commented 5 years ago

At first glance, you might think so, but as @jeffhollan has said above

The path forward to adding new trigger support for Azure Functions is having the service integrate with Azure Event Grid. We have made this request to the Azure Storage team and continue to do so, but the big item here that would enable this would be Azure Files supporting Azure Event Grid for 1st party events around file operations.

Basically, Azure Functions doesn't want to create triggers, they want to work off of triggers. And, really, that's how it should be. We don't need Azure Functions polling our file share like crazy when the Storage team can just know when something happens and raise an event.

glittle commented 5 years ago

Will that do away with Function triggers?

Since Blobs are files, I may be able to do what I need using Blob Storage triggers.

MisinformedDNA commented 5 years ago

Blobs work fine.

ltdavids commented 5 years ago

The issue I have with using Blobs is the warning here: MS Warning In addition, storage logs are created on a "best effort" basis. There's no guarantee that all events are captured. Under some conditions, logs may be missed.

mathewc commented 5 years ago

BlobTrigger currently employs a hybrid strategy - it uses blob log scan to get notified of changes, but it also does a background receipt based scan of the actual blobs as well. So even if a blob log is missed, the blob will eventually get processed.

GFoley83 commented 5 years ago

For those of us who work in the IT services industry, using a service that has no guarantee of working is a deal-breaker for SLAs.

ak09 commented 4 years ago

Any update here?

JoePyx commented 4 years ago

I too would appreciate this feature.

pogonica commented 4 years ago

any update?

aventius-software commented 4 years ago

We could really do with this, any update?

DavidDeeds commented 4 years ago

Another here that would like this feature in Functions. Why? Because you cant mount blob storage in Non-Linux Web Apps.

andrewCluey commented 4 years ago

This would be really useful. End users don't like using Blobs (trying to teach 'Ernie' from Accounts to use Storage explorer is a non-starter when he's been accessing files from an 'S:' Drive for 25 years).

Having input & output for Storage Account File Shares would be really useful.

It's been 5 years since the original post though, can't see it'll get picked up any time soon..

knurdnerd commented 3 years ago

I have some use cases for this as well; please finally support this.

e14mattc commented 3 years ago

Looks like we're closer to being able to do this.. I see they have Azure Storage Logs as a preview feature, which is along the right lines for being able to trigger something from changes in the File Share.

https://docs.microsoft.com/en-us/answers/questions/153703/azure-trigger-function-for-file-share.html

Shame it isn't generally available yet.. could really do with this for a solution we're trying to put together. Sigh.

dariofinocchiaro2 commented 3 years ago

Any updates on the matter??

kjartanvalur commented 3 years ago

Any updates?

Abhishekqwerty commented 3 years ago

Please implement this feature. It will be really useful

cash1981 commented 3 years ago

We would also like to see this feature

gautamsamal commented 3 years ago

@MisinformedDNA For the time being, does it make sense to poll the file share and find out changes based on LastModifiedAt and Length? We also need to have an active mapping or table somewhere to compare the changes in the share - add, delete, modified.

MisinformedDNA commented 3 years ago

@gautamsamal A SHA is probably best, but I just use LastModifiedAt. Length can be misleading if the content changes but the length stays the same.

paulmupeters commented 2 years ago

Any updates?

MisinformedDNA commented 2 years ago

Has anyone tried this? https://docs.microsoft.com/en-us/azure/storage/files/storage-files-monitoring?tabs=azure-portal

chrfrenning commented 2 years ago

Adding myself in here, this feature would make a world of difference for so many!!!

HugoMcD101 commented 2 years ago

Jumping onto the thread, hoping for implementation!

prolog commented 2 years ago

This would make a huge difference. Hoping it can be implemented.

petersgiles commented 2 years ago

Please implement this, it would make a huge difference to my solutions.

mkp-bartsch commented 2 years ago

Hoping for implementation too! We need it urgently.

gautamsamal commented 2 years ago

Has anyone tried this? https://docs.microsoft.com/en-us/azure/storage/files/storage-files-monitoring?tabs=azure-portal

I am able to work out an alternative solution for now with the diagnostics logs. I am pushing it to an event hub and then filtering out the write events

fileEvent.OperationName == "Write" || fileEvent.OperationName == "PutRange"

Needed to add also a data layer to compare old vs new files and as well as a check for open file handles. So far it seems to be working.