Can I subscribe to FTP events when files are added or changed in a folder?

[robinrodricks]

Question:

Ref: https://github.com/robinrodricks/FluentFTP/issues/1660

I would like to ask wether it's possible to subscribe on ftp command events via FluentFTP, because I would like to se wether it's possible to make a solution where I don't have to pull a ftp every 5 seconds.

When files are being added to a particular folder I would be notified.

That way I don't have to pull and check ftp for new files every 5 seconds (not super great for ftp) - I will only pull, when I get notified.

Proposed solution:

This is a very common usecase especially on the cloud, where FTP uploads are used to trigger cloud workflows.

I suppose we could consider creating a new class for this. Something like FtpFolderMonitor. This class would then allow you to monitor a specific remote folder on the FTP server. It would trigger events when files were added/removed. Internally it would poll the folder and check for new files (this is the only way technically possibly via the FTP protocol).

Config:

• Config - the FTP client config (FTP client will be internally created automatically and kept alive based on this config)
• FolderPath - which folder to monitor
• PollInterval - how often to check the folder - default : 1 sec
• WaitTillFileFullyUploaded - flag to enable detection of partially uploaded files (this works by monitoring the filesize and only firing an event when the filesize is stable)

Events:

• FilesChanged - if the date/filesize of any files changed, this event is fired with the list of changed files
• FilesAdded - when new files are added, this event is fired with the list of added files
• FilesDeleted - when files are removed, this event is fired with the list of deleted files
• ChangeDetected - a generic event fired when anything changes

In event handlers, file lists are provided as List<string> or List<FtpListItem>.

[FanDjango]

Maybe I'm too stoopid, but

are we talking about monitoring for local folder changes triggering an event

or

are we talking about monitoring (either with a permanent connection or regular repeated connections) remote folder changes triggering an event

???

[markat1]

I like @robinrodricks idea of having a wrapper class FtpFolderMonitor that would allow monitoring a specific folder!

Would it be possible to pub/sub on FtpFolderMonitor?

[robinrodricks]

are we talking about monitoring (either with a permanent connection or regular repeated connections) remote folder changes triggering an event

@FanDjango monitoring of a remote folder.

Would it be possible to pub/sub on FtpFolderMonitor?

how would pub/sub be implemented?

[FanDjango]

monitoring of a remote folder

So you would propose to do a MLST, LIST or NLST regularly? And compare the contents?

PollInterval - how often to check the folder - default : 1 sec

You will be kicked from the server. Either because you don't ever transfer a file and time out (some servers enforce this), or if you try to circumvent this by QUIT and reconnect, you will be noticed and banned.

In any case you are severely misusing the FTP server, you might get away with it if you reduce the timing to, say every 5 minutes.

Unless of course it is your own server, then you are free to do this.

Otherwise, this is a job for NFS, RSYNC or other stuff higher up the tree.

how would pub/sub be implemented?

I don't even know what that is. Enlighten me, please, someone.

[robinrodricks]

So you would propose to do a MLST, LIST or NLST regularly? And compare the contents?

Yes.

You will be kicked from the server.

Then we can keep the default to 10 seconds or 60 secs.

Unless of course it is your own server

Yes this would be the most common use case. Or even if its a web host, they should allow polling every 1 min.

I don't even know that that is. Enlighten me, please, someone.

Its normally used on the cloud. I'm not sure why the user is asking us to implement it at the library level.

https://aws.amazon.com/what-is/pub-sub-messaging/

[markat1]

Reason I ask for pub/sub is some sort of control of broadcasting certain events to specific subscribers. It was just the architecture that came to mind.

Anyway, I guess pub/sub doesn't have to be part of this solution - it could be integrated seperately, which problably makes more sense.

[robinrodricks]

Reason I ask for pub/sub is some sort of control of broadcasting certain events to specific subscribers

You can surely handle that as part of your user code.

[robinrodricks]

@markat1 I have committed the first version of these classes. Can you download the FluentFTP project, build from source and try using it?

New classes:

• AsyncFtpFolderMonitor - async version
• FtpFolderMonitor - sync version

Usage:

var client = new FtpClient(...);
var monitor = new FtpFolderMonitor(client, "/remote/folder");

// optional config
monitor.WaitTillFileFullyUploaded = false;
monitor.Recursive = false;
monitor.PollInterval = 60;

// add events
monitor.FilesAdded += your_event;
monitor.FilesDeleted += your_event;
monitor.FilesChanged += your_event;
monitor.Start();

Since I'm very busy I am hoping you can fix minor bugs and submit a PR with a working version. I don't have time to test it on my end right now.

I have used System.Threading.Timer. I am not sure if it works in all use cases. Please change this to whatever timer is best suited to such projects.

File change detection is done by filesize. I suppose an alternate way would be using the date modified.

[markat1]

I testet following in a console program - I have replaced connection information in the examples shown here.

both Sync and async version - doesn't respond back on filesAdded, FilesDeleted or FilesdDeleted unfortunately

using FluentFTP;
using FluentFTP.Monitors;

var client = new FtpClient("Server", "User", "Password");

client.Connect();

Console.WriteLine("Running FtpFolderMonitor");

var monitor = new FtpFolderMonitor(client, "OutboxDirectory");

monitor.WaitTillFileFullyUploaded = false;
monitor.Recursive = false;
monitor.PollInterval = 5;

monitor.FilesAdded += Monitor_FilesAdded;
monitor.FilesChanged += Monitor_FilesChanged;
monitor.FilesDeleted += Monitor_FilesDeleted;

void Monitor_FilesAdded(object? sender, List<string> e) {
    Console.WriteLine("File added");
}

void Monitor_FilesChanged(object? sender, List<string> e) {
    Console.WriteLine("File changed");
}
void Monitor_FilesDeleted(object? sender, List<string> e) {
    Console.WriteLine("File removed");
}

Console.ReadLine();

using FluentFTP;
using FluentFTP.Monitors;

var asyncclient = new AsyncFtpClient("Server", "User", "Password");

client.Connect();

Console.WriteLine("Running FtpFolderMonitor");

var monitor = new AsyncFtpFolderMonitor(asyncclient,"OutboxDirectory");

monitor.WaitTillFileFullyUploaded = false;
monitor.Recursive = false;
monitor.PollInterval = 5;

monitor.FilesAdded += Monitor_FilesAdded;
monitor.FilesChanged += Monitor_FilesChanged;
monitor.FilesDeleted += Monitor_FilesDeleted;

void Monitor_FilesAdded(object? sender, List<string> e) {
    Console.WriteLine("File added");
}

void Monitor_FilesChanged(object? sender, List<string> e) {
    Console.WriteLine("File changed");
}
void Monitor_FilesDeleted(object? sender, List<string> e) {
    Console.WriteLine("File removed");
}

Console.ReadLine();

[robinrodricks]

You have missed to start the monitor.

monitor.Start();

Just test one version (sync is fine).

The class is easy to understand. You can add breakpoints into the PollFolder method and see what's going on.

Since I'm very busy I am hoping you can fix minor bugs and submit a PR with a working version. I don't have time to test it on my end right now.

[markat1]

I played around with it for a couple of hours - does work fine - maybe a small change would be to put restart time in the try catch finally - just to make sure it's always running

/// <summary>
/// Polls the FTP folder for changes
/// </summary>
private async void PollFolder(object state) {
    try {

        // exit if not connected
        if (!_ftpClient.IsConnected) {
            return;
        }

        // stop the timer
        StopTimer();

        // Step 1: Get the current listing
        var currentListing = await GetCurrentListing();

        // Step 2: Handle unstable files if WaitTillFileFullyUploaded is true
        if (WaitTillFileFullyUploaded) {
            currentListing = HandleUnstableFiles(currentListing);
        }

        // Step 3: Compare current listing to last listing
        var filesAdded = new List<string>();
        var filesChanged = new List<string>();
        var filesDeleted = new List<string>();

        foreach (var file in currentListing) {
            if (!_lastListing.TryGetValue(file.Key, out long lastSize)) {
                filesAdded.Add(file.Key);
            }
            else if (lastSize != file.Value) {
                filesChanged.Add(file.Key);
            }
        }

        filesDeleted = _lastListing.Keys.Except(currentListing.Keys).ToList();

        // Trigger events
        if (filesAdded.Count > 0) FilesAdded?.Invoke(this, filesAdded);
        if (filesChanged.Count > 0) FilesChanged?.Invoke(this, filesChanged);
        if (filesDeleted.Count > 0) FilesDeleted?.Invoke(this, filesDeleted);

        if (filesAdded.Count > 0 || filesChanged.Count > 0 || filesDeleted.Count > 0) {
            ChangeDetected?.Invoke(this, EventArgs.Empty);
        }

        // Step 4: Update last listing
        _lastListing = currentListing;
    }
    catch (Exception ex) {
        // Log the exception or handle it as needed
        Console.WriteLine($"Error polling FTP folder: {ex.Message}");
    }
    finally {
        // restart the timer
        StartTimer(PollFolder);
    }

}

[FanDjango]

Why don't you create a PR from this, then it can be an official change? I'll merge it then.

[markat1]

hmm don't think I'm allowed to make a pull request - anyway it's just a small change :)

[FanDjango]

Never mind, I'll take care of it...

[FanDjango]

I merged it.