Discussion on the SFTP config

[antoniocorreia]

As @GorkiGonzalez sent we made two options to "input" the info about which directory would be linked to what directory https://github.com/TORCH-TCN/torch_hub/tree/feature/one-project-webapp @lauraferraz5

• steps to check the UI out

access the src folder and run python torch.py (for testing purposes we left only the web app startup there) there's a public (for now) page you can access on the web app /workflows/settings where you can register the workflow id and the directory you want to monitor

• steps to check the CLI

access the src/torch/config/watchdog folder and run in the terminal python watchdog_cli.py -h to check the available args and the watchdog_cli.py is the script responsible for registering the monitors, the idea here is after the integration work is done pull the info from the databases (prefect and ours) and then iterate to check the entries from the database to its respective workflows or the entry given through the command line. Another important file to check is the HubHandler.py in the same folder.

Question

@jbest does this config solve the case where the old users are used to uploading the files through SFTP? we were wondering if this folder is something simple inside the web app itself for example or is this something that could be in a different SFTP server and we need to set up and be prepared for this with the watchdog config? (you can check how we are setting up this directory in the watchdog_cli.py and HubHandler.py files)

[jbest]

@antoniocorreia From what I see, this should address the need to upload via SFTP. My assumption is that the Hub server (Prefect + Flask etc) will be running on/have access to the same filesystem that the SFTP server is using for uploads.

[antoniocorreia]

@jbest ok, we can cover that scenario, I guess we can do this by polling the server, but I'll check the options and confirm if watchdog has a solution for it.

[jbest]

@antoniocorreia I want to make sure we're on the same page about this so I'll make up an example with entirely made up directory structure. I'm assuming that the watchdog module imported in the Hub code (Prefect, Flask, etc) will have direct file system access to the files that are uploaded via SFTP. So the hub task workflow and web service might be at hub_server:/web/torch_hub/ and the SFTP server also running on the same hub_server and uploads are placed in hub_server:/data/uploads/ which is a directory monitored by the watchdog module in the workflow. If there is a desire/need to have SFTP running on a different server, please let me know so we can discuss this in more detail.

[antoniocorreia]

@jbest perfect, thank you, and yes, this was exactly the scenario I had in mind, watchdog having direct access to it, by any chance you can validate it? The watchdog modules aren't running any specific workflow tasks yet, but they are logging some info to validate file access.