search

ucphhpc / docker-migrid

Containerized MiG
GNU General Public License v2.0
3 stars 7 forks source link

Logging up and download of files via SFTP/FTPS #8

Closed Bjarke42 closed 12 months ago

Bjarke42 commented 1 year ago

We are using MiG-sftp-subsys for ssh access. I cannot see what users are up and downloading. I have searched the logs but it seems that its not being logged.

I would like it to be as with webdav and web page that I can see what user, by email, data size, and file name they are up and downloading.

It also has to be logged outside the container so that we can retain the file in case of docker container restart. Also we would like to use log parsing so we can build statistics from the activity on ERDA.

Bjarke42 commented 1 year ago

I would like to add that this might be required for us to use MiG for SIF.

jonasbardino commented 1 year ago

Thanks for reporting your findings - we will look into the logged operations for sftpsubsys. If anything is missing for particular operations like uploads/downloads it's actually in the core migrid code repo, but the log persistence belongs here :)

GDPR- and sensitive data sites like SIF specifically rely on the grid_sftp.py service rather than sshd+subsys to deliver SFTP. Everything gets logged in more detail in that setup in line with GDPR requirements.

jonasbardino commented 12 months ago

The logs from /var/log/ of the sftp-container are generally exposed in log/migrid-sftp/ on the host. So you'll find e.g. the usual openssh access logs in the secure log file there and the specific sftpsubsystem and/or native sftp operation logs are similarly exposed in state/log/.

Logging every file access is extremely resource demanding for high throughput sites like our ERDA site, so we do not currently do that in general, but only for sensitive data setups (enable_gdp) like our SIF system where security comes over performance. With some effort it would be possible to implement complete access logging in migrid I/O daemons and and offer it as a configuration option. Feel free to create an issue on migrid-sync about such a feature, but it's unlikely to appear in any near future.

Adjustments or even a complete overhaul of the logging infrastructure in the containers are of course possible, but it's not essential for the use and not something we can prioritize.