[YABI-525] Storage space issues with the lucien Backend

[ccgbuildbot]

[reporter="jmccooke", created="Tue, 12 Aug 2014 13:09:40 +0800", resolved="Wed, 29 Oct 2014 11:17:29 +0800"]

There is currently an intermittent problem with storage issues on lucien. One of the backends of the ccg.murdoch instance of YABI.

This is primarily due to an issue of clients/YABI users not removing their data, in association with the increased number of clients.

As a feature to help ease the issue, is it possible for the automatic removal of data on lucien following 2-3 months of the data being generated. That is when a new job is run, is the application able to check time stamps and remove data if it is older than 2months before staging out the data. Hopefully this will encourage users to move there data on.

For users that are no longer using the system. Is it possible for yabi to track user usage and delete the files of users on a backend that have not log on for 3 months.

[ccgbuildbot]

[author="tszabo", created="Wed, 27 Aug 2014 13:25:52 +0800"]

John, to continue our discussion on this one, I don't feel very comfortable with adding features to Yabi that automatically delete user's data.
I am happy to add some reporting to Yabi that makes it easier to detect when people aren't using Yabi anymore etc.

As a first step I've just added a last login field to Yabi users in Admin. You can sort by that field so it can be used to identify people who haven't been using the system.
Will that help?

Any other things we can make to help with this issue?

[ccgbuildbot]

[author="jmccooke", created="Wed, 27 Aug 2014 13:52:51 +0800"]

Thanks Tamas,

Fair enough on the automatic removal of data, Notice that emails can be captured in the users accounts. What about a friendly automated email to "Flagged" non current users to please remove or tidy up their memory footprint.

[ccgbuildbot]

[author="tszabo", created="Wed, 27 Aug 2014 19:07:23 +0800"]

I think that is harder to do. Some problems with it:

• what do we consider Yabi data that it is appropriate to warn the users about? The Backend Credentials can point to a users home directory or any other directory on any servers. We can't just warn them to clean up all data. It would make sense to warn them just about servers we administer, like lucien for example. But from Yabi's perspective there isn't a difference between lucien or let's say an S3 bucket the users pay for themselves. Warning them to delete data from the later isn't appropriate IMO.
• another slightly related problem is permissions. If the users haven't logged in for a long time, we won't have access to their decrypted credentials and therefore Yabi can't inspect their FS Backends to see how much data they are using.
• even if we could list the dirs (with some special admin credential let's say) we kind of have to guess what is Yabi data (ie. data loaded up by the file browser or another sftp client) and what is other user data.

Your thoughts?

[ccgbuildbot]

[author="tszabo", created="Wed, 29 Oct 2014 11:17:29 +0800"]

Closing this for now.

We do display the last login of Yabi users in Admin.

As for rest I think it is a sysadmin problem and I don't like solutions when Yabi does delete user data. All we could do is help in identifying the problem, but there are some open questions in comments on how to do that.

Please feel free to re-open to discuss more if needed at any time.