Message limit/message expiration - message retention policy

[fjorgemota]

Hello!

I'm researching chat tools that are opensource and Rocket.Chat seems like a great possibility.

However, one thing that I am searching is someway to limit the number of messages posted in the service and/or expiring old messages. In the RocketChat's documentation, I found in Schema Definition the field 'expireAt': https://github.com/RocketChat/Rocket.Chat/wiki/Schema-Definition, but, when installing the version 0.5.0 here, it seems to not create that, and there's no more references to message expiration in RocketChat's platform.

Is there a hidden setting to expire messages? Or it's just a plan for now? And, yet, it's there a plan for message limit too? I didn't found anything for that in RocketChat documentation.

Thanks. And sorry for my bad english. :v:

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[geekgonecrazy]

Curious what is the application for this? This is one of the reasons why a lot of people move away from a lot of other services..

[fjorgemota]

Just to avoid excessive use of storage space. =)

I have a "team" of colleagues on the university where I study and we discuss a lot about the various disciplines of our course and about a lot of other nerd topics and these type of things.

Up to now, we're using Hipchat's service, but, as there a problems that cause downtime in the last days/months, we are searching to use other services. We thought about migrating to Slack, but we really don't want to depend too much about third party services that can have uptime problems as Hipchat is having recently, just because we generally talk in periods that do not include normal work times (like weekends, for example). Whatsapp isn't a possibilty because not everyone in our group have a smartphone (as we're just students), and Telegram, well, can suffer from downtime as well as we can't install a server for it.

So, that's my use case. Seems like a joke, but it's not, and for our study the use of a chat service improves resolutions of problems, allow us to study together to exams and many other things. (we have a group at facebook too, but it's virtually abandoned after we started using a chat service like Hipchat)

As MongoDB have support to expiring documents automatically, and RocketChat seems to have a field just for that - at least in the documentation, I thought that RocketChat have that support, and I don't found any other issues about that. (which seems to confirm your comment about this being a reason why a lot of people move away from a lot of other services)

Thanks.

What you think about?

PS: I thought about capped collections. But, I don't know about the possible behavior of Meteor/RocketChat with it.

PS 2: Note that, because we're students, we really don't need all the messages. At maximum the messages of a semester or two is sufficient for us. (and it probably will be cheaper to host, too)

[geekgonecrazy]

Space wise your biggest limitation is going to be file uploads. I've hosted forums and stuff in the past that had many thousands of lines of text and was only a few megs. Infact I think I saw a slack export of 10k messages that was under 1Mb.

That aside. I don't think its necessarily a bad idea. What about a tool that could prune messages older then a certain date? Lets you pick the date and prunes everything older then that?

Expiring is obviously an option. Just seems a bit scary to give that much power to the database. Also we would have to switch that functionality on and off for the db when the user changed. Idk how much of an issue that would or wouldn't be.

[fjorgemota]

Yep, maybe the messages will not ocuppy so much storage, my main concern is about some texts and longer codes that we post on the chat, that have a bit of frequency..I don't have a idea, however, of how much storage space that occupies

A tool to prune messages is a good option too. I think initially about 'expiring' or 'limiting', but I think that a tool with that utility is good too. (specially if it delete the attachments of the messages too)

Nice idea. =)

[fjorgemota]

Well.

I exported the messages of the group and we that generated a folder with 21MB of chat (in JSON format). Considering that Hipchat's probably cropped that export at the limit of 25k messages, and our biggest room have 81k messages, well...It's a bit. :P (and that without considering attachments..)

[geekgonecrazy]

@fjorgemota so it could be big :) I can see where small teams after a while would need some way of pruning down their disk footprint.

[Sing-Li]

:+1: Great idea! It will make Rocket.Chat adaptable to many more installation scenarios (fixed sized SD card on a Raspberry Pi 2 - for example).

In the longer term, with two classes of users in mind:

• those who want to host Rocket.Chat for the masses
• enterprise and corporate users faced with compliance and audit

Message retention and per-user-quota will be completely driven by policies. Policies will require a 'roles and permissions' system being in place (there is a ticket for that somewhere). Basically, a policy may say - users in a student role will have only 5 MB storage quota, and their message retention is 3 weeks. The system will then enforce the policy.

Meanwhile, any implementation of pruning or per-user capping of storage compatible with the above will add value immediately.

[rodrigok]

O liked the idea, we can do this in 2 steps

1. General quantity/size limit or date expiration (we can do this direct in MongoDB)
2. Limits for quantity or time based on user's role (depends on role implementation and needs to be implemented in the application layer) and we can't limit by size.

The first step is relatively easy to do.

[AdrianoCahete]

enterprise and corporate users faced with compliance and audit

For audit, we need that ALL the operations are logged. From "giving permissions" to "leave channel".

[Sing-Li]

:grinning: @AdrianoCahete I'd say you're about a year ahead of your time.

[AdrianoCahete]

@Sing-Li Just habit.... I work in company who developing a system for Governance, Risks and Compliance. :P

[ma-karai]

Maybe Indicate the History the following way.

In this case every message before Saturday, November 8, 2014 will be deleted

[dandv]

Perhaps a more stringent use case than space limitations, is message retention policy from legal departments. In some large companies, chat messages must not be stored beyond X days, to limit the risk of exposure in case of subpoenas/legal discovery processes.

[freddielam]

O liked the idea, we can do this in 2 steps

1) General quantity/size limit or date expiration (we can do this direct in MongoDB)

was looking for how to do this,

so while waiting for their feature to be done here's how it can be done.

from mongo shell

use rocketchat
db.rocketchat_message.createIndex( { "ts": 1 }, { expireAfterSeconds: 3600 } )

this will expire any message after 1 hour, even existing messages.

[geekgonecrazy]

@freddielam this is for sure an easy way of accomplishing this. 😄

[ghost]

@freddielam is it possible to only expire the oembed messages?

[freddielam]

I would suppose so. I haven't looked into the system since.

[m-ephraim]

As a dev in a corporate setting evaluating RocketChat, I can say this feature would be make or break for our adoption of RocketChat. Our company must be able to define a message retention policy per channel. This feature would enable RocketChat to be used in our corporate environment, which I am pushing for.

[geekgonecrazy]

@dggtydnk feel free to reach out to us if you're interested in driving this feature forward. Also we welcome Pull Requests and would gladly work with you or anyone else in getting it merged 😄

[djha-skin]

What would need to be done to add this feature? I'm interested in submitting a pull request if it's not too large a task :)

[timmc]

Additional use-case, along the same lines as the legal/corporate one: Activist groups or people talking about sensitive issues often want auto-expiring messages/retention policies.

[freddielam]

This product is not designed for activist use from ground up. I dont think it's a good idea to recommend it. Minimally, it must have some form of forward secrecy such as the Signal Protocol that must not be disabled. In addition, there should be more pen testing to see where it might break. In this current climate, a such a recommendation just scares me.

[flantascience]

I'd be interested in having an expiration date on file uploads as well. Maybe messages too... but definitely file uploads. Is anything brewing with this possibility?

[localguru]

Is there anything going on in this direction?

@geekgonecrazy is the above workaround on database level "secure" or could that cause inconsistency, e.g. files attached to a post left in database?

use rocketchat
db.rocketchat_message.createIndex( { "ts": 1 }, { expireAfterSeconds: 3600 } )

[geekgonecrazy]

@localguru I think it'd leave orphaned files :( if using gridfs you could likely add the index there as well. If you are using a object store on s3 / file system or others.. It'd leave the file there.

[maxpen]

Yes again! lol

[hartois]

I'm planning on using a Rocket.Chat in our company. And I'm also very interested in the space limitation for the user. I'm afraid that users in a short time using attachments will quickly deplete the available space. Is there any news on this issue?

[marzie82]

Are there any news regarding space limitation? We would be very interested in such function! Of course it's great if you don't need to delete old Messages but after some years it's a different situation. Thnanks!!!

[docsncode]

Closing as this feature is not on our short term roadmap, we will revisit once other priorities have been addressed.

[engelgabriel]

Done via https://github.com/RocketChat/Rocket.Chat/pull/11236