redlink-gmbh / smarti

Smarti is a service component for conversational data.
http://dev.redlink.io/smarti
Apache License 2.0
6 stars 4 forks source link

Limit Messages per Conversation #281

Closed westei closed 5 years ago

westei commented 6 years ago

The current storage model of conversations require to limit the number of messages per conversation to ensure respecting the document size limit of MongoDB of 16 megabytes.

Assuming ~200 chars average per message and ~200 chars required for additional information a limit of 5000 messages seams to be a reasonable limit.

westei commented 6 years ago

Just as explanation:

The Smarti conversation was meant to represent a section within a communication channel. This lead to the decision to store the messages with the conversation.

However when integrations do establish a 1:1 mapping between communication channels and Smarti conversations their is a risk that the amount of messages in those channels do overrun the storage limit of MongoDB. To avoid this situations this issue will limit the number of messages/conversation and only store the 5000 most recent messages in Smarti

Depending on the goals future versions might change the storage model to directly represent Channels and their messages. With messages stored as single documents in an own collections. Conversations would than be an additional class that just refer their the channel and the messages they contain

westei commented 6 years ago

The number of messages per conversation is configureable via smarti.storage.mongodb.maxConvMsg. The default is set to 5000

westei commented 5 years ago

NOTE: This feature is also useful to avoid the memory issue while indexing public channels with many messages (see #299 and #302)