Add a option to delete old messages from the server

[webratte]

If someone use a free mailer like GMX only for Delta Chat he/she never will check if the storage limit is reached.

If messages will deleted after a time (as userpreference) there should be no problems.

[r10s]

related to deltachat/deltachat-core#244

[webratte]

I'm not sure if I was clear enough. I mean "delete from Server".

[r10s]

ahh, i see :) in fact, this is another request. thanks for clarification :)

[testbird]

all aspects are gathered at: https://github.com/deltachat/deltachat-core/issues/120

[webratte]

Thank you, your right.

I didn't find it because I didn't understood the title of this issue.

[testbird]

Good point, I added specifics to the title.

[r10s]

just got some mails from a cubanian user who has only 50 mb available in his inbox.

EDIT: this is the used provider: http://www.etecsa.cu/telefonia_movil/movil_nauta/

however, while Delta Chat could mitigate this by deleting old messages, i am not convinced that this is the best solution, it's feels more like a workaround.

[testbird]

If the IMAP mailbox can only hold 50MB, deltachat has to delete messages on the server when reaching a threshold. How can that fact be a workaround?

With the actionable threshold limit of "5% or 100MB", as proposed in deltachat/deltachat-core#120, there would be no messages kept on a server with only 50MB user storage, which makes sense and is necessary to allow for buffering the reception of a couple of incoming emails that may include attachments.

[r10s]

How can that fact be a workaround?

i think it would be more up to the server to delete old messages, not to one of a bunch of clients. imagine eg. a multi-client-setup - if all clients delete messages it is at least a little confusing. also - what is with messages that are no delta-chat-messages - should they be deleted as well? if so, it might be unexpected - if not, the whole system may not work.

otoh i know that these servers typically have no option to prune old messages :)

[testbird]

I think deleting should only be done by the user, or an email client that is made or defaults to be responsible for specific messages, e.g. like deltachat for chat messages. Only the client can make informed choices, like downloading or archiving old messages first.

If it can't be avoided, mulit-client support might later require deltachat to have a special status message in the email chat folder on the server, e.g. for better coordination of such maintenance work.

Edit: But generally I think old chat messages can be assumed to have been downloaded, and if several clients try to delete some old messages from the server, even different ones (until a threshhold is reached) there should not be a problem. After they are deleted, all clients will sync to the same state again (server state taking precedence).

[testbird]

moved to deltachat/deltachat-core#120

[ltsilva911003]

It's very simple, just do not save the messages sent on the server, leave them local, but not on the server, it's all that is needed to solve the problem

[ltsilva911003]

it does not matter if they are in the application for three months or a year, what worries me is the server, simply not keeping them in the server would be the perfect solution, which is clearly optional, if someone needs them in the server, they stay, but if you have my problem with such a small tray, it's best not to save them on the server

[r10s]

it's best not to save them on the server

i think it is also about incoming messages. also some providers save the messages on the imap-server on smtp automatically (eg. gmail).

[ltsilva911003]

I do not care about incoming, obviously those if they go to the server, but I do not have to save the sent ones too, that incurs data expenditure and slows down the application

[ltsilva911003]

generates much more traffic

[ltsilva911003]

apart I think that if the ones sent on the server are not saved, the application would be much faster and would cost much less data, maybe more than one problem will be solved

[r10s]

well, saving a copy on the server is also needed for multi-device. we could save the additional traffic maybe by adding self as a receiver when doing smtp.

however, the initial issue is about cleaning up the server because of storage limits. not saving outgoing messages will still leave the incoming messages on the imap-server consume quota - so, while this may be an option, it probably won't fix the initial issue.

[ltsilva911003]

you implemented a folder that is created on the server called DeltaChat, that's why I think that if you keep the senties there, it is likely that they may not also save them. a comparison, when I send messages from my native android email application, these do not fit on the server

[ltsilva911003]

things that I have analyzed of the application, the messages can be deleted, for that reason I think that there is a pop configuration, that is the protocol that allows to delete in the server, and correct me if I am wrong. You can use that configuration when sending messages, and they would not stay on the server

[r10s]

sorry, i do not understand the last two posts.

[ltsilva911003]

in order that it is very expensive for me to save the messages sent on the server, and it should not be just my problem, apart from limiting the use of the application, because when I send several photos and consume the 50MB the application stops work until I delete all sent and received from the server, but it would work for a while longer, if only those received are stored, do you understand? If this problem is solved, I will be ready to use the application again

[r10s]

but it would work for a while longer, if only those received are stored

true :)

[ltsilva911003]

a question that I should have asked from the first day, is there any possibility that the messages sent by the application do not remain on my server in the DeltaChat folder or in any other?

[r10s]

by default, they remain on the server. this is the current state.

[ltsilva911003]

and that can be changed?

[r10s]

well, at least the discussion has started. and, of course, someone has to do the changes and different issues and proposals need to be in sync with the limited resources :)

[webratte]

Another idea about this:

Every "normal" messenger stores messages only on the device. Messages will deleted as soon it's stored on the device.

DeltaChat could do the same to keep the mailbox clear and save storage.

And to keep it multi device capable (and as backup) you can add an option "keep messages stored on server".

On first start you can ask for that.

[testbird]

Every "normal" messenger stores messages only on the device.

Not really, almost all have at least a simple store-and-forward server, so all devices can pick up messages after having been offline.

On first start you can ask for that.

Why, if the required maintenance functions can work properly out of the box (even in the most basic autodelete-only implemention stage without any configuration UI, see https://github.com/deltachat/deltachat-core/issues/120#issuecomment-417267057)?

• say keeping the last X messages in chats untouched, but set all further to the "archived message" state, if the field is not set to "stared".
• check free local and server space on incoming messages, and start (warning and) auto-deleting the oldest "archived" messages, if required

[csb0730]

In general, I agree with webratte's opinion at least a (big) portion. It is a use case to delete mails at server immediately after downloading (incoming mails). From the programmer point of view this is an easy to implement option. Much more complicated are options to delete later or at a certain time. In my opinion a low hanging fruit :-)

Another option could be: not to store messages sent at server.

All this affect's multi device support, but is maybe important or even better for single device usage.

[webratte]

Thank you @csb0730 👍

@testbird

It seems you misunderstand me. If I say "ask on first start" I mean:

Ask on first start after installing DeltaChat should be all messages stored on server or should it work as @csb0730 said.

So the user have a choice. And of cource, this choice should be changeble in settings.

[testbird]

Maybe I have misunderstood you, and did not write understandable enough.

I think you filed this issue, and it is concerned about server space limits. And you suggested an option that would delete messages after a time (as userpreference).

I don't think that the suggested feature would solve the problem properly, and that it would necessarily require front-end UI changes.

• It would delete messages even for users that have many GB of free storage, if enabled by default.
• It would't prevent running out of space, where storage space and traffic is very limited (e.g. like the 64MB a user reported). One reason is that the number or age of messages is a bad measure for the storage usage.

What I wanted to add is, that I think there is a way (not the simplest but rather simple still) to implement proper storage limits now (a requirement for stable, continuous usage), that does not require any front-end UI changes, but would at the same time be already at least be compatible or extend-able with frontend configuration support for age or history limits in the clients later. (An efficient way to cover a range of features.)

I think the most basic but complete default mode (https://github.com/deltachat/deltachat-core/issues/120#issuecomment-417267057)

• should not need any front-end changes, or configuration
• ensures local and server limits are kept by deleting oldest chat messages if really required (chatting would otherwise become impossible unnecessarily)
• stll ensures a minimal operational capacity (stop deleting if it doesn't make any sense anymore, never completely delete existing chats automatically, instead let the chatting get stuck cleanly in this case, and require the user to delete a chat or other emails, to prevent automatically doing anything stupid, like crippling further DC usage)

[webratte]

My english isn't good enough to understand all you write 😉

But another thing for delete all messages (as a userpreferece):

The most DeltaChat useres will be non tech users. They will know nothing about stuff like autocrypt or PGP. They only want chat like they know it from WhatApp. But secure 😉

This users will find in the chat folder only unreadable mails which mess storage.

You know what I mean?!

And technical interested users should have a option to keep chat mails on the server to read it with a normal mail client or via mailvelope by using web login.

[testbird]

Talking about non-tech users, it is even better to have the messages (incl. setup message) on the server. So that deltachat can easily restore them if a device is lost or stolen, or a new device was bought.

[testbird]

I mean when hammering on an issue, one needs to be careful not to hit deltachat on the foot. ;-)

[webratte]

Taking about non-tech users, it is even better to have the messages (incl. setup message) on the server. So that deltachat can easily restore them if a device is lost or stolen, or a new device was bought.

That's a good argument :+1:

[testbird]

It just came to my mind that user preferences also have the drawback to break the advantage of knowing the common functionality that all users have, and benefit from out of the box.

If important features or requirements can be made to work well without any configuration, I think it is worth while to pursue this with good defaults. (Tweaking allowed, but not required.)

[r10s]

"Power to Defaults!"

[testbird]

Cool, if the field for "starred" could also be set to "archived", it would not only allow robust and safe, configuration-less auto-delete, but would also make that "Local Chat History" configuration option from https://github.com/deltachat/deltachat-core/wiki/IMAP-strategy obsolete: Server preference can be used for the most recent messages (multi-client sync) and local preference for archived messages.

Is it reasonably possible to also allow setting the "starred" property to "archived"?

[r10s]

so you would like the field "starred" being tri-state? normal, starred, starred+archived? and "archived" would mean that a message is never deleted from any device in a multi-device-setup?

[testbird]

I had first only thought of three mutual exclusive message states (EDIT: thanks to your comments), there are now four: "synced-new", "synced" may be the default (unflagged), "starred" and "archived" (e.g. "starred" prevents "archived").

"synced-new" (new messages get this state)

• never auto-deleted
• later: to be kept in sync with server

"synced" ("synced-new" messages are changed to this status, if they are not among the most recent 3 (or if smaller, the "number of synced messages") in a chat anymore after a new message was sent or received)

• may be deleted without breaking current chat operations
• used for storage space limit enforcing auto-deletes for local and server limits (delete oldest from all chats locally or on server as required, until the limit is reached again)
• later: also to be kept in sync with server
• later: (when the "archived" messages already get removed from the server) and the local limit can't be met by auto-deleting any more "synced" messages, ask user to (auto-)delete from "archived"

"starred" (manual selection)

• unmutable, neither auto-archived, auto-deleted, nor manually deletable (without first removing "starred" status)
• synced like synced-new
• when the user removes the "starred" state, set the message state to "synced-new" again (should get updated to "synced-old" with the next sent/received massage checks, if there are enough newer messages)

"archived" (manual selection or auto-archive based on a "number of synced messages" setting)

• not synced with server, allows for larger local histories later
• later: configurable auto-archive, allows archiving messages from "synced" messages
• later: (by default) automatically delete messages from server when setting the "archived" state, this allows influencing the used server space with the auto-archive configuration (the client with the smallest number of "synced" messages defines the number of messages that stay on the server (as long as this default "to delete from server if a message gets archived" is not disabled on the client, which is a good way to allow selected clients to archive everything (number of synced messages set to zero) without deleting everything from the server)

The states may be implemented in stages, starting with an auto-delete that would allow to always stay operational within the given server and local storage limits, by deleting from the "synced" messages, see https://github.com/deltachat/deltachat-core/issues/120#issuecomment-417267057

---

Did you see a use for "starred+archived"?

[testbird]

Could think of one more state that could be needed later:

("synced -history" added in an edit above)

[r10s]

Did you see a use for "starred+archived"?

no, i am just wondering if "archived" messages in your idea can also be "starred" or if "archived" messages are always also implicitly "starred". or can manual starring only take place until a message gets archived automatically.

[testbird]

I thought starring could take place at any time, saving the message from being deleted even manually. (Switching the state from any other to "starred".)

Hm, but I think I get your point?: It could make more sense not to auto-delete "archived" messages, but instead only from a "synced-history".

It may then make sense to use "synced-history" for the space limiting auto-deletions already in the minimal implementation. That could stay so later, and results in "archived" messages not getting auto-deleted by default.

(EDIT: added that to the oveview above)

[testbird]

Oh, or is it this: I think the chat history could later always list synced and starred, and be toggled to also include the "X further archived messages". Listing everything together then.

[testbird]

Ok, I merged all states into the post above.

[csb0730]

Please keep things simple! Only a simple solution is a good solution ;-)

As I read all these thoughts I think @testbird is using the server as the main message pool, in contrast to @webratte.

I'm doing a mix by setting the mail server to autodelete mails after some weeks so I have the chance to supply more clients with the same mails but I'm not expecting from mailserver to control all message archiving and control. Additionally, I really don't want the mailserver archiving mails for longer period! And I think webratte does the same? Maybe this is not the absolute strict multi device sync philosophy, but IMHO that's simply what some people are doing ;-)

To come back to the original issue here: Webratte described only the simple request to delete a message immediately after downloading, (correct? @webratte ).

[r10s]

Webratte described only the simple request to delete a message immediately after downloading

the original request reads as if the messages should not be deleted immediately but after some time:

If messages will deleted after a time (as userpreference) there should be no problems.

[csb0730]

There is a statement from webratte 3 days ago. Maybe this explains the issue more:

Every "normal" messenger stores messages only on the device. Messages will deleted as soon it's stored on the device.

[testbird]

Mind you that a simple solution that oversimplifies may work for one (you) but not for all use-cases. And notice that simply deleting old messages and keeping say only 100 in every chat would not prevent running out of space on the server or on the device at all. If there is not much space, a dozen of messages with attachments spread across a low number of chats can already fill up the device or server.

Therefore I proposed to base the auto-delete on a simple database query for deletable messages (exempting the last three of each chat) sorted by oldest first (the query lists messages from all chats).

Simply delete messages from that list until a storage percentage limit is met again. A message property (flag) makes sure not to delete the latest messages of each chat. That's all that is needed for the most basic solution, but having planned for a complete solution helps to avoid mistakes.

The solution is not about how I use it, but needs to be simple to configure and to work universally.

A user should for sure not have to configure anything for deltachat to take proper care about not filling up the local or server space (auto-delete oldest messages to stay operational by default).

Only when implemented in full, and one needs the history to be further limited (for any reason) all relevant use-cases should be configurable by the three proposed settings of the auto-archiving 1) auto-archive messages exceeding: some number (default last 100?) 1) keep archived messages on server: yes or no (default) 1) limit archive size: no (default, i.e. only physical limits) or some number of messages or MB