"Removed by Server"

[n00b12345]

Quite often I notice some of my messages not going through. When I press the "i" button, I see that the OMEMO fingerprint has been "removed by server". I authenticate it again by verifying it and then it works perfectly fine. The same thing repeats quite often (sometimes after the SSL certificate is renewed).

Is this something to do with ChatSecure or the XMPP server software I'm using?

[chrisballinger]

This can be caused by the other party using a client that removes/replaces keys. There's not much we can do about it other than changing the default behavior for when keys are removed... but if we don't respect the server you may end up encrypting to keys revoked by the other party.

On Mon, Jul 10, 2017 at 12:19 AM, n00b12345 notifications@github.com wrote:

Quite often I notice some of my messages not going through. When I press the "i" button, I see that the OMEMO fingerprint has been "removed by server". I authenticate it again by verifying it and then it works perfectly fine.

Is this something to do with ChatSecure or the XMPP server software I'm using?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ChatSecure/ChatSecure-iOS/issues/814, or mute the thread https://github.com/notifications/unsubscribe-auth/AAfqHy2zwNrNgdkQG2MG5YDJ_lNa_jVDks5sMaY3gaJpZM4OSVrG .

[n00b12345]

@chrisballinger I'm using ChatSecure with someone who is using Conversations for android. Should that be the root of the problem?

[a11fruitless]

I'm seeing similar strange behavior to n00b12345 with only some of my clients. This is on a Prosody server. I am using multiple instances of Gajim and Conversations with another user who is using ChatSecure. All my clients stay permanently online. Then the other user will send ChatSecure to the background where it eventually logs out. Once the user opens ChatSecure back up, the OMEMO Conversations' key is still trusted, but the Gajim keys have been "removed by server." Despite the fact that none of the keys have changed. Once they are trusted again, everything works great, but the process will repeat itself once ChatSecure logs out. This process is pretty repeatable for me right now.

It is strange that n00b12335 sees this for a Conversations client, but I see it for my Gajim clients.

[chrisballinger]

Hmmmmm well that should only happen if the keys were removed from the server. Some OMEMO clients still don't update their device list properly and wipe out other (active) devices.

[hschraep]

Same i have with a colleague he have a iphone with chat secure installed and from time to time he has the problem that some omemo fingerprints was deactivated. After he enable it back all is working again.

We both are connected to the same prosody server. But i have a android connected with conversations but here never would be fingerprints deactivated.

So the question is how we could help to solve the prolbem? If we can provide some logs, tell us how and what you need.

[a11fruitless]

I think I can clarify the root issue for myself here (which is admittedly a little different from what I posted earlier). Hopefully this coincides with the other user's experinces as well. I am presently using 4.2.1

The primary issue for me is that if a device is logged out for an extended period of time from the server, the verification of that OMEMO key is forgotten in the ChatSecure client when they log back in finally. I can manually verify it once they log back in and everything again works fine.

This causes problems for desktop clients that may be logged out for a while. If the user also has a phone client that stays online consistently (like Conversations which that key never seems to be unverified by ChatSecure) then the desktop client won't receive messages sent by ChatSecure until that particular key is reverified in ChatSecure.

Even if this issue is related to the server or the other client, it seems strange that ChatSecure wouldn't remember that the key has already been manually verified before and automatically verify the key again. ChatSecure can still receive messages from the other client and they show up with the caution symbol (that client just doesn't receive messages from ChatSecure until the key is reverified).

[chrisballinger]

The original intention was to prevent you from forever encrypting to someone's unused keys. When receiving a message it should update the "last seen date" of a key, and then it will untrust a key that hasn't been seen in 60 (? i forget) days. This logic could be improved, or at least the duration could be extended.

"Remove By Server" is a different distinction and it's when the list of keys on the server no longer contains a key. Sometimes servers or clients are buggy and wipe out the list by accident. The logic here could also be improved so previously trusted keys are re-trusted when added back by the server (or you receive a message from them again), but is limited by the current OMEMOTrustLevel states we store in the database model.

typedef NS_ENUM(NSUInteger, OMEMOTrustLevel) {
    OMEMOTrustLevelUntrustedNew = 0,
    OMEMOTrustLevelUntrusted    = 1,
    OMEMOTrustLevelTrustedTofu  = 2,
    OMEMOTrustLevelTrustedUser  = 3,
    /** If the device has been removed from the server */
    OMEMOTrustLevelRemoved  = 4,
};

We'd need to rethink these states a little bit to fix these edge cases.

[jotwewe]

I experienced the same as described in https://github.com/ChatSecure/ChatSecure-iOS/issues/814#issuecomment-332338264 I was using conversations 1.23.6, the other guy chatsecure/ios 4.2.1. Both accounts on a prosody 0.10, which never showed similar behaviour with other clients (multiple versions of conversations, dino and recently gajim). We exchanged omemo-encrypted messages one day (both directions), but ten hours later his messages didn't arrive in conversations anymore (only in my dino client). Maybe conversations was offline during the day and dino was not. It turned out that the key for my conversations client was marked as untrusted in chatsecure and something like "removed by server" was written beside it.

If I remember correctly, my gajim key (installed after the said message exchange) was also not marked as trusted. What is the intended behaviour of chatsecure regarding new keys?

[iNPUTmice]

In Conversations a key that is removed from the PEP device list will only appear as inactive (grayed out in Conversations). We won’t send messages to inactive devices but they can become active again when they get re-announced/re-added to the list or when we receive a message from that device. Inactive keys aren't marked as untrusted. Trust state in Conversations is independent of active/inactive. Prosody removes all keys on restart. Devices going online one after the other will add themselves to the announcement again.

[jotwewe]

iNPUTmice, thanks for your comment. Indeed the related Prosody server was restarted shortly before the messages from Chatsecure stopped appearing in Conversations and I guess Conversations was offline during this time. I feel https://github.com/siacs/Conversations/issues/1901 is related in this context.

[Spydar007]

I too am having this problem after almost every message sent with the other person using Conversations for Android. Various keys get "removed" by the server, and I think it is mostly whilst the devices are offline.

[LeoniePhiline]

I can confirm what a11fruitless describes in https://github.com/ChatSecure/ChatSecure-iOS/issues/814#issuecomment-357463824

Incidentally, I can probably also confirm jotwewe (https://github.com/ChatSecure/ChatSecure-iOS/issues/814#issuecomment-377687546): Gajim's "Server Status" dialog shows that prosody (dismail.de) has an uptime of only 7h, so it must have been restarted just before the message loss occured again.

I am using Conversations and Gajim, the other person is using ChatSecure. Conversations never misses any messages, but Gajim regularly gets cut off from the conversation: When logging in with Gajim, message Carbons are then not received. New messages from Conversations got Gajim do then appear in Gajim, but messages from ChatSecure arrive only at Conversations but not at Gajim, until the other person edits the fingerprints list in ChatSecure to manually re-enable the Gajim fingerprint.

It would be very desirable if, like Conversations does, also ChatSecure would automatically re-trust the key Gajim key when it is re-added to the receivers list after having been offline (and after a possible prosody restart?!).

[LeoniePhiline]

Apart from that, thank you so much for your fantastic work!

[mimi89999]

@LeoniePhiline Please make sure that your server is running Prosody >= 0.11.0 as it is the first version that introduced support for persistent nodes after server restart. In a commit I made recently it XMPPFramework https://github.com/robbiehanson/XMPPFramework/commit/4c060bfaf88777dcc75192f658ad2089ac33bc50 , I added setting the persist_items items option to true to make sure that they will be persistent on the server.

[anu12anu12]

I am using Mangoose IM. I am using the version (3.6.1)of it. Still for me keys are getting removed by the server in Chatsecure. In Android I am using Conversations. And I am always replacing the keys in Android and maintain one key in Android client. Still no clue. Will taking robbiehanson/XMPPFramework@4c060bf will solve the issue? Do you have any suggestion what changes are required in MangooseIm server.