WhatsMeow vs. BitlBee

[yourealwaysbe]

Apologies if this is premature. I had a go with the whatsmeow branch on BitlBee but was unable to progress past the login. After commanding a connect, there is no output in BitlBee until the connection attempt times out. I have tried this with qrcode-size set to 0 and spectrum-compatibility set to false (and true).

I had a play trying to debug in the connect method of login.go, but failed to find a way of getting any visible logging output. I expect the issue is that purple_display_qrcode is used, and that is probably not supported by BitlBee. In the master branch gowhatsapp_display_message was used.

Happy to help get this working, but i'm not sure where to start at the moment.

[hoehermann]

Thank you very much for the report. It is good to have you on board for tests regarding bitlbee. With the qrcode-size set to 0, I am using a simple text message. However, I just noticed I forgot to set PURPLE_MESSAGE_RECV. I suspect bitlbee drops the message because of that. I want to fix that soon.

The whatsmeow version is rather verbose, but all logging is done via purple_debug. It is quite possible that bitlbee does not support that function. In src/go/login.go and src/go/init.go, you can replace all instances of PurpleLogger with waLog.Stdout("whatsmeow", logLevel, true) – that should bypass the logging mechanisms of purple/bitlbee and give you the output directly.

[yourealwaysbe]

Thanks -- saw the PURPLE_MESSAGE_RECV push today. Unfortunately it didn't seem to help. I'll have a play with debugging later.

[hoehermann]

I had hoped the PURPLE_MESSAGE_RECV would solve the issue. I copied most of the code from the gowhatsapp variant. The whatsmeow version should be better in virtually every aspect – yet for bitlbee and spectrum it seems I managed to bork it completely.

After a chat with the admin of https://jabber.hot-chilli.net/, I did some changes to the login mechanism and how the plug-in connects to its database. Maybe that changes something for bitlbee as well.

[yourealwaysbe]

Thanks. I had a go with 686e996bfad4421451d5f56c1d3642302e33ce9c but still no luck. I eventually managed to get debug output just by using good old printfs. I don't know where the other logs go.

It seems that it gets as far as callling purple_display_qrcode in the connect method of login.go, but i can't find what happens to this call. By my reckoning it should go through the bridge and end up in gowhatsapp_process_message in process_message.c, but it seems that it does not. It does reach the purple_display_qrcode function in bridge.go.

[hoehermann]

Thank you for testing. You are correct. The flow should be:

1. loop in connect
2. purple_display_qrcode
3. gowhatsapp_process_message_bridge
4. process_message_bridge
5. gowhatsapp_process_message
6. gowhatsapp_handle_qrcode

The bridge is a bit finicky. Go can make asynchronous calls into purple. Pidgin does not like that (I do not know purple's exact feelings on this). With Pidgin (and in purple, I assume) everything related to the UI must be executed on the main thread. The bridge tells purple "this message came in asynchronously, execute on the main thread as soon as you can". It is really weird the message is lost in bitlbee. Especially so when considering the fact this mechanic has been working well ever since Eion Robb himself implemented it. I suspect something is wrong blocking the gtk event loop so purple_timeout_add does not do anything and/or process_message_bridge is never executed. By default, logging is also done via this route – that is why you are not seeing any debug output. :/

[yourealwaysbe]

Thanks. Digging further, it gets as far as the account exists check in process_message in bridge.c. It fails this test, and gives up. It looks like purple_accounts_get_all() returns an empty list in gowhatsapp_account_exists.

edit: if i force gowhatsapp_account_exists to return 1, it fails with an invalid free sometime later, but i wouldn't be surprised if that was related.

[hoehermann]

That is indeed a difference from the gowhatsapp version I have totally forgotten. Thank you for pointing that out.

In the gowhatsapp version, I am communicating between go and C by using the connection pointer and purple_connections_get_all.
For the whatsmeow version, I chose to use the account pointer instead – so I can forward information related to an account even if there is no active connection. But it looks like bitlbee does not implement purple_accounts_get_all.

I think I see a way to combine the strengths of both variants. I probably won't implement it before Friday, though.

[yourealwaysbe]

Cool, thanks. No need to rush to get it done.

[hoehermann]

Changed in a5038f3. Please let me know if this helped.

[yourealwaysbe]

Great, i'm in. In took me a while because i didn't realise i had to enable the multi-device beta. I had assumed it would work without. I have only tested with a group chat with myself so far.

1. Images do not download and display as file system links like they used to.
2. Group chats are a bit weird -- one of mine is shown in the chat list without the topic, the one i was using for testing isn't (but maybe because i'm the only person in it).
3. Messages sent by me from BitlBee also appear as received messages.

I can take a look at the group chats in time. The log has an assertion failure in the get chat data method.

[hoehermann]

Good to hear it is working. :)

1. The "download automatically and display link" mechanics was criticized by a number of users, mostly the spectrum folks. The plug-in now uses standard PurpleXfer API for file transfer. This currently happens with all attachments, including images. It is working fine in Pidgin. I will probably add the "inline images" feature, but it is not exactly my top priority right now.
2. Yes, group chats are indeed a bit wonky. I mostly use direct messages and rarely test group chats. Text messages are sent and received reliably. Files cannot be sent to group chats. I think it is a limitation of purple. Images might be send if I enable inline images, but I am not sure. I am mostly copying from the tdlib plug-in. ;)
3. In order to cope with #95, sending a message is now done asynchronously. Upon success, the message is echoed into the conversation explicitly. The spectrum-compatibility setting changes this behaviour. I can bitlbee make use of that, too?

[yourealwaysbe]

Thanks.

1. the xfer window doesn't appear, i can have a look what's happening under the hood here.
2. this one's also in #5, just noting it's the same here too.
3. spectrum-compatibility true fixes the echo.

[yourealwaysbe]

Preliminary observations on file transfer: it gets as far as calling purple_xfer_request in handle_attachment.c, but that doesn't appear in BitlBee. I'm not sure why not since the code looks very similar to the way skype4pidgin does it and file transfers come through with that.

[yourealwaysbe]

A possible fix for the file transfer i'm experimenting with is to use the sender instead of the chat name in handle_attachment. It looks like BitlBee -- or possibly Weechat -- doesn't know what to do with a transfer from a room rather than a nick.

Is this there a reason it was set up to come from the chat not the true sender? Is replacing the chat with the sender a reasonable change? I'll put together a pull request at some point, but i'm still testing it for now.

[hoehermann]

Is this there a reason it was set up to come from the chat not the true sender?

Pidgin shows the link to a downloaded file in the conversation window. I wanted to have it consistent in the same conversation. As of today, I did not manage to implement it correctly (see sub-section "Known issues"). It works fine in tdlib-purple. It may or may not be related to the list of group chat participants not being populated correctly.

Is replacing the chat with the sender a reasonable change?

Yes, that is totally okay. Maybe one day I figure out how to do it properly, but until then a working bodge is better than a dysfunctional one. :)

I'll put together a pull request at some point

I am very grateful for your help. Maybe I can be of some assistance in regard of where to put GetGroupInfo and GetJoinedGroups. Can you give me an outline of how bitlbee does request the list of groups? A link to the code is fine – not needing to look it up saves me quite some time. :)

[yourealwaysbe]

Pidgin shows the link to a downloaded file in the conversation window. I wanted to have it consistent in the same conversation. As of today, I did not manage to implement it correctly (see sub-section "Known issues"). It works fine in tdlib-purple. It may or may not be related to the list of group chat participants not being populated correctly.

Sorry, you mentioned that already above and i had not noticed it. In BitlBee at least i don't think it's due to the list of participants not being populated -- i've tried it with participants who are recognised as being in the chat because of a previous message.

Can you give me an outline of how bitlbee does request the list of groups? A link to the code is fine – not needing to look it up saves me quite some time. :)

Currently the whatsmeow version doesn't seem to fetch contacts for me. I'm not sure if this is expected at this stage.

In the old version, group chats were set up alongside contacts when the contact list data was received. If the buddy was a group chat, it would create a new purple chat for it, and then set the topic if it could find it.

After that, the entry points for using chats is here for listing them and here for joining them. Both rely on the buddy list having had the chats added already.

[hoehermann]

I completely turned the group chat functions upside down. Instead of hoping they would be pushed, I am now explicitly requesting a roomlist from WhatsApp. It is not finished yet, but I am done for this weekend. Can you try the version in the dev branch? It would be nice to know if that is going into the right direction.

[yourealwaysbe]

Looks like a good start -- BitlBee is listing all my group chats.

When i try to join i get the error hehoe-whatsmeow - Error: Joining chatrooms not supported by this protocol. I don't see an obvious cause as the join_chat field is set.

[yourealwaysbe]

I made a few changes and can now join chats in BitlBee and send/receive messages and receive images. Changes committed here. Apologies for the scrappy commit, i stripped some trailing spaces and was too lazy to undo.

Essentially i just had to implement a chat_info_defaults function, and make sure it included a "name" field (the remoteJid).

I also commented out one call to purple_roomlist_room_add_field because there was a mismatch with the number of fields declared.

[hoehermann]

Dang it. I just did the very same thing. Took me three hours. But now at least I understand how room lists and their fields, conversations and their components, and the buddy list and its nodes have in common. Or more exactly – what they do not have in common.

Thank you for the error message. It pointed me to the right part of bitlbee code.

Also, participant lists are now populated. It works perfectly in Pidgin. Can you test https://github.com/hoehermann/purple-gowhatsapp/tree/dev?

[yourealwaysbe]

At least you know it works on BitlBee :)

[yourealwaysbe]

Btw, noticing #105 might explain why i keep complaining about missing contacts: if WhatsApp only sends them on pairing, and BitlBee forgets Purple account info, the contact list won't persist after restarting BitlBee. Not sure there's an easy fix for that.

[hoehermann]

I rely on the persistence of Pidgin's buddy list. Does bitlbee persist buddies? If not: Do you happen to know which function is called to get them from the plug-in? Or is the plug-in implicitly expected to deliver a list of contacts after it connects?

[yourealwaysbe]

Also, participant lists are now populated. It works perfectly in Pidgin. Can you test https://github.com/hoehermann/purple-gowhatsapp/tree/dev?

I'm getting the participant list, but a couple of bugs:

1. the topic isn't being set in gowhatsapp_chat_info_defaults but is set as required in gowhatsapp_chat_info.
2. i can no longer send messages to group chats -- i receive them ok, but they don't seem to go in the other direction.

[hoehermann]

Thank you so much for testing.

1. I forgot to change one instance of "remoteJid".
2. I now provide an empty string in the defaults.

Both should be fine now in dev. :)

[yourealwaysbe]

Great -- just spotted the remoteJid also :)

Dev seems to be working ok now.

[hoehermann]

In the past two weeks, I have not received any reports about severe defects. Is there any issue I forgot about? Especially in regard to bitlbee? The section about known issues lists the issues I know of but do not want to tackle right now.

If there are no critical show-stoppers, I want to make the whatsmeow branch the main branch and officially deprecate the old gowhatsapp branch.

[yourealwaysbe]

Thanks -- i've been using it and it's been fine for me.

There are some minor niggles, but i don't think they're urgent. At least they're not on my priority list.

• Fetched contacts don't have contact names (possibly a WhatsApp limitation, the purple-gowhatsapp version was the same, i just use the BitlBee rename command to identify people i talk to).
• inline-images doesn't work -- something strips out the HTML tags, i think that's BitlBee -- so the \<img src="..."> doesn't show up.
• The file transfer protocol does work though, and it feels cleaner than how purple-gowhatsapp did it. An annoyance though is that it's harder to connect the downloaded images with the conversation in situ. A bodge fix is to make whatsmeow send a message to the chat from the file sender that contains the filename or similar. A bit of WeeChat scripting can turn that into a full file link that can be opened from the terminal.

[hoehermann]

Thank you for the feedback.

• Contact names seem to be hit or miss in Pidgin, spectrum and bitlbee. I, too, guess it is whatsmeow or the WhatsApp servers. I do not think I can do much about it.
• I put linking of downloaded files on the list of improvements.

[hoehermann]

There has been a lot of development this week. Participant lists in group chats should be much more reliable now. Commit d0b3982 finally adds an option to write a message to the conversation when an attachment is received. It should be possible to turn this into links for bitlbee. Feel free to try and let me know of the results.

@yourealwaysbe Do you happen to know who has write access to https://wiki.bitlbee.org/? I would like to have this project advertised there instead of David's now obsolete plug-in.

[yourealwaysbe]

Thanks for the heads up. The messages on file look great and seem to work so far.

However, tbh, file downloads haven't worked properly for me for a while. I get a transfer started, but it (almost?) always fails. I'm not a massive WhatsApp user these days, so it hasn't bothered me enough for me to make a serious effort to debug the issue.

Still, i am getting the chat messages when images do come through :)

I'm afraid i don't know who's in charge of the bitlbee wiki.

[hoehermann]

Thank you for your response. I am glad the file messages are useful.

In issue #148 an Arch user notified me of a change in purple 2.14.10 which affects how file transfers are handled. Maybe this affects bitlbee, too?

WhatsApp servers occasionally reply with a 404 error code. I am afraid, I cannot really do anything about that. A "retry failed downloads" feels out of scope for this plug-in.

[yourealwaysbe]

I'm also on Arch. The update seems to have made a difference. I managed to send a picture to myself, but yesterday a group chat picture sent by someone else never arrived (though the message saying they'd sent it did). I noticed you've updated a bit since 5 days ago, including a commit that has something to do with file sending. I don't get many people sending me pictures, so it's kind of a long-term test :)

[samtrek]

Do you by any chance have autosave plugin installed?

[yourealwaysbe]

Do you by any chance have autosave plugin installed?

I don't think so. Which plugin is it?

[samtrek]

Should have rather asked you if you had inline image display option on? Sorry for the mix up. And autosave is the name of a plugin in pidgin

[hoehermann]

yesterday a group chat picture sent by someone else never arrived

Oh no. Another user just messaged me via Mail and told me the same. They were missing voice messages coming from groups, though.

Update: I just had everyone in my test group send me a random file (photo, voicemessage, video). All of them were received just fine. I hope we can find the cause of this issue together for I cannot reproduce with my test devices.

[yourealwaysbe]

Just had someone send me a picture. I got this message:

Message contained an attachment, but the download failed: &fmt.wrapError{msg:"file length does not match: expected 1778215, got 1767761", err:(*errors.errorString)(0x1c00007b3c0)}

This is not the same as before, where i got a message in the attachment-message format for a file that didn't exist.

[hoehermann]

That is https://github.com/tulir/whatsmeow/issues/127 creeping up on us. It looks like I can ignore the error and just forward the (incomplete) data. Feel free to try 8680dff and let me know if it is doing any good.

[yourealwaysbe]

Feel free to try 8680dff and let me know if it is doing any good.

That worked (though the file length can be intermittent). I'll report back if i see any other download issues.

Thanks for your work on this.

[yourealwaysbe]

yesterday a group chat picture sent by someone else never arrived

This seems to be a persistent bug, but only with group chats. Pictures sent to me individually transfer ok, but no transfer is initiated when the picture is sent to a group chat i'm a part of. In both cases the attachment-message comes through fine.

(This is not an urgent issue for me, so no worries if you have better things to be looking at.)

[hoehermann]

Although the defect is problematic, consistency is good. At least we have that. Are you in control of the bitlbee configuration? You could try setting group-is-file-origin to false. I do not remember why I came up with the setting – bitlbee might have been the reason. 😅

[yourealwaysbe]

Aha, that would make sense. I remember there being some problem along those lines last year.

Changing the setting appears to fix it (at least for a single test picture). Sorry i didn't spot/forgot the setting. Thanks.