search

ubports / ubuntu-touch

Ubuntu Touch's issue inbox is now migrated to GitLab.
https://gitlab.com/ubports/ubuntu-touch
1.28k stars 110 forks source link

Some received SMS are truncated (from the begining). Very last words are visible. #1022

Closed Thieb67k closed 2 years ago

Thieb67k commented 5 years ago

Description of the feature The 1st sms of the day, for one contact, is truncated: I got only the end of the sms. That happened today and yesterday with the same contact. Following sms are complete.

Expected behavior To get a complete text...

Actual behavior incomplete sms even if "Status" in properties is "received".

Flohack74 commented 5 years ago

Was it longer than 160 chars?

Thieb67k commented 5 years ago

Nope, in both cases. And everyday I reboot the phone hence that sounds like recurrent, or random. Plus I do have the end of the text. The beginning is missing and I would expect the opposite if the text is too long... Is there a link with the issue #976 ? Because in that case, I cannot send/receive a MMS and even the associated text is not send/received.

dobey commented 5 years ago

Were they simply out of order multiple SMS messages? When an SMS is > 160 characters, it is split into multiple messages. I have seen these arrive and/or display out of order before.

Thieb67k commented 5 years ago

The point is I do not have multiple messages. I have only one, truncated. Not only one word but a significant part. By the way, even if that is not systematic, that happens only with my wife, which is seriously problematic...

ntorquet commented 5 years ago

My wife has two problems that are occurring frequently:

  1. With two of her contacts, from time to time, she gets a truncated message with the end of a previous message (from the same contact) pasted at the end of this truncated message.
  2. Some messages of the same two contacts (most of the time long messages, or mms) are not received at all, while other (shorter) ones are received correctly. Her phone is a OPO, with the last UBPorts release, but it happens since OTA 9 at least.
lduboeuf commented 4 years ago

@ntorquet still having that issue ?

ntorquet commented 4 years ago

@ntorquet still having that issue ?

Yes! It is very uncool because you can't evn know that someone send you a message. And the case of truncated messages is... very strange

lduboeuf commented 4 years ago

yes very strange, never experienced that. on OTA 11 stable ?

ntorquet commented 4 years ago

Yes.

lduboeuf commented 4 years ago

What can be done, is to check if message are effectively truncated in history database, you can retrieve it there: /home/phablet/.local/share/history-service/history.sqlite and with a sqlite client (e.g DB Browser ) see messages in "textevent" table.

ntorquet commented 4 years ago

@lduboeuf Thanks for your help. I will take a look a come back to you.

ntorquet commented 4 years ago

@lduboeuf I had a look in the history database. The text messages are stored exactly as they appear on the screen. They are truncated and mixed up with old messages from the same sender (only from Orange/Josh users as far as I know).

lduboeuf commented 4 years ago

ok, just to know, do you have lots of messages stored ?. I think i saw a comment in code about message overload.

ntorquet commented 4 years ago

Right now, 11861 messages stored :o)

lduboeuf commented 4 years ago

Ok, would be interesting to see if this can be the culprit. What can be done: backup of history db, remove it and see if it happen again. And tell us if you still have the pb (or not).

ntorquet commented 4 years ago

My wife removed a lot of sms. But, she still doesn't receive some sms sometime. Very annoying because if no body tell you that he send a message to you, you, you will never know... It happens to me also sometime.

lutin11 commented 4 years ago

On Nexus 5, and devel channel (20191218) I've got truncated message as well. Affected message are mms as i can see from text_events table (/org/ofono/mms/208150116711414/2b4eaaff472848ef67964b89e41754ac) Into the message column I've got NULL

lduboeuf commented 4 years ago

i don't know how are handled mms for now. Do you have this behavior by removing the database and started again with a clean db ? ( don't forget backup )

Thieb67k commented 4 years ago

Update: That issue does not happen every times even though I feel it is getting worse (most frequent, begining and end of messages are truncated). The text contains some usual french characters that are not problematic with others messages. Sending the exact same message does not help. I even noticed that one or two messages are missing, yet the sender's messages notifications show that is allright... DB browser does not show the complete message. That happens on mako but if someone has a spare compatible device... ;)

Thieb67k commented 3 years ago

Hello, Unfortunately, that still happens with OTA-17. Is there any news for this issue?

lduboeuf commented 3 years ago

@Thieb67k do you confirmed that your message that is stored in the db (~/.local/share/history-service/history.sqlite) table text_event is the same as it is displayed ? Do you know if it only happens from the same contact, or maybe same provider ? If your contact re-send it, does it display the same ?

If you can reproduce, before the contact send you the message, you can monitor the sms recevied with that: dbus-monitor --system "type='signal',sender='org.ofono',interface='org.ofono.MessageManager'"

You will have something a bit noisy but you should read the content ( starts after "string' ): e.g:


   signal time=1621517592.684904 sender=:1.25 -> destination=(null destination) serial=3892 path=/ril_0; interface=org.ofono.MessageManager; member=IncomingMessage
   string "Du 07/05 au 07/06 votre conso hors forfait est de 0.00€.

Voix : 00:03:57
SMS : 14
Données : 301.33 Mo
Retrouvez plus d'infos sur http://mobile.free.fr/moncompte/
Identifiant client: XXXX"
   array [
      dict entry(
         string "LocalSentTime"
         variant             string "2021-05-20T15:33:11+0200"
      )
      dict entry(
         string "SentTime"
         variant             string "2021-05-20T15:33:11+0200"
      )
      dict entry(
         string "Sender"
         variant             string "555"
      )
   ]
Thieb67k commented 3 years ago

Hello, -Messages stored into the db are the same as they are displayed on the messaging App. -I know that happens with 2 contacts and their provider are differents. -If the contact resend the text, it looks identical.

Because I know absolutely nothing about IT programming, should I paste the command (dbus-monitor --system "type='signal',sender='org.ofono',interface='org.ofono.MessageManager'") as is into the terminal? Thanks

lduboeuf commented 3 years ago

Because I know absolutely nothing about IT programming, should I paste the command (dbus-monitor --system "type='signal',sender='org.ofono',interface='org.ofono.MessageManager'") as is into the terminal? Thanks

Yes, connect with adb or ssh to your device and paste that command, then press enter. You should see the incoming sms and see if it is truncated as it is displayed in messaging-app

Thieb67k commented 3 years ago

Well, I am not comfortable with those tools (Terminal, adb, ssh...). Anyway, I managed to get that screenshot. "Unable", "error"... I need to spend more time on using that I guess... screenshot20210525_134609028 1

lduboeuf commented 3 years ago

@Thieb67k yes, noticed that when doing it from the terminal, it won't work. If you can connect your device to a PC, then do adb shell on your PC and then the paste command above

Thieb67k commented 3 years ago

Ok, Thanks! That works. I got a text from my provider and that looks similar to yours. Next step is to do the same with a problematic text.

Thieb67k commented 3 years ago

Ok, I have checked the command (dbus-monitor --system "type='signal',sender='org.ofono',interface='org.ofono.MessageManager'"). I see the incoming sms and each time it appears like as it is displays in the Messaging-App: full or truncated.

When I have the "Oups" notification because I cannot see the message (here, a sms, not a MMS) into the Messaging-App, the Terminal show me that text: The difference starts at line5 "member=MessageAdded object path "ril/0/message" instead of "member=IncomingMessage..." Then line18 ("member=MessageRemoved..."). I have not deleted anything.

I hope that helps.

screenshot20210526_105501523 1

lduboeuf commented 3 years ago

humm, so if you said that they appear like it is in Messaging-app, it means that ofono component have the issue. Thanks for reporting

lduboeuf commented 3 years ago

@Thieb67k or anyone willing to help testing. I can't provide any guarantee but i want to try things at lower level. But i need a tester in devel channel and that can reproduce the issue

Thieb67k commented 3 years ago

Thanks for your help. I hope that story will conclude with an happy end :) I will see what I can do. Thanks again.

lduboeuf commented 3 years ago

@Thieb67k are you on Telegram ? I would like to first try some stuffs locally If so please pm me on Telegram or Forum ( lduboeuf )

lduboeuf commented 3 years ago

so as @dobey says, the message is not truncated but out of order ( at least for what you've sent me @Thieb67k )

So we can eliminate the modem, i'm looking into Ofono, that should be there but i'm a noob in clang, sms spec, glib, etc... Probably a long journey for me. I'm trying to look around if no fix has been already done upstream

lduboeuf commented 3 years ago

oh not fixed in sailfish too https://together.jolla.com/question/161920/bug-sms-are-sentreceived-in-bad-order/ could be related: https://issuetracker.google.com/issues/36944392#c5

Flohack74 commented 3 years ago

@lduboeuf well the last comment there is from 2017, so maybe it was fixed silently. Such an old report would need reconfirmation with a current version before we make this assumption. You could comment there asking for a state ;)

lduboeuf commented 3 years ago

I'm trying to look around if no fix has been already done upstream

Well, nothing exist to fix that afaik, looked in sailfish repo and usptream ofono. I've tried to dig the issue, but it is really hard to get into it. Could be an encoding issue, bad character somewhere, some buffer not freed correctly... I admit i will not be able to go further, sorry.

As the issue is on non UT sender phone, they seems to not respect the sms protocol correctly, so not a UT Ofono issue i would say. Or we could make a hack, but honestly it can takes days for me at least to figure out what is the issue and more importantly without breaking things. For users who face that issue, i invite you to ask users to update their phone if this possible.

cheers

lduboeuf commented 3 years ago

OK, maybe i've found something, i need to give some few hours more... :)

lduboeuf commented 3 years ago

So i've tried to put logs everywhere, the sms chunks are received in proper sequence but for some reason, they are mixed. I can't go further sorry. Hopefully someone with better low level ubderstanding can spot something.

Flohack74 commented 3 years ago

Can you point me to some code? I can take a look. Sometimes thats a minor thing, easy to fix, but with great consequences.

lduboeuf commented 3 years ago

@Flohack74 So here are some debug logs, but you won't find all them in the code: The use case here is a sms that is splitted in 2 parts, received in the proper order 1,2 , but the output becomes 2, 1

phablet ofonod[2005]: drivers/rilmodem/sms.c:ril_sms_notify() req: 1003; data_len: 676
May 29 17:07:44 ubuntu-phablet ofonod[2005]: [0,UNSOL]< UNSOL_RESPONSE_NEW_SMS {XXXX}
May 29 17:07:44 ubuntu-phablet ofonod[2005]: drivers/rilmodem/sms.c:ril_sms_notify() smsc_len is 8
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:ofono_sms_deliver_notify() len 167 tpdu len 159
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:handle_deliver() 
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:handle_deliver() ref: 141 max: 2 seq: 1
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_assembly_add_fragment_backup() 1 inserted at position 0, bit 2, num_fragments 1
May 29 17:07:44 ubuntu-phablet ofonod[2005]: [0,0176]> RIL_REQUEST_SMS_ACKNOWLEDGE  (1,0)
May 29 17:07:44 ubuntu-phablet ofonod[2005]: drivers/rilmodem/sms.c:ril_sms_notify() req: 1003; data_len: 180
May 29 17:07:44 ubuntu-phablet ofonod[2005]: [0,UNSOL]< UNSOL_RESPONSE_NEW_SMS {XXXX}
May 29 17:07:44 ubuntu-phablet ofonod[2005]: drivers/rilmodem/sms.c:ril_sms_notify() smsc_len is 8
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:ofono_sms_deliver_notify() len 43 tpdu len 35
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:handle_deliver() 
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:handle_deliver() ref: 141 max: 2 seq: 2
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_assembly_add_fragment_backup() 2 inserted at position 1, bit 4, num_fragments 2
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:sms_dispatch() 
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:sms_dispatch() dst -1 src -1
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:sms_dispatch() dst -1 src -1
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_decode_text() Guess size 160
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_decode_text() decode_text charset other than 7bit
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_decode_text() decode_text charset 7bit, max chars: 11
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_decode_text() converted after: emi épaisse
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_decode_text() utf16
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_decode_text() converted after: emi épaisseSteack burger ( on a du pain ) , bleu ou roquefort, crème fraîche s

so here: https://github.com/ubports/ofono/blob/xenial/src/smsutil.c#L2214 the GSList *sms_list is already out of order, was wondering if the for loop guarantee the order.

this list is filled here https://github.com/ubports/ofono/blob/xenial/src/smsutil.c#L2651

I've used that tool to make sure sms PDU was received ok: https://www.diafaan.com/sms-tutorials/gsm-modem-tutorial/online-sms-pdu-decoder/

lduboeuf commented 2 years ago

@Thieb67k Would be interesting to see if it occurs on a Device Android >=9 too or Pinephone as it doesn't use the same ofono.

Flohack74 commented 2 years ago

Humm interesting I totally forgot about this. Yes the for loop guarantees the order, thats a usual iterator, and there is nothing that can manipulate those positions. So I assume the insert is done with the wrong positions. Also, if you

Flohack74 commented 2 years ago

Probably that decoding has a bug: grafik

lduboeuf commented 2 years ago

it could, idk :) , i've search for any updates in ofono uptream and mer branch and haven't found anything related to that kind of issue

lduboeuf commented 2 years ago

well the position seems to be fine because the sequence is : First chunk: seq 1 inserted at position 0, bit 2, num_fragments 1 Second Chunk: seq 2 inserted at position 1, bit 4, num_fragments 2

I can't find where i've put the logs before but it was nearby that https://github.com/ubports/ofono/blob/xenial/src/smsutil.c#L2651-L2654

Flohack74 commented 2 years ago

Do we have test data that we can capture to manually check whats going on? Also, I think multipart SMS was always fine for me, maybe its happening only for some providers, so it will be hard to debug.

lduboeuf commented 2 years ago

Yes we have to found someone that send bad sms. I personally never had that. I used to test it a bit by receiving sms from the suspected phone directly from @Thieb67k

I have the raw sms data received (PDU) locally here, but as it is sensitive, i did not share it here.

lduboeuf commented 2 years ago

I don't see updates here too: https://together.jolla.com/question/161920/bug-sms-are-sentreceived-in-bad-order/ and don't know where to find the bug tracker for that issue

Flohack74 commented 2 years ago

Ok if its sensitive can you just replace the message data by garbage strings?

lduboeuf commented 2 years ago

like the one above: received in proper sequence 1,2 and decoded has 2,1

May 29 17:07:44 ubuntu-phablet ofonod[2005]: drivers/rilmodem/sms.c:ril_sms_notify() req: 1003; data_len: 676
May 29 17:07:44 ubuntu-phablet ofonod[2005]: [0,UNSOL]< UNSOL_RESPONSE_NEW_SMS {##PDU SEQUENCE 1##}
May 29 17:07:44 ubuntu-phablet ofonod[2005]: drivers/rilmodem/sms.c:ril_sms_notify() smsc_len is 8
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:ofono_sms_deliver_notify() len 167 tpdu len 159
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:handle_deliver() 
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:handle_deliver() ref: 141 max: 2 seq: 1
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_assembly_add_fragment_backup() 1 inserted at position 0, bit 2, num_fragments 1
May 29 17:07:44 ubuntu-phablet ofonod[2005]: [0,0176]> RIL_REQUEST_SMS_ACKNOWLEDGE  (1,0)
May 29 17:07:44 ubuntu-phablet ofonod[2005]: drivers/rilmodem/sms.c:ril_sms_notify() req: 1003; data_len: 180
May 29 17:07:44 ubuntu-phablet ofonod[2005]: [0,UNSOL]< UNSOL_RESPONSE_NEW_SMS {##PDU SEQUENCE 2##}
May 29 17:07:44 ubuntu-phablet ofonod[2005]: drivers/rilmodem/sms.c:ril_sms_notify() smsc_len is 8
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:ofono_sms_deliver_notify() len 43 tpdu len 35
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:handle_deliver() 
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:handle_deliver() ref: 141 max: 2 seq: 2
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_assembly_add_fragment_backup() 2 inserted at position 1, bit 4, num_fragments 2
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:sms_dispatch() 
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:sms_dispatch() dst -1 src -1
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:sms_dispatch() dst -1 src -1
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_decode_text() Guess size 160
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_decode_text() decode_text charset other than 7bit
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_decode_text() decode_text charset 7bit, max chars: 11
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_decode_text() converted after: emi épaisse
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_decode_text() utf16
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/smsutil.c:sms_decode_text() converted after: emi épaisseSteack burger ( on a du pain ) , bleu ou roquefort, crème fraîche s
May 29 17:07:44 ubuntu-phablet ofonod[2005]: src/sms.c:sms_dispatch() dispatch_text_message

As you notice "emi épaisse" should be the second part, but is inserted at the beginning

lduboeuf commented 2 years ago

humm interesting, when using that tool: https://www.diafaan.com/sms-tutorials/gsm-modem-tutorial/online-sms-pdu-decoder/ if you check the PDUs, the first one have: Data coding: Unicode the second one: Data coding: | SMS Default Alphabet

see comment about charset here : https://github.com/ubports/ofono/blob/xenial/src/sms.c#L1308