Reworked PDU parsing; Delivery reports

magcks commented 4 years ago

Hey there again :)

In this PR, I reworked the whole PDU parsing and building code. This includes:

Timestamp are parsed now and forwarded to the dial plan and the manager
SMS-STATUS-REPORT PDUs can be processed from now on

To handle the status reports correctly, this PR creates a database row for each message part of a concatenated SM (i.e. for each single SM) including the message reference returned from the modem. Once, a status report is received, a flag is updated in the database. When all reports are received, a "report" will be issued to the dial plan. This PR also gives you the opportunity to pass a payload to each message to identify it when the delivery report is completed (useful for a Mail-to-SMS gateway).

I also included commit 3a85d35 @miopa to support sending to national numbers (e.g. 0049... instead of +49...) and 'fixed' the USSD sending functions.

Major things that changed:

The SMSDB file (I didn't include an migration script for #75, just do $ sudo rm /var/lib/asterisk/smsdb.sqlite)
You have to apply an identifier string to DongleSendSMS for the delivery reports
The manager events for delivery reports are handled differently from now on (I assume nobody used them, as they were just a complete hack)
This PR drops support for sending raw PDUs from the manager
Disable TEXT mode for sending and receiving (now PDU-only)

I hope there are some people that can test this PR, as it changes almost the whole SMS and USSD part of the project. I only tested with an E1762 and didn't used my code in 'production' yet.

@wdoekes: Thanks for reviewing #75 :)

magcks commented 4 years ago

..ah, and sorry for that large commit. I usually have that strange workflow that I code everything that is in my mind and afterwards I remove the compiler issues and do only fixup commits ;)

magcks commented 4 years ago

I think I've fixed all your comments. Let's wait for the feedback from #78.

magcks commented 4 years ago

Thanks for your comments. I dropped it, as I was too lazy to fully implement it (the initial version didn't parse the SCTS field). Theoretically there is no reason to use the text mode as SMs are always transmitted as PDUs. The text mode is only for users that use the modem directly via the TTY port.

Does your modem work correctly (does the PDU mode work)? It would be great if you can tell me which model you use and if voice, USSD and Multi-SMS (sending and receiving) works.

magcks commented 4 years ago

Use "static const" before:

channel_var_t vars[] =

I can do that. But static won't work as the content is basically not static, right?

hselasky commented 4 years ago

Use "static const" before: channel_var_t vars[] =

I can do that. But static won't work as the content is basically not static, right?

Isn't vars[] just an array of constants?

magcks commented 4 years ago

Nope, at least the payload for delivery reports is always included and not constant.

hselasky commented 4 years ago

You're right. Now I see.

m7mdcc commented 4 years ago

I just come to say , Good job @magcks & @wdoekes

I have test the PR , there is issue let me explain it .. its seems same issue like https://github.com/wdoekes/asterisk-chan-dongle/pull/75#issuecomment-597015030

if you send concatenated message its will be store as 1 message , but if you send another short message after the concatenated message its will contain text from previous message "concatenated message" .

my dongle.conf is

`

[general] interval=15 [defaults] context=dongle-incoming group=0 rxgain=0 txgain=0 autodeletesms=yes resetdongle=yes u2diag=-1 usecallingpres=yes callingpres=allowed_passed_screen disablesms=no language=en mindtmfgap=45 mindtmfduration=80 mindtmfinterval=200 callwaiting=auto disable=no ;exten=+1234567890 dtmf=relax unaltered diaplan [dongle0] audio=/dev/ttyUSB1 data=/dev/ttyUSB2

`

wdoekes commented 4 years ago

So, is this ready to ship @magcks ?

Waiting for feedback from #78 ?

magcks commented 4 years ago

Thanks for your comment @m7mdcc . Can you please tell me the content of the PDU? You can log the content using ast_log(LOG_WARNING, "PDU: %s\n", str);

in at_parse.c#322

Does the second SM contain the complete content from the first SM or only parts?

Thanks! :)

m7mdcc commented 4 years ago

@magcks , do you want to add this line in 322 in file at_parse.c before compiling ? ` EXPORT_DEF int at_parse_cmgr(char str, size_t len, int tpdu_type, char sca, size_t sca_len, char oa, size_t oa_len, char scts, int mr, int st, char dt, char msg, size_t msg_len, pdu_udh_t udh) { ==> here ? / skip "+CMGR:" */ str += 6; len -= 6;

`

the second message contain only one part of first message .

magcks commented 4 years ago

Yes, please. It will print the PDU content as a warning to the asterisk console before parsing it.

m7mdcc commented 4 years ago

Ok here is the log , first i receive concatenated message then i receive short message which only contain "Test" , if you see the log its say : [2020-05-04 21:58:24] VERBOSE[2164] at_response.c: [dongle0] Got SMS part from +XXXXXXXXXXXX: 'Test'; Concatenated [ref,parts,order]: 0 1 2 its marked as Concatenated since its not .

[2020-05-04 21:58:21] VERBOSE[2164] at_response.c: [dongle0] Got Response for user's command:'+CMGR: 0,,159 0791695650309199400C916956854351020008025040228201218C05000325020106450631062D06280627002006450631062D06280627002006450631062D062806270020002006450631062D06280627002006450631062D06280627002006450631062D062806270020002000200020002006450631062D06280627002006450631062D06280627002006450631062D062806270020002006450631062D0628062700200645' [2020-05-04 21:58:21] NOTICE[2164] at_response.c: [dongle0] Got Response for user's command:'+CMGR: 0,,159 0791695650309199400C916956854351020008025040228201218C05000325020106450631062D06280627002006450631062D06280627002006450631062D062806270020002006450631062D06280627002006450631062D06280627002006450631062D062806270020002000200020002006450631062D06280627002006450631062D06280627002006450631062D062806270020002006450631062D0628062700200645' [2020-05-04 21:58:21] WARNING[2164] at_parse.c: PDU: +CMGR: 0,,159 0791695650309199400C916956854351020008025040228201218C05000325020106450631062D06280627002006450631062D06280627002006450631062D062806270020002006450631062D06280627002006450631062D06280627002006450631062D062806270020002000200020002006450631062D06280627002006450631062D06280627002006450631062D062806270020002006450631062D0628062700200645 [2020-05-04 21:58:21] VERBOSE[2164] at_response.c: [dongle0] Got SMS part from +XXXXXXXXXXXX: 'مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا م'; Concatenated [ref,parts,order]: 37 2 1 [2020-05-04 21:58:21] VERBOSE[2164] at_response.c: [dongle0] Waiting for following parts [2020-05-04 21:58:23] VERBOSE[2164] at_response.c: [dongle0] Got Response for user's command:'+CMGR: 0,,49 0791695650309199400C916956854351020008025040228211211E0500032502020631062D06280627002006450631062D0628062700200020' [2020-05-04 21:58:23] NOTICE[2164] at_response.c: [dongle0] Got Response for user's command:'+CMGR: 0,,49 0791695650309199400C916956854351020008025040228211211E0500032502020631062D06280627002006450631062D0628062700200020' [2020-05-04 21:58:23] WARNING[2164] at_parse.c: PDU: +CMGR: 0,,49 0791695650309199400C916956854351020008025040228211211E0500032502020631062D06280627002006450631062D0628062700200020 [2020-05-04 21:58:23] VERBOSE[2164] at_response.c: [dongle0] Got SMS part from +XXXXXXXXXXXX: 'رحبا مرحبا '; Concatenated [ref,parts,order]: 37 2 2 [2020-05-04 21:58:23] VERBOSE[2164] at_response.c: [dongle0] Got full SMS from +XXXXXXXXXXXX: 'مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا ' [2020-05-04 21:58:23] VERBOSE[2403][C-00000001] pbx.c: Executing [sms@dongle-incoming:1] Verbose("Local/sms@dongle-incoming-00000000;1", "Incoming SMS from +XXXXXXXXXXXX مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا ") in new stack [2020-05-04 21:58:23] VERBOSE[2403][C-00000001] app_verbose.c: Incoming SMS from +XXXXXXXXXXXX مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا [2020-05-04 21:58:23] VERBOSE[2403][C-00000001] pbx.c: Executing [sms@dongle-incoming:2] System("Local/sms@dongle-incoming-00000000;1", "echo '2020-05-04 21:58:23 - - +XXXXXXXXXXXX: مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا ' >> /var/log/asterisk/sms.txt") in new stack [2020-05-04 21:58:23] VERBOSE[2403][C-00000001] pbx.c: Executing [sms@dongle-incoming:3] Hangup("Local/sms@dongle-incoming-00000000;1", "") in new stack [2020-05-04 21:58:23] VERBOSE[2403][C-00000001] pbx.c: Spawn extension (dongle-incoming, sms, 3) exited non-zero on 'Local/sms@dongle-incoming-00000000;1' [2020-05-04 21:58:24] VERBOSE[2164] at_response.c: [dongle0] Got Response for user's command:'+CMGR: 0,,23 0791695650309199040C9169568543510200000250500085512104D4F29C0E' [2020-05-04 21:58:24] NOTICE[2164] at_response.c: [dongle0] Got Response for user's command:'+CMGR: 0,,23 0791695650309199040C9169568543510200000250500085512104D4F29C0E' [2020-05-04 21:58:24] WARNING[2164] at_parse.c: PDU: +CMGR: 0,,23 0791695650309199040C9169568543510200000250500085512104D4F29C0E [2020-05-04 21:58:24] VERBOSE[2164] at_response.c: [dongle0] Got SMS part from +XXXXXXXXXXXX: 'Test'; Concatenated [ref,parts,order]: 0 1 2 [2020-05-04 21:58:24] VERBOSE[2164] at_response.c: [dongle0] Got full SMS from +XXXXXXXXXXXX: 'Testحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا ' [2020-05-04 21:58:24] VERBOSE[2406][C-00000002] pbx.c: Executing [sms@dongle-incoming:1] Verbose("Local/sms@dongle-incoming-00000001;1", "Incoming SMS from +XXXXXXXXXXXX Testحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا ") in new stack [2020-05-04 21:58:24] VERBOSE[2406][C-00000002] app_verbose.c: Incoming SMS from +XXXXXXXXXXXX Testحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا [2020-05-04 21:58:24] VERBOSE[2406][C-00000002] pbx.c: Executing [sms@dongle-incoming:2] System("Local/sms@dongle-incoming-00000001;1", "echo '2020-05-04 21:58:24 - - +XXXXXXXXXXXX: Testحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا ' >> /var/log/asterisk/sms.txt") in new stack [2020-05-04 21:58:24] VERBOSE[2406][C-00000002] pbx.c: Executing [sms@dongle-incoming:3] Hangup("Local/sms@dongle-incoming-00000001;1", "") in new stack [2020-05-04 21:58:24] VERBOSE[2406][C-00000002] pbx.c: Spawn extension (dongle-incoming, sms, 3) exited non-zero on 'Local/sms@dongle-incoming-00000001;1

magcks commented 4 years ago

Okay, thanks. Looks like an uninitialized value. I will have a look.

magcks commented 4 years ago

Could you please check now. The code at least compiles now, but I currently don't have time to test and I don't know where I put my sticks :D

m7mdcc commented 4 years ago

@magcks , Thanks .. I can confirm issue is solved now .

magcks commented 4 years ago

@m7mdcc thank you very much for testing :) @wdoekes alrighty. in this case it should be ready to merge.

wdoekes commented 4 years ago

Can you get these conflicts resolved? Thanks!

magcks commented 4 years ago

I resolved the conflicts, but didn't test if #79 still works. Nevertheless, I recommend to merge this PR as soon as possible, as there are serious bugs (smsdb threads and string termination) in #75 that are fixed with this PR.

hselasky commented 4 years ago

Maybe you could squash that last change commit. Minor nit though.

wdoekes / asterisk-chan-dongle

Reworked PDU parsing; Delivery reports #77