Open gonsalez-ru opened 3 years ago
I bisected this problem because occasionally I use USSD to load prepaid cards, the problem appears for me in commit 868bbd41fbf2911cdfba483e50c8e99a79d84258 by @magcks
Can you verify? Last good: 1d00a5d3c31dc5b61541cd6cbc4152e58bb5c898 May 4 First bad: 868bbd41fbf2911cdfba483e50c8e99a79d84258 May 5 1e5b916613e973e41d9f22e901c2739090a705f3 and b18ca62539834a850e7cc2a7bdc343d3ca573cc8 from May 14 have loading problems 71e3206122a36f45f097a712a3c8cb611db365c4 ff. also bad.
This commit should not have happened imho in this form, much too large and complex for a single commit. Always split up such changes in multiple steps to make it easier to bisect and diagnose problems.
The trouble could be that the module sends this
AT+CUSD=1,"AA182C3602",15
while the old module sent this
AT+CUSD=1,"002A0031003000310023",15
Both signifying *101# as a simple USSD code. Try this patch which forces UCS-2 for USSD:
diff --git a/at_command.c b/at_command.c
index 4882d82..408691e 100644
--- a/at_command.c
+++ b/at_command.c
@@ -335,36 +335,26 @@ EXPORT_DEF int at_enqueue_ussd(struct cpvt *cpvt, const char *code)
static const char cmd_end[] = "\",15\r";
at_queue_cmd_t at_cmd = ATQ_CMD_DECLARE_DYN(CMD_AT_CUSD);
ssize_t res;
- int length;
- char buf[4096];
+ char buf[160*4+sizeof(cmd)+sizeof(cmd_end)+1];
+
+ int length = STRLEN(cmd);
+ memcpy (buf, cmd, length);
- memcpy (buf, cmd, STRLEN(cmd));
- length = STRLEN(cmd);
int code_len = strlen(code);
+ if (code_len == 0 || code_len > 80) { // UCS-2 encoding
+ chan_dongle_err = E_INVALID_USSD;
+ return -1;
+ }
- // use 7 bit encoding. 15 is 00001111 in binary and means 'Language using the GSM 7 bit default alphabet; Language unspecified' accodring to GSM 23.038
- uint16_t code16[code_len * 2];
- uint8_t code_packed[4069];
- res = utf8_to_ucs2(code, code_len, code16, sizeof(code16));
+ res = utf8_to_ucs2(code, code_len, buf+length, sizeof(buf)-sizeof(cmd)-sizeof(cmd_end));
if (res < 0) {
chan_dongle_err = E_PARSE_UTF8;
return -1;
}
- res = gsm7_encode(code16, res, code16);
- if (res < 0) {
- chan_dongle_err = E_ENCODE_GSM7;
- return -1;
- }
- res = gsm7_pack(code16, res, code_packed, sizeof(code_packed), 0);
- if (res < 0) {
- chan_dongle_err = E_PACK_GSM7;
- return -1;
- }
- res = (res + 1) / 2;
- hexify(code_packed, res, buf + STRLEN(cmd));
- length += res * 2;
- memcpy(buf + length, cmd_end, STRLEN(cmd_end)+1);
+ hexify(buf+length, res*2, buf+length);
+ length += res * 4;
+ memcpy(buf+length, cmd_end, STRLEN(cmd_end)+1);
length += STRLEN(cmd_end);
at_cmd.length = length;
@@ -378,6 +368,7 @@ EXPORT_DEF int at_enqueue_ussd(struct cpvt *cpvt, const char *code)
chan_dongle_err = E_QUEUE;
return -1;
}
+
return 0;
}
diff --git a/at_response.c b/at_response.c
index 125a688..41a8990 100644
--- a/at_response.c
+++ b/at_response.c
@@ -1456,12 +1456,12 @@
typebuf[0] = type + '0';
typebuf[1] = 0;
- // sanitize DCS
- if (dcs & 0x40) {
+ // FIXME: strictly check USSD encoding and detect encoding
+ if (dcs >=0 && dcs & 0x40) {
dcs = (dcs & 0xc) >> 2;
if (dcs == 3) dcs = 0;
} else {
- dcs = 0;
+ dcs = 2;
}
ast_verb (1, "[%s] USSD DCS=%d (0: gsm7, 1: ascii, 2: ucs2)\n", PVT_ID(pvt), dcs);
@bsdice
If you try:
AT+CUSD=1,"AA182C3602",0
With 0 or 1 or 2 or 3, it should also work, right?
I assume that 15
will mean UCS2 (+8bit +some other info..), so then sending 7bit is probably wrong when DCS is 15.
Docs from 27007-g50.zip ( https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=1515 ):
+CUSD=[<n>[,<str>[,<dcs>]]]
<n>: integer type (sets/shows the result code presentation status to the TE).
0 disable the result code presentation to the TE
1 enable the result code presentation to the TE
<str>: string type USSD‑string (when <str> parameter is not given,
network is not interrogated):
‑ if <dcs> indicates that 3GPP TS 23.038 [25] 7 bit default
alphabet is used:
‑ if TE character set other than "HEX" (refer command select TE
character set +CSCS): MT/TA converts GSM alphabet into current TE
character set according to rules of 3GPP TS 27.005 [24] Annex A
- if TE character set is "HEX": MT/TA converts each 7‑bit character of
GSM alphabet into two IRA character long hexadecimal number (e.g.
character (GSM 23) is presented as 17 (IRA 49 and 55))
‑ if <dcs> indicates that 8‑bit data coding scheme is used: MT/TA
converts each 8‑bit octet into two IRA character long hexadecimal number
(e.g. octet with integer value 42 is presented to TE as two characters
2A (IRA 50 and 65))
- if <dcs> indicates that 16-bit data coding scheme (UCS2) is used:
MT/TA splits the 16 bits into two 8-bit octets. Each of those octets
are converted as per the 8-bit data coding scheme, with the most
significant octet first (e.g. decimal value 4906 is presented to TE as
four characters 132A (IRA 49, 51, 50 and 65))
<dcs>: integer type (shows Cell Broadcast Data Coding Scheme, see
3GPP TS 23.038 [25]). Default value is 0.
See also:
/* bits 3..2 */
#define PDU_DCS_ALPHABET_SHIFT 2
#define PDU_DCS_ALPHABET_7BIT (0x00 << PDU_DCS_ALPHABET_SHIFT)
#define PDU_DCS_ALPHABET_8BIT (0x01 << PDU_DCS_ALPHABET_SHIFT)
#define PDU_DCS_ALPHABET_UCS2 (0x02 << PDU_DCS_ALPHABET_SHIFT)
#define PDU_DCS_ALPHABET_MASK (0x03 << PDU_DCS_ALPHABET_SHIFT)
#define PDU_DCS_ALPHABET(dcs) ((dcs) & PDU_DCS_ALPHABET_MASK)
and
/* Apparently bits 0..3 are not reserved anymore:
* bits 3..2: {7bit, 8bit, ucs2, undef} */
alphabet = PDU_DCS_ALPHABET(dcs);
/* Bits 3..2 set to 11 is reserved, but
* according to 3GPP TS 23.038 v14.0.0 (2017-03)
* for HIGH 1111 bit 3 (regardless of bit 2) is
* reserved. */
if (alphabet == PDU_DCS_ALPHABET_MASK) {
reserved = 1;
}
/* if 0x1 || 0xF then (dsc_lo & 3): {
* class0, class1-ME-specific,
* class2-SIM-specific,
* class3-TE-specific (3GPP TS 27.005)} */
This commit should not have happened imho in this form, much too large and complex for a single commit. Always split up such changes in multiple steps to make it easier to bisect and diagnose problems.
But that changeset contained a bunch of good stuff too. It's always hard to make people adhere to best practices when no one is getting paid. I hereby welcome others to review changes too :grin:
But that changeset contained a bunch of good stuff too. It's always hard to make people adhere to best practices when no one is getting paid. I hereby welcome others to review changes too 😁
I totally get what you are saying. But did you ask? Younger programmers imho just don't know that splitting changes into smallish chunks aids in debugging immensely. Would have saved me hours yesterday. 😅
If you don't mind me asking, why are you keeping this fork current, if you don't even have hardware and are not using it?
If you try:
AT+CUSD=1,"AA182C3602",0
With 0 or 1 or 2 or 3, it should also work, right?
I tried, but I can't tell, because my German provider has likely locked me out. For, I guess, sending too many USSD queries to prevent brute-forcing prepaid cash token numbers? Hope they reset this at some point because I have zero hope of reaching any competent technician at that company.
Yesterday USSD started working again for me after switching to UCS-2 format for send, and pretty much forcing UCS-2 for receive. So I am keeping my patch from above active on Asterisk 13.37 (what a nice number).
Worthless dialog below. I also tried 0..3. Hardware is a Huawei K3765 11.126.03.09.00 USB 2G/3G dongle.
OK
AT^CURC=0
OK
AT+CFUN=1
OK
AT^U2DIAG=0
OK
AT^SYSCFG=13,1,2000000400380,1,3
OK
AT+CUSD=1,"AA182C3602",15
OK
+CUSD: 2
AT+CUSD=1,"002A0031003000310023",15
OK
+CUSD: 2```
I totally get what you are saying. But did you ask? Younger programmers imho just don't know that splitting changes into smallish chunks aids in debugging immensely. Would have saved me hours yesterday.
I don't recall. And I'm not making this a blame game. If you don't have infinite time and don't have infinite goodwill, you sometimes have to accept what's there.
If you don't mind me asking, why are you keeping this fork current, if you don't even have hardware and are not using it?
That is indeed a good question :thinking: Fear of stale projects? I should probably abandon this.. do you want it?
Fear of stale projects? I should probably abandon this.. do you want it?
Uhh can't. Many reasons. But thanks. :) If you are serious though, one of the past committers will surely take it. Possibly write a note of this in README.md. Or just wait until somebody else forks it and starts to pull in patches from others.
For reference to future readers I rolled back to 1d00a5d3c31dc5b61541cd6cbc4152e58bb5c898 again and USSD works again. So my fix is worthless or not 100%. Or my USB stick is special aka firmware oddity. I vote to keep this open, maybe somebody will stumble onto this with time and energy to track the difference on the wire down.
Some progress... maybe even insights:
I compiled interceptty and played around with sending USSD manually using screen. Couldn't get interceptty to work with the running Asterisk module itself though, only screen. My USB stick (Vodafone=Huawei K3765) powers up with its character-set set to "IRA", instead of "GSM" or "UCS2". In this state it will accept ASCII text as USSD and will return ASCII as answer from the network (even with German ASCII umlauts in > 0x7F range). But it won't accept UCS2 or GSM7 in this mode. It will also not accept GSM7 USSD text when character-set is switched to GSM, just ASCII (in IRA mode) or UCS2 (in UCS2 mode). Nuts. I extended my patch to reintroduce the AT+CSCS command in at_command.c line 19 ff so GSM is preset at startup (it used to be UCS2).
Just setting everything to UCS2 on startup will probably collide with the current SMS code. Instead I emit a change to UCS2 command just before an USSD message is sent. Initially I immediately switched back to GSM right after the USSD message is sent, but this causes the USSD reply from the network to be ignored. I haven't found out why. Instead, I reset the character-set back to GSM in the USSD response handler. Might be a problem if you send something else like SMS within the 3 seconds or so it takes for the network to reply to the USSD request.
Pretty sure this all needs a review, also I am positive there is a more elegant fix to this problem code-wise. Going back to UCS2 encoding for everything would of course be wasteful for SMS, because every character will then be 2 bytes, reducing text space to 140 SMS size minus 6 bytes user data header / 2 = 134/2 = 67 text characters per SMS.
With these changes USSD works again here. 🍰
diff --git a/at_command.c b/at_command.c
index 4882d82..b704ae0 100644
--- a/at_command.c
+++ b/at_command.c
@@ -87,7 +87,7 @@ static int __attribute__ ((format(printf, 2, 3))) at_fill_generic_cmd (at_queue_
* \return 0 on success
*/
-static int __attribute__ ((format(printf, 4, 5))) at_enqueue_generic(struct cpvt *cpvt, at_cmd_t cmd, int prio, const char *format, ...)
+EXPORT_DEF int __attribute__ ((format(printf, 4, 5))) at_enqueue_generic(struct cpvt *cpvt, at_cmd_t cmd, int prio, const char *format, ...)
{
va_list ap;
int rv;
@@ -132,7 +132,7 @@ EXPORT_DEF int at_enqueue_initialization(struct cpvt *cpvt, at_cmd_t from_comman
// static const char cmd18[] = "AT+CLIP=0\r";
static const char cmd19[] = "AT+CSSN=1,1\r";
static const char cmd20[] = "AT+CMGF=0\r";
- static const char cmd21[] = "AT+CSCS=\"UCS2\"\r";
+ static const char cmd21[] = "AT+CSCS=\"GSM\"\r";
static const char cmd22[] = "AT+CPMS=\"SM\",\"SM\",\"SM\"\r";
static const char cmd23[] = "AT+CNMI=2,1,0,2,0\r";
@@ -164,7 +164,7 @@ EXPORT_DEF int at_enqueue_initialization(struct cpvt *cpvt, at_cmd_t from_comman
ATQ_CMD_DECLARE_ST(CMD_AT_CSSN, cmd19), /* activate Supplementary Service Notification with CSSI and CSSU */
ATQ_CMD_DECLARE_ST(CMD_AT_CMGF, cmd20), /* Set Message Format */
-// ATQ_CMD_DECLARE_STI(CMD_AT_CSCS, cmd21), /* UCS-2 text encoding */
+ ATQ_CMD_DECLARE_STI(CMD_AT_CSCS, cmd21), /* GSM text encoding */
ATQ_CMD_DECLARE_ST(CMD_AT_CPMS, cmd22), /* SMS Storage Selection */
/* pvt->initialized = 1 after successful of CMD_AT_CNMI */
@@ -335,36 +335,26 @@ EXPORT_DEF int at_enqueue_ussd(struct cpvt *cpvt, const char *code)
static const char cmd_end[] = "\",15\r";
at_queue_cmd_t at_cmd = ATQ_CMD_DECLARE_DYN(CMD_AT_CUSD);
ssize_t res;
- int length;
- char buf[4096];
+ char buf[160*4+sizeof(cmd)+sizeof(cmd_end)+1];
+
+ int length = STRLEN(cmd);
+ memcpy (buf, cmd, length);
- memcpy (buf, cmd, STRLEN(cmd));
- length = STRLEN(cmd);
int code_len = strlen(code);
+ if (code_len == 0 || code_len > 80) { // UCS-2 encoding
+ chan_dongle_err = E_INVALID_USSD;
+ return -1;
+ }
- // use 7 bit encoding. 15 is 00001111 in binary and means 'Language using the GSM 7 bit default alphabet; Language unspecified' accodring to GSM 23.038
- uint16_t code16[code_len * 2];
- uint8_t code_packed[4069];
- res = utf8_to_ucs2(code, code_len, code16, sizeof(code16));
+ res = utf8_to_ucs2(code, code_len, buf+length, sizeof(buf)-sizeof(cmd)-sizeof(cmd_end));
if (res < 0) {
chan_dongle_err = E_PARSE_UTF8;
return -1;
}
- res = gsm7_encode(code16, res, code16);
- if (res < 0) {
- chan_dongle_err = E_ENCODE_GSM7;
- return -1;
- }
- res = gsm7_pack(code16, res, code_packed, sizeof(code_packed), 0);
- if (res < 0) {
- chan_dongle_err = E_PACK_GSM7;
- return -1;
- }
- res = (res + 1) / 2;
- hexify(code_packed, res, buf + STRLEN(cmd));
- length += res * 2;
- memcpy(buf + length, cmd_end, STRLEN(cmd_end)+1);
+ hexify(buf+length, res*2, buf+length);
+ length += res * 4;
+ memcpy(buf+length, cmd_end, STRLEN(cmd_end)+1);
length += STRLEN(cmd_end);
at_cmd.length = length;
@@ -374,10 +364,15 @@ EXPORT_DEF int at_enqueue_ussd(struct cpvt *cpvt, const char *code)
return -1;
}
+ if (at_enqueue_generic(cpvt, CMD_AT_CSCS, 0, "AT+CSCS=\"UCS2\"\r") != 0) {
+ chan_dongle_err = E_QUEUE;
+ return -1;
+ }
if (at_queue_insert(cpvt, &at_cmd, 1, 0) != 0) {
chan_dongle_err = E_QUEUE;
return -1;
}
+
return 0;
}
diff --git a/at_command.h b/at_command.h
index 32535d6..0b5b81d 100644
--- a/at_command.h
+++ b/at_command.h
@@ -123,5 +123,6 @@ EXPORT_DECL int at_enqueue_activate(struct cpvt *cpvt);
EXPORT_DECL int at_enqueue_flip_hold(struct cpvt *cpvt);
EXPORT_DECL int at_enqueue_conference(struct cpvt *cpvt);
EXPORT_DECL void at_hangup_immediality(struct cpvt *cpvt);
+EXPORT_DECL int __attribute__ ((format(printf, 4, 5))) at_enqueue_generic(struct cpvt *cpvt, at_cmd_t cmd, int prio, const char *format, ...);
#endif /* CHAN_DONGLE_AT_SEND_H_INCLUDED */
diff --git a/at_response.c b/at_response.c
index 125a688..541459c 100644
--- a/at_response.c
+++ b/at_response.c
@@ -140,9 +140,8 @@ static int at_response_ok (struct pvt* pvt, at_res_t res)
break;
case CMD_AT_CSCS:
- ast_debug (1, "[%s] UCS-2 text encoding enabled\n", PVT_ID(pvt));
-
- pvt->use_ucs2_encoding = 1;
+ ast_debug (1, "[%s] Text encoding set\n", PVT_ID(pvt));
+ //pvt->use_ucs2_encoding = 1;
break;
case CMD_AT_CPMS:
@@ -1438,6 +1437,9 @@ static int at_response_cusd (struct pvt * pvt, char * str, size_t len)
char typebuf[2];
const char* typestr;
+
+ at_enqueue_generic(&pvt->sys_chan, CMD_AT_CSCS, 1, "AT+CSCS=\"GSM\"\r");
+
manager_event_message("DongleNewCUSD", PVT_ID(pvt), str);
if (at_parse_cusd (str, &type, &cusd, &dcs))
@@ -1456,12 +1458,12 @@ static int at_response_cusd (struct pvt * pvt, char * str, size_t len)
typebuf[0] = type + '0';
typebuf[1] = 0;
- // sanitize DCS
- if (dcs & 0x40) {
+ // FIXME: strictly check USSD encoding and detect encoding
+ if (dcs >=0 && dcs & 0x40) {
dcs = (dcs & 0xc) >> 2;
if (dcs == 3) dcs = 0;
} else {
- dcs = 0;
+ dcs = 2;
}
ast_verb (1, "[%s] USSD DCS=%d (0: gsm7, 1: ascii, 2: ucs2)\n", PVT_ID(pvt), dcs);
Thanks for the updates.
Okay. Calling
+ static const char cmd21[] = "AT+CSCS=\"GSM\"\r";
first in your case sounds good and portable.
Switching to back and from UCS2 feels icky if it can be avoided. If it must be switched, then it should be called before every other command that uses any encoding (set ucs2 before cusd, set gsm before sms).
And after setting GSM, I wonder if the data should be hexlify'd at all..
And have you tried ,32
instead of ,15
? See https://stackoverflow.com/questions/30125721/how-to-write-a-atcusd-ussd-command-to-support-maximum-handsets
P.S.
even with German ASCII umlauts in > 0x7F range
There is no >0x7F in ASCII. All 8bit-on characters uses some charset (latin1, cp437, cp850...).
I tried ,32 but all the stick would accept is ,15, no matter what CSCS is set to (IRA, GSM, UCS2).
Then I experimented with sending plain ASCII *102# while CSCS=GSM, but parsing here is totally broken. The string that comes back from my stick uses the GSM charset alright, together with some escapes for Euro-sign etc. But it is not GSM-7 encoded i.e. 7-bit characters shifted into 8-bit. Also my string is not "hexified", so whatever @magcks's stick is producing, this will never work with my stick. Only the UCS2 code path works.
While there I also discovered a serious parsing flaw in at_parse.c, function at_parse_cusd. If non-UCS2-text comes back from the network and it contains commas, the code will fall over and produce pruned strings, because it assumes commas are unique delimiters of fields, which they aren't if between double quotes. There was an earlier code from back in 2010 (the bug is this old) which works that I adapted:
EXPORT_DEF int at_parse_cusd (char* str, int * type, char** cusd, int * dcs)
{
/*
* parse cusd message in the following format:
* +CUSD: <m>,[<str>,<dcs>]
*
* examples
* +CUSD: 5
* +CUSD: 0,"100,00 EURO, valid till 01.01.2010, you are using tariff "Mega Tariff". More informations *111#.",15
*/
int state;
int len = strlen (str);
char* p = NULL;
size_t i;
*cusd = NULL;
*dcs = 0;
if (!sscanf(str, "+CUSD: %u,", type)) {
return -1;
}
for (i = 0, state = 0; i < len && state < 5; i++)
{
switch (state)
{
case 0:
if (str[i] == '"')
{
state++;
}
break;
case 1:
if (cusd)
{
*cusd = &str[i];
state++;
}
break;
case 2:
if (str[i] == '"')
{
str[i] = '\0';
state++;
}
break;
case 3:
if (str[i] == ',')
{
state++;
}
break;
case 4:
p = &str[i];
state++;
break;
}
}
if (state != 5)
{
return -1;
}
errno = 0;
*dcs = (unsigned char) strtol (p, (char**) NULL, 10);
if (errno == EINVAL)
{
return -1;
}
return 0;
}
As far as I see, the problem is: Different devices reqire different code implementation and we can't automaticaly find the correct way. Is it posible to leave previous implementation with posibility to turn it on at dongle.config?
P.S. Do you have any workaround ideas? e.g. custom dongle init string or switching dongle encoding mode by extra command at extention.conf ussd section?
Hello! Previous version worked fine at the same equipment, but now USSD answers are empty:
[dongle-incoming-ussd] exten => ussd,1,Noop(Incoming USSD: ${USSD} ${USSD_BASE64}) exten => ussd,n,Hangup()
[Sep 18 11:51:20] NOTICE[26464]: at_response.c:268 at_response_ok: [gsm1] Successfully sent USSD 0x76602ad8 [gsm1] USSD DCS=2 (0: gsm7, 1: ascii, 2: ucs2) [gsm1] Got USSD type 0 'USSD Notify': '' -- Executing [ussd@dongle-incoming:1] NoOp("Local/ussd@dongle-incoming-00000000;1", "Incoming USSD: AAAAAAAAAAAAAAAAAAA=") in new stack ...