Solution for not receiving duplicate inbound messages

[GoogleCodeExporter]

This is a solution for issue 473

Reading of SMS messages and deletion of the message should be done without 
interruption by some other thread.
The delete must be integrated within the code of readMessagesTEXT and 
readMessagesPDU.
I want to remain compatible with the past where the delete is done by the upper 
level application.
I propose to add a method setAutodeleteReceivedMsg (boolean flag) in the class 
AGateway.  Default the flag is false.
The application should set the flag to true.  In that case readMessagesTEXT and 
readMessagesPDU will do the autodelete.

Additions to AGateway.java
    /**
     * Sms messages should be deleted from the modem as soon as they have been read.
     * Forgetting to do so may lead to messages being returned to the application more than once
     * In order to remain compatible with previous versions of smslib the default setting is false 
     */
    private boolean autodeleteReceivedMsg = false;
    public boolean isAutodeleteReceivedMsg() {
        return autodeleteReceivedMsg;
    }

    public void setAutodeleteReceivedMsg(boolean autodeleteReceivedMsg) {
        this.autodeleteReceivedMsg = autodeleteReceivedMsg;
    }

Updated method readMessagesTEXT in class ModemGateway.java
My modifications are between // WIM STEVENS

    private void readMessagesTEXT(Collection<InboundMessage> msgList, MessageClasses msgClass, int myLimit) throws TimeoutException, GatewayException, IOException, InterruptedException
    {
        int i, j, memIndex;
        int limit = (myLimit < 0 ? 0 : myLimit);
        String response, line, tmpLine, msgText, originator, dateStr, refNo;
        BufferedReader reader;
        StringTokenizer tokens;
        InboundMessage msg;
        Logger.getInstance().logDebug("readMessagesTEXT called", null, getGatewayId());
        Calendar cal1 = Calendar.getInstance();
        Calendar cal2 = Calendar.getInstance();
        for (int ml = 0; ml < (getATHandler().getStorageLocations().length() / 2); ml++)
        {
            if (getATHandler().switchStorageLocation(getATHandler().getStorageLocations().substring((ml * 2), (ml * 2) + 2)))
            {
                response = getATHandler().listMessages(msgClass);
                response = response.replaceAll("\\s+OK\\s+", "\nOK");
                reader = new BufferedReader(new StringReader(response));
                for (;;)
                {
                    line = reader.readLine();
                    if (line == null) break;
                    line = line.trim();
                    if (line.length() > 0) break;
                }
                while (true)
                {
                    if (line == null) break;
                    if (line.length() <= 0 || line.equalsIgnoreCase("OK")) break;
                    i = line.indexOf(':');
                    j = line.indexOf(',');
                    memIndex = 0;
                    try
                    {
                        memIndex = Integer.parseInt(line.substring(i + 1, j).trim());
                    }
                    catch (NumberFormatException e)
                    {
                        // TODO: What to do here?
                        Logger.getInstance().logWarn("Incorrect Memory Index number parsed!", e, getGatewayId());
                    }
                    tokens = new StringTokenizer(line, ",");
                    tokens.nextToken();
                    tokens.nextToken();
                    tmpLine = "";
                    if (Character.isDigit(tokens.nextToken().trim().charAt(0)))
                    {
                        line = line.replaceAll(",,", ", ,");
                        tokens = new StringTokenizer(line, ",");
                        tokens.nextToken();
                        tokens.nextToken();
                        tokens.nextToken();
                        refNo = tokens.nextToken();
                        tokens.nextToken();
                        dateStr = tokens.nextToken().replaceAll("\"", "");
                        cal1.set(Calendar.YEAR, 2000 + Integer.parseInt(dateStr.substring(0, 2)));
                        cal1.set(Calendar.MONTH, Integer.parseInt(dateStr.substring(3, 5)) - 1);
                        cal1.set(Calendar.DAY_OF_MONTH, Integer.parseInt(dateStr.substring(6, 8)));
                        dateStr = tokens.nextToken().replaceAll("\"", "");
                        cal1.set(Calendar.HOUR_OF_DAY, Integer.parseInt(dateStr.substring(0, 2)));
                        cal1.set(Calendar.MINUTE, Integer.parseInt(dateStr.substring(3, 5)));
                        cal1.set(Calendar.SECOND, Integer.parseInt(dateStr.substring(6, 8)));
                        dateStr = tokens.nextToken().replaceAll("\"", "");
                        cal2.set(Calendar.YEAR, 2000 + Integer.parseInt(dateStr.substring(0, 2)));
                        cal2.set(Calendar.MONTH, Integer.parseInt(dateStr.substring(3, 5)) - 1);
                        cal2.set(Calendar.DAY_OF_MONTH, Integer.parseInt(dateStr.substring(6, 8)));
                        dateStr = tokens.nextToken().replaceAll("\"", "");
                        cal2.set(Calendar.HOUR_OF_DAY, Integer.parseInt(dateStr.substring(0, 2)));
                        cal2.set(Calendar.MINUTE, Integer.parseInt(dateStr.substring(3, 5)));
                        cal2.set(Calendar.SECOND, Integer.parseInt(dateStr.substring(6, 8)));
                        msg = new StatusReportMessage(refNo, memIndex, getATHandler().getStorageLocations().substring((ml * 2), (ml * 2) + 2), cal1.getTime(), cal2.getTime());
                        msg.setGatewayId(getGatewayId());
                        Logger.getInstance().logDebug("IN-DTLS: MI:" + msg.getMemIndex(), null, getGatewayId());
                        msgList.add(msg);
                        incInboundMessageCount();
                    }
                    else
                    {
                        line = line.replaceAll(",,", ", ,");
                        tokens = new StringTokenizer(line, ",");
                        tokens.nextToken();
                        tokens.nextToken();
                        originator = tokens.nextToken().replaceAll("\"", "");
                        tokens.nextToken();
                        dateStr = tokens.nextToken().replaceAll("\"", "");
                        cal1.set(Calendar.YEAR, 2000 + Integer.parseInt(dateStr.substring(0, 2)));
                        cal1.set(Calendar.MONTH, Integer.parseInt(dateStr.substring(3, 5)) - 1);
                        cal1.set(Calendar.DAY_OF_MONTH, Integer.parseInt(dateStr.substring(6, 8)));
                        dateStr = tokens.nextToken().replaceAll("\"", "");
                        cal1.set(Calendar.HOUR_OF_DAY, Integer.parseInt(dateStr.substring(0, 2)));
                        cal1.set(Calendar.MINUTE, Integer.parseInt(dateStr.substring(3, 5)));
                        cal1.set(Calendar.SECOND, Integer.parseInt(dateStr.substring(6, 8)));
                        msgText = "";
                        while (true)
                        {
                            tmpLine = reader.readLine();
                            if (tmpLine == null) break;
                            if (tmpLine.startsWith("+CMGL")) break;
                            if (tmpLine.startsWith("+CMGR")) break;
                            msgText += (msgText.length() == 0 ? "" : "\n") + tmpLine;
                        }
                        msgText = msgText.trim();
                        msg = new InboundMessage(cal1.getTime(), originator, msgText, memIndex, getATHandler().getStorageLocations().substring((ml * 2), (ml * 2) + 2));
                        msg.setGatewayId(getGatewayId());
                        Logger.getInstance().logDebug("IN-DTLS: MI:" + msg.getMemIndex(), null, getGatewayId());
                        msgList.add(msg);
                        incInboundMessageCount();
                    }
                    // Start Modifications by Wim Stevens
                    // Delete the received PDU from the modem if required and if valid memIndex was found
                    if ((memIndex != 0) && isAutodeleteReceivedMsg ( )) {
                        // do not switch memory location -> "--"
                        Logger.getInstance().logDebug("Autodeleting message at index: " + memIndex , null, getGatewayId());
                        deleteMessage(memIndex, "--");
                    }

                    // End Modifications by Wim Stevens
                    while (true)
                    {
                        //line = reader.readLine();
                        line = ((tmpLine == null || tmpLine.length() == 0) ? reader.readLine() : tmpLine);
                        if (line == null) break;
                        line = line.trim();
                        if (line.length() > 0) break;
                    }
                    if ((limit > 0) && (msgList.size() == limit)) break;
                }
                reader.close();
            }
        }
    }

Updated method readMessagesPDU

    private void readMessagesPDU(Collection<InboundMessage> msgList, MessageClasses messageClass, int myLimit) throws TimeoutException, GatewayException, IOException, InterruptedException
    {
        int i, j, memIndex, pduSize;
        String response, line, pduString;
        BufferedReader reader;
        Logger.getInstance().logDebug("readMessagesPDU called", null, getGatewayId());
        int limit = (myLimit < 0 ? 0 : myLimit);
        for (int ml = 0; ml < (getATHandler().getStorageLocations().length() / 2); ml++)
        {
            if (getATHandler().switchStorageLocation(getATHandler().getStorageLocations().substring((ml * 2), (ml * 2) + 2)))
            {
                response = getATHandler().listMessages(messageClass);

                response = response.replaceAll("\\s+OK\\s+", "\nOK");
                reader = new BufferedReader(new StringReader(response));
                for (;;)
                {
                    line = reader.readLine();
                    if (line == null) break;
                    line = line.trim();
                    if (line.length() > 0) break;
                }
                // use the parser to determine the message type
                PduParser parser = new PduParser();
                while (true)
                {
                    if (line == null) break;
                    line = line.trim();
                    if (line.length() <= 0 || line.equalsIgnoreCase("OK")) break;
                    if (line.length() <= 0 || line.equalsIgnoreCase("ERROR")) break;
                    i = line.indexOf(':');
                    j = line.indexOf(',');
                    memIndex = 0;
                    // Start modifications by Wim Stevens
                    // A line contains something like +CMGL: 1,1,,25
                    // first parameter is the memory index
                    // last parameter is the length of the PDU, not counting the addressing part
                    // the parser must always have an addressing part -> we will add it if required
                    try
                    {
                        memIndex = Integer.parseInt(line.substring(i + 1, j).trim());
                    }
                    catch (NumberFormatException e)
                    {
                        // TODO: What to do here?
                        Logger.getInstance().logWarn("Incorrect Memory Index number parsed!", e, getGatewayId());
                    }
                    i = line.lastIndexOf(',');
                    j = line.length();
                    pduSize = 0;
                    try {
                        pduSize = Integer.parseInt(line.substring(i + 1, j).trim());
                    }   catch (NumberFormatException e)
                    {
                        // TODO: What to do here?
                        Logger.getInstance().logWarn("Incorrect pdu size parsed!", e, getGatewayId());
                    }
                    pduString = reader.readLine().trim();
                    if ( (pduSize > 0) && ((pduSize*2) == pduString.length()) ) {
                        pduString = "00" + pduString;
                    }
                    // end of modifications by Wim Stevens
                    try
                    {
                        Logger.getInstance().logDebug("READ PDU: " + pduString, null, getGatewayId());
                        // this will throw an exception for PDUs
                        // it can't classify
                        Pdu pdu = parser.parsePdu(pduString);
                        // NOTE: maybe a message validity vs the current
                        //       date should be put here.
                        //       if the message is invalid, the message should
                        //       be ignored and but logged
                        if (pdu instanceof SmsDeliveryPdu)
                        {
                            Logger.getInstance().logDebug(pdu.toString(), null, getGatewayId());
                            InboundMessage msg;
                            String memLocation = getATHandler().getStorageLocations().substring((ml * 2), (ml * 2) + 2);
                            if (pdu.isBinary())
                            {
                                msg = new InboundBinaryMessage((SmsDeliveryPdu) pdu, memIndex, memLocation);
                                if (Service.getInstance().getKeyManager().getKey(msg.getOriginator()) != null) msg = new InboundEncryptedMessage((SmsDeliveryPdu) pdu, memIndex, memLocation);
                            }
                            else
                            {
                                msg = new InboundMessage((SmsDeliveryPdu) pdu, memIndex, memLocation);
                            }
                            msg.setGatewayId(getGatewayId());
                            Logger.getInstance().logDebug("IN-DTLS: MI:" + msg.getMemIndex() + " REF:" + msg.getMpRefNo() + " MAX:" + msg.getMpMaxNo() + " SEQ:" + msg.getMpSeqNo(), null, getGatewayId());
                            if (msg.getMpRefNo() == 0)
                            {
                                // single message
                                msgList.add(msg);
                                incInboundMessageCount();
                            }
                            else
                            {
                                // multi-part message
                                int k, l;
                                List<InboundMessage> tmpList;
                                InboundMessage listMsg;
                                boolean found, duplicate;
                                found = false;
                                for (k = 0; k < this.mpMsgList.size(); k++)
                                {
                                    // List of List<InboundMessage>
                                    tmpList = this.mpMsgList.get(k);
                                    listMsg = tmpList.get(0);
                                    // check if current message list is for this message
                                    if (listMsg.getMpRefNo() == msg.getMpRefNo())
                                    {
                                        duplicate = false;
                                        // check if the message is already in the message list
                                        for (l = 0; l < tmpList.size(); l++)
                                        {
                                            listMsg = tmpList.get(l);
                                            if (listMsg.getMpSeqNo() == msg.getMpSeqNo())
                                            {
                                                duplicate = true;
                                                break;
                                            }
                                        }
                                        if (!duplicate) tmpList.add(msg);
                                        found = true;
                                        break;
                                    }
                                }
                                if (!found)
                                {
                                    // no existing list present for this message
                                    // add one
                                    tmpList = new ArrayList<InboundMessage>();
                                    tmpList.add(msg);
                                    this.mpMsgList.add(tmpList);
                                }
                            }
                        }
                        else if (pdu instanceof SmsStatusReportPdu)
                        {
                            // Wim Stevens
                            Logger.getInstance().logDebug(pdu.toString(), null, getGatewayId());
                            // Wim Stevens
                            StatusReportMessage msg;
                            msg = new StatusReportMessage((SmsStatusReportPdu) pdu, memIndex, getATHandler().getStorageLocations().substring((ml * 2), (ml * 2) + 2));
                            msg.setGatewayId(getGatewayId());
                            msgList.add(msg);
                            incInboundMessageCount();
                        }
                        else
                        {
                            // this theoretically will never happen, but it occasionally does with phones 
                            // like some Sony Ericssons (e.g. Z610i, SENT messages are included in this list)
                            // instead of throwing a RuntimeException, just ignore any messages that are not of type
                            // SmsDeliveryPdu
                            // SmsStatusReportPdu
                            if (this.displayIllegalReceivedMessages)
                            {
                                Logger.getInstance().logError("Wrong type of PDU detected: " + pdu.getClass().getName(), null, getGatewayId());
                                Logger.getInstance().logError("ERROR PDU: " + pduString, null, getGatewayId());
                            }
                        }
                    }
                    catch (Exception e)
                    {
                        // PduFactory will give an exception
                        // for PDUs it can't understand
                        UnknownMessage msg;
                        msg = new UnknownMessage(pduString, memIndex, getATHandler().getStorageLocations().substring((ml * 2), (ml * 2) + 2));
                        msg.setGatewayId(getGatewayId());
                        msgList.add(msg);
                        incInboundMessageCount();
                        Logger.getInstance().logError("Unhandled SMS in inbox, skipping...", e, getGatewayId());
                        Logger.getInstance().logError("ERROR PDU: " + pduString, null, getGatewayId());
                    }
                    // Start Modifications by Wim Stevens
                    // Delete the received PDU from the modem if required and if valid memIndex was found
                    if ((memIndex != 0) && isAutodeleteReceivedMsg ( )) {
                        // do not switch memory location -> "--"
                        Logger.getInstance().logDebug("Autodeleting message at index: " + memIndex , null, getGatewayId());
                        deleteMessage(memIndex, "--");
                    }

                    // End Modifications by Wim Stevens
                    while (true)
                    {
                        line = reader.readLine();
                        if (line == null) break;
                        line = line.trim();
                        if (line.length() > 0) break;
                    }
                    if ((limit > 0) && (msgList.size() == limit)) break;
                }
                reader.close();
            }
        }
        checkMpMsgList(msgList);
    }

Original issue reported on code.google.com by wim.stev...@gmail.com on 20 Mar 2012 at 10:46

[GoogleCodeExporter]

Issue 473 has been merged into this issue.

Original comment by T.Delenikas on 22 Mar 2012 at 9:08

[GoogleCodeExporter]

WARNING, i don't think this is working 100%.
I currently have the AsyncMessageProcessor + InboundPollingThread (actually 
don't know if it's ok).
The messages are being received by the AsyncMessageProcessor, they are delete 
from card: 'Autodeleting message at index: 1'. BUT they are not stored in the 
Database.

Original comment by tiago.pe...@gmail.com on 28 Mar 2012 at 2:37

[GoogleCodeExporter]

The messages are only stored in when they are catched by the 
InboundPollingThread.
What i've made, was override the messagesReceived in Database.java, to validade 
that the message is unique, not stored in the database.

 IF NOT EXISTS (SELECT * FROM [dbo].[smsserver_in] WHERE [originator] = @originator AND [message_date] = @message_date )
    BEGIN   
    INSERT INTO [dbo].[smsserver_in] (...) VALUES (..)    
    END

Original comment by tiago.pe...@gmail.com on 28 Mar 2012 at 2:41

[GoogleCodeExporter]

--
The messages are being received by the AsyncMessageProcessor, they are delete 
from card: 'Autodeleting message at index: 1'. BUT they are not stored in the 
Database.
--

Hi - are you talking about the smsserver app?

Original comment by T.Delenikas on 28 Mar 2012 at 7:29

• Changed state: Open

[GoogleCodeExporter]

Yes, I'm running the SMSServer, with the parameter -Dsmslib.serial.polling
Since "SMSLib behaves as if no modem is connected."

I would appreciate your help, since this solution is able to delete partially 
received messages, that aren't being removed with the standard library.

Original comment by tiago.pe...@gmail.com on 29 Mar 2012 at 9:56

[GoogleCodeExporter]

Digging in the code, when using ModemGateway, the method:
  readMessages(Collection<InboundMessage> msgList, MessageClasses msgClass) 
is called in 2 places:
 - AModemDriver.java - AsyncMessageProcessor.run()
 - SMSServer.java - readMessages(), after which the Interfaces.messagesReceived() methods are called.

If the messages is caught in the first one, she won't be inserted in the 
DataBase.

Original comment by tiago.pe...@gmail.com on 29 Mar 2012 at 11:11

[GoogleCodeExporter]

OK, just found MY problem, indeed I'm using the SMSServer, and in the 3.5.1 
version the InboundMessageNotification, isn't used. So no treatment for the 
AsyncMessageProcessor.

Uncomented the line:
Service.getInstance().setInboundMessageNotification(this.inboundNotification);

and Added in the loadConfiguration():
gtw.getGateway().setAutodeleteReceivedMsg(true);

Now, those messages caught in the AsyncMessageProcessor and 
InboundPollingThread are treated (both call Interfaces.messagesReceived())

Thanks again for your help

Original comment by tiago.pe...@gmail.com on 29 Mar 2012 at 4:46

[GoogleCodeExporter]

Original comment by admin@smslib.org on 1 Jan 2014 at 9:25

• Changed state: Review

[GoogleCodeExporter]

Original comment by thana...@smslib.org on 26 Apr 2014 at 10:09

• Changed state: WontFix