SIM800 support?

[thijstriemstra]

I've recently bought one of these and now I'm looking for a Pythonic way to control it. I stumbled upon this project and I'm wondering if this is supported in your library, how it can be supported.

http://www.adafruit.com/datasheets/sim800h_hardware_design_v1.00.pdf

[thijstriemstra]

Looks like Python 3 isn't supported because twisted.internet.serialport isn't ported yet. I've opened ticket #8099 for that issue.

UPDATE: ticket is fixed, in trunk and part of 15.5 release (or 16.0 if 15.4 was last for this year).

[smn]

To be very honest with you, I don't know. The AT commands I'm using for this project are quite "standard" but I can't guarantee that it'll work. Would love for you to try out and let me know how far you get and what would need to be done to get it working.

[thijstriemstra]

Thanks for the response. I'll check if I can get the example C code working (https://github.com/itead/ITEADSW_GSM) and move to txgsm from there.

[thijstriemstra]

After examining the code it looks like the dispatch object in txgsm.service maps the various subcommands to methods on the service maker itself.

dispatch = {
    'send-sms': self.send_sms,
    'list-sms': self.list_sms,
    'ussd-session': self.ussd_session,
    'probe-modem': self.probe_modem,
}

I haven't tested my particular modem yet but after looking at the AT commands there's bound to be a difference and the commands you've hardcoded won't work for my modem.

What do you suggest for making this more modular so it can have different backends? That way a 'sim800 backend' can be contributed. The current implemented methods can be moved to a 'fallback' or default backend in that case. Thoughts?

[smn]

I'd suggest a way to load a backend that implements one or more defined zope.interfaces and have the rest of the system just work through the interfaces. The current backend I have is simply whatever was needed to get it to work with my South African 3G dongle for Vodacom, another external USB based GSM modem and Truteq's GSM CommServ. I have no idea what chipsets those use but I'm happy to find out and split it out into pluggable backends.

[smn]

I'd love a sim800 backend contributed, happy to give you commit access to the repository. I'd like to add contributing.md & AUTHORS files before I do though. Would you be happy to commit to adding the backend? I'm happy to do code reviews.

[thijstriemstra]

@smn I'm definitely interested in adding such a backend. I'll start with adding a Sphinx doc project for documenting tested modems etc.

[smn]

awesome, you now have push permissions to this repository. I've created http://txgsm.rtfd.org, if you send me your readthedocs account I can add you as a collaborator there too.

[thijstriemstra]

thanks @smn. My username there is the same as here.

[smn]

Added you as a maintainer

[thijstriemstra]

Basically a backend consists of a bunch of AT commands that are accessible using the command line and I want to be able to programmatically call them. Perhaps you can create a simple backend interface for your existing code?

[smn]

I can certainly look at doing that. Ideally it would be a matter of referencing AT commands but in practice I'm afraid it's going to be more complex than that. My gut feel is that it won't just be mapping of what commands to send to a modem but also how to parse the response and that's probably where the most work is needing to be done.

[thijstriemstra]

To give you an idea, here's some of the stuff sim800 supports:

• answering and making calls
• read/write to phonebook
• require and change simcard PIN
• FM radio
• delete/store/send SMS
• delete/store/send MMS
• jamming detection
• antenna detection
• text-to-speech
• play/record audio file
• FTP/HTTP/UDP server and client
• POP3 Email
• GSM location

Here's the list of AT commands: SIM800 Series_AT Command Manual_V1.09.pdf

[smn]

wow, that's quite a bit more than what I've dabbled in so far.

Just a heads up, I'm away for a couple of days and then travelling to west africa for the whole of next week. It'll be a while before I can start digging into this.

[thijstriemstra]

Finally got to test the raspberrypi modem, first command get a response, 2nd doesn't. Investigating.

$ twistd -n txgsm --verbose --device=/dev/ttyAMA0 probe-modem
2015-11-12 13:26:52+0100 [-] Log opened.
2015-11-12 13:26:52+0100 [-] twistd 15.4.0 (/home/pi/.virtualenvs/uboat-server/bin/python2 2.7.9) starting up.
2015-11-12 13:26:52+0100 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2015-11-12 13:26:52+0100 [-] Sending: 'ATE0'
2015-11-12 13:26:52+0100 [-] Received: {'command': ['ATE0'], 'expect': 'OK', 'response': ['OK']}
2015-11-12 13:26:52+0100 [-] Sending: 'AT+CIMI'

[thijstriemstra]

The AT+CIMI request doesn't work yet, likely because I took out the sim-card.. When I skip that command and request AT+CGMM, the 3rd call of probe-modem, it returns:

2015-11-12 13:35:41+0100 [-] Received: {'command': ['AT+CGMM'], 'expect': 'OK', 'response': ['SIMCOM_SIM800', 'OK']}

Nice!

[thijstriemstra]

I've tested lots of AT commands, results are here: https://gist.github.com/thijstriemstra/25a078663fe031efc474

[thijstriemstra]

I've been going through the AT documentation to figure out what we need in terms of API. I'll write up something later today.

[thijstriemstra]

Text AT Commands

There are 4 types of commands:

Command	Command	Description
Test	AT+<x>=?	The mobile equipment returns the list of parameters and value ranges set with the corresponding Write Command or by internal process.
Read	AT+<x>?	This command returns the currently set value of the parameter or parameters.
Write	AT+<x>=<...>	This command sets the user-definable parameter.
Execute	AT+<x>	The execution command reads non-variable parameters affected by internal processes in the GSM engine.

Take for example the TTS (text-to-speech) operation 18.2.2 AT+CTTSPARAM on page 341 of the attached PDF.

It's test command (AT+CTTSPARAM=?) takes no parameters and returns a list of ranges:

+CTTSPARAM: (0-100),(0-3),(1-100),(1-100),(0,1)

The read command (AT+CTTSPARAM?) takes no parameters and returns the values:

+CTTSPARAM: <volume>,<mode>,<pitch>,<speed>,<channel>

The write command (AT+CTTSPARAM=<volume>,<mode>,<pitch>,<speed>[,<channel>]) takes 5 parameters (one optional) and returns nothing of interest.

AMP AT Commands

So how do we async test/read/write with these methods? Perhaps we can use Twisted's AMP for this..

Let's take the test command from earlier. It returns the ranges for the various parameters. The AMP Command below allows us to structure the response of our AT command:

from twisted.protocols.amp import String, Command

class PlaybackSettingsTest(Command):
    response = [
        ('volumeRange', String()),
        ('modeRange', String()),
        ('pitchRange', String()),
        ('speedRange', String()),
        ('channelRange', String())
    ]

For every command we also add a responder:

from twisted.protocols.amp import CommandLocator

class TTS(CommandLocator):

    @PlaybackSettingsTest.responder
    def playbackSettingsTest(self):
        commandName = 'AT+CTTSPARAM=?'

        # XXX: call AT command
        response = '+CTTSPARAM: (0-100),(0-3),(1-100),(1-100),(0,1)'

        # XXX: parse AT command result

        # XXX: return AT command result
        return {'volumeRange': '(0-100)', 'modeRange': '(0-3)',
            'pitchRange': '(1-100)', 'speedRange': '(1-100)',
            'channelRange': '(0,1)'}

API for test command

We can now use an AMP client and server to communicate with the GSM modem.

First the server that use the custom TTS locator for the AMP protocol:

from twisted.protocols import amp

from txgsm.backend.sim800.tts.commands import TTS

def main():
    from twisted.internet import reactor
    from twisted.internet.protocol import Factory
    pf = Factory()
    pf.protocol = lambda: amp.AMP(locator=TTS())
    reactor.listenTCP(1234, pf)
    print('started')
    reactor.run()

if __name__ == '__main__':
    main()

And the AMP client that calls the test method:

from twisted.internet import reactor, defer, endpoints
from twisted.internet.endpoints import TCP4ClientEndpoint, connectProtocol
from twisted.protocols.amp import AMP

from txgsm.backend.sim800.tts.commands import PlaybackSettingsTest

def doOperations():
    destination = TCP4ClientEndpoint(reactor, '127.0.0.1', 1234)

    # playbacksettings test
    playbackSettingsTest = connectProtocol(destination, AMP())
    def connected(ampProto):
        return ampProto.callRemote(PlaybackSettingsTest)
    playbackSettingsTest.addCallback(connected)

    def done(result):
        print('TTS operation: {0}'.format(result))
        reactor.stop()

    operations = [playbackSettingsTest]
    defer.DeferredList(operations).addCallback(done)

if __name__ == '__main__':
    doOperations()
    reactor.run()

Output on the client:

TTS operation: [(True, {'modeRange': '(0-3)', 'speedRange': '(1-100)', 'pitchRange': '(1-100)', 'channelRange': '(0,1)', 'volumeRange': '(0-100)'})]

API for write command

The write command defines the arguments:

class PlaybackSettingsWrite(Command):
    arguments = [
        ('volume', Integer()),
        ('mode', Integer()),
        ('pitch', Integer()),
        ('speed', Integer()),
        ('channel', Integer(optional=True))
    ]
    requiresAnswer = False

And a responder would look something like this:

@PlaybackSettingsWrite.responder
def playbackSettingsWrite(self, volume, mode, pitch, speed, channel=None):
    commandName = 'AT+CTTSPARAM= <volume>,<mode>,<pitch>,<speed>[,<channel>]'

    # XXX: call AT command
    cmd = '{0}{1},{2}'.format(commandName, volume, mode) # etc..
    print('running: {0}'.format(cmd))

    # XXX: parse AT command result

    # no interesting result to return
    return {}

And added to the client:

from txgsm.backend.sim800.tts.commands import PlaybackSettingsWrite

settingsWrite = connectProtocol(destination, AMP())
def connected(ampProto):
    return ampProto.callRemote(PlaybackSettingsWrite, volume=50, mode=0 # etc..
settingsWrite.addCallback(connected)

Conclusion

So AMP provides:

• a way to structure method arguments, responses and error handling
• ability to remotely communicate securely (TLS built-in) with the GSM modem (even though the server/client part can, and should be, optional IMO so you can still use it as a single commandline tool)

What do you think?

[smn]

I like your thinking, I'm working on ways of making the sending of commands & catching the relevant response more reliable (which would help with the parsing). I'm not entirely sure yet about the idea of using Amp though. I like the Command pattern for this but I'm not yet seeing what problem we're solving by adding a RPC-ish network layer between the modem and the rest of the application.

I'm in Nigeria this week with limited Internet connectivity, I'll try and commit my stuff shortly and turn it into a PR so you can see the approach I'm taking.

[thijstriemstra]

True, but being able to add AMP support for those messages would be very nice. I'll also push my branch where I've documented all the commands for sim800 and split them into modules.

[thijstriemstra]

I've added some modules, for example the error codes: https://github.com/thijstriemstra/txgsm/blob/sim800/txgsm/backend/sim800/errors.py

And implemented the signatures of some of the commands for TTS for example: https://github.com/thijstriemstra/txgsm/blob/sim800/txgsm/backend/sim800/tts/commands.py

I think I'll keep sketching out those commands using twisted's Command for now, I have a feeling we'll have a system where I'll need all of this info anyway (method/param signature, command response structure etc) so then I'll port these commands again.

[thijstriemstra]

I've been looking around the web for other AT command parser implementations and found one for Android from 2007 that looks promising. My idea is to create a standalone AT command parser library, with no dependencies, that can also be used by other projects like txgsm.

I'll try to commit the ported Python source later today.

[smn]

Nice! Looking forward.

[thijstriemstra]

Pushed it to https://github.com/collab-project/atcmd @smn.