Things get weird with 115200 baud

[CoolNamesAllTaken]

Commands work fine at 9600 baud, but something goes wonky at 115200 baud (none of the responses are received, and all I get is timeouts). My guess is that since the send and receive commands execute sequentially, the slight delay between send and receive at the higher baud rate causes something to get missed. It might be worth rearranging things slightly such that the script starts reading the serial port before the command is sent.

Thanks for making this script! Super helpful as both a tool and a reference :)

[CoolNamesAllTaken]

Things started going south for me on 9600 baud as well, but this fix did the trick. Things work great on 115200 baud now, too. I think that python stops monitoring the serial port once you exit a "with" statement, so things were getting lost in the gap between "send" and "watch".

def watch_and_send(cmd, timeout=10, success=None, failure=None, echo_cmd=None):
    with serial.Serial(PORT, BAUD, timeout=1) as ser:
        ser.write(cmd.encode('utf-8') + CMD_LINEBREAK)
        t_start = time.time()
        reply = list()
        while True:
            if ser.in_waiting:
                line = ser.readline()
                echo = False
                if echo_cmd:
                    echo = line.decode('utf-8').strip().endswith(echo_cmd)
                if line != CMD_LINEBREAK and not echo:
                    line = line.decode('utf-8').strip()
                    reply.append('\t' + line)
                    if success and line.startswith(success):
                        return ("Success", reply, time.time()-t_start)
                    if failure and line.startswith(failure):
                        return ("Error", reply, time.time()-t_start)
            if (time.time()-t_start) > timeout:
                return ("Timeout", reply, time.time()-t_start)
            time.sleep(0.02)

[CoolNamesAllTaken]

I'm going to keep this thread going with some more things I've run into:

• Restart function seems to crash things, at least for me (SIM7000A running 1351B04SIM7000A firmware).
• I had to add a time.sleep(5) after every certificate load operation for things to work properly, at least at 115200 baud. It seems like the modem was choking on too many certificates being uploaded with no pause in between, and was returning "+CME Error: operation not permitted". Once the delay was added after each certificate upload, things worked fine.
• The last command in the initialization sequence was looking at the wrong index in a reply string: cgcontdrp[1][0] should be cgcontdrp[1][1], since cgcontdrp[1][0] is the echo reply and contains no commas (was causing an issue with the split(",") command). This might not have been an issue before if replies were being missed on the serial port, but after implementing watch_and_send it began to act up.
• I noticed that the SSL key conversion step worked with your key/certificate pair, but not the one that I generated. I dumped the output of our key files, and they seem identical except that yours starts with "BEGIN RSA PRIVATE KEY" and mine begins with "BEGIN PRIVATE KEY". As far as I can tell, both of our keys are rsa keys. Do you mind sharing what command you used to generate your keys? I used:

  openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem

[tmcadam]

@CoolNamesAllTaken I followed the guide here on test.mosquitto.org to generate the certs and key.

[CoolNamesAllTaken]

Ha, just saw your comment above. Will leave this here in case it's helpful to anyone.

Figured out the key problem. I was generating the wrong format of RSA Key, and then using that key as a certificate instead of using it to create a certificate request. The correct command sequence is below.

openssl genrsa -out azure.key
openssl req -out azure.csr -key azure.key -new

Then, use azure.csr with the mosquitto guide linked in the post above to get an actual client certificate.

Apologies for the spam--I'm picking this up for the first time, and hoping that if I generate some sort of documentation people might have a better time down the line. I'd be happy to do a PR with my code changes and maybe some readme improvements if you'd like @tmcadam .

[tmcadam]

Yes please @CoolNamesAllTaken

[CoolNamesAllTaken]

I'm still working through the code changes, but seem to have connections working to some degree for MQTT with only a remote server certificate (I put mqtt-bothcerts on the backburner for now since Azure doesn't actually have a nice key signing tool like mosquitto as far as I can tell).

@tmcadam dumb question: what is the <cid> that is referred to in all of the SIMCOMM manuals? The manual defines it as:

(PDP Context Identifier) a numeric parameter which
specifies a particular PDP context definition. The parameter
is local to the TE-MT interface and is used in other PDP
context-related commands. The range of permitted values
(minimum value=1) is returned by the test form of the
command.
 1…24 

But I don't really understand what it's saying. My best guess is that it's a connection ID value used for multiplexed connections with multiple IPs, but I'm not sure. So far I've set it to 0 or 1 and things work (as long as all the <cid> values are the same), but I'd like to understand what it's actually for.

[tmcadam]

I'm not really sure to be honest. I guess you could test if you get a second IP using another index while having the first connected. I guess not all carriers would support that as well.

[tmcadam]

I've found the SIMCOMM docs pretty useless and out of step with firmware updates on occasions. It's painful working with AT as well with no meaningful feed back from commands. Just all trial and error.

[CoolNamesAllTaken]

Yeah it's unfortunate how bad the documentation is. The worst part is that there seems to be 3 different ways of doing things for every command, and using any mix of the two methods causes things to break. I'm trying to integrate with the TinyGSM library on Arduino right now and things are blowing up royally.

Would you like me to fork your repo to add changes, or would you like to give me write access? I have a branch that I'm working on at dev/jm/mojo that I can push if you'd like to see any of my changes so far.

[tmcadam]

@CoolNamesAllTaken Added you as collaborator, you should be able to push. Be aware our modules are on fairly different firmware versions (I'm Sim7000e @ B07), but we can add that as flag in the command line and handle the variations in the script if we need to.

[captFuture]

Yeah it's unfortunate how bad the documentation is. The worst part is that there seems to be 3 different ways of doing things for every command, and using any mix of the two methods causes things to break. I'm trying to integrate with the TinyGSM library on Arduino right now and things are blowing up royally.

I am trying to do this as well (based on an esp32 and the sim7000 connected) the initial setup in the current master of TinyGSM is a merge from my fork. -> locally i have the most examples of TinyGSM running (but everything where certificates are used - even the basic https get example are not working at all)

Compared to the SIM800 stuff the AT+CIPSSL command is not present on the SIM7000 which would simply set TCP to use SSL function.

[CoolNamesAllTaken]

@captFuture One of the things that was confusing me about the TinyGSM code was that it was doing a very complicated setup for an "IP Bearer" configuration. It is my impression that a TLS connection is referred to in the AT manual as an "App", which uses an entirely different set of AT commands. For instance, AT+CAOPEN instead of AT+CIPSTART. I'm currently rewriting the Sim7000 TinyGSM code to use SSL only, and consequently use the AT+CA___ type commands. This corresponds to the set of commands that seems to work fine in @tmcadam's python script for mqtt over ssl with a ca cert only.

Is my understanding correct?

[JoshYang1977]

Thank very much for your thread. It answers my questions. Appreciate your experience-sharing.

[gduan2000]

thanks for the post here, it helped a lot. the code works for me except for the SSL part. my goal is to be able to connect to https://httpbin.org with AT+SHCONN. I downloaded the root cert from https://httpbin.org, named it "httpbin-ca.cer". It looks like:

-----BEGIN CERTIFICATE----- MIIDQTCCAimgAwIBAgITBmyfz5m/jAo54vB4ikPmljZbyjANBgkqhkiG9w0BAQsF ADA5MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRkwFwYDVQQDExBBbWF6 b24gUm9vdCBDQSAxMB4XDTE1MDUyNjAwMDAwMFoXDTM4MDExNzAwMDAwMFowOTEL MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEZMBcGA1UEAxMQQW1hem9uIFJv b3QgQ0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALJ4gHHKeNXj ca9HgFB0fW7Y14h29Jlo91ghYPl0hAEvrAIthtOgQ3pOsqTQNroBvo3bSMgHFzZM 9O6II8c+6zf1tRn4SWiw3te5djgdYZ6k/oI2peVKVuRF4fn9tBb6dNqcmzU5L/qw IFAGbHrQgLKm+a/sRxmPUDgH3KKHOVj4utWp+UhnMJbulHheb4mjUcAwhmahRWa6 VOujw5H5SNz/0egwLX0tdHA114gk957EWW67c4cX8jJGKLhD+rcdqsq08p8kDi1L 93FcXmn/6pUCyziKrlA4b9v7LWIbxcceVOF34GfID5yHI9Y/QCB/IIDEgEw+OyQm jgSubJrIqg0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMC AYYwHQYDVR0OBBYEFIQYzIU07LwMlJQuCFmcx7IQTgoIMA0GCSqGSIb3DQEBCwUA A4IBAQCY8jdaQZChGsV2USggNiMOruYou6r4lK5IpDB/G/wkjUu0yKGX9rbxenDI U5PMCCjjmCXPI6T53iHTfIUJrU6adTrCC2qJeHZERxhlbI1Bjjt/msv0tadQ1wUs N+gDS63pYaACbvXy8MWy7Vu33PqUXHeeE6V/Uq2V8viTO96LXFvKWlJbYK8U90vv o/ufQJVtMVT8QtPHRh8jrdkPSHCa2XV4cdFyQzR1bldZwgJcJmApzyMZFo6IQ6XU 5MsI+yMRQ+hDKXJioaldXgjUkK642M4UwtBV8ob2xJNDd2ZhwLnoQdeXeGADbkpy rqXRfboQnoZsG4q5WTP468SQvvG5 -----END CERTIFICATE-----

Then I used python sim7000.py cert-load to upload the cert to the SIM7000A module, no issue. I am able to check it with QPST EFS explorer.

Finally I ran the mqtt-cacert function (updated some commands) and getting the following response, I tried running it through putty, SSCOMV5.13.1, I tried different cert, different sites. It gets ERROR on AT+SHCONN no matter what. am I missing some critical step? No issue with HTTP calls.

I appreciate anybody's help here, if you have a working root cert and matching site that you know is working, please share it. at this point, I don't know if this is a defective module. The firmware shows Revision:1351B04SIM7000A. thanks a lot!

... ... ... ++++++++++++++++++++ MQTT - CA Cert Only +++++++++++++++++++++

----------- AT+CNACT=1 ----------- Error (0.04secs): AT+CNACT=1 +CME ERROR: operation not allowed

----------- AT+CNACT? ----------- Success (0.08secs): AT+CNACT? +CNACT: 1,"10.99.3.170" OK

----------- AT+SHCONF="URL","https://httpbin.org" ----------- Success (0.05secs): AT+SHCONF="URL","https://httpbin.org" OK

----------- AT+CSSLCFG="sslversion",1,3 ----------- Success (0.04secs): AT+CSSLCFG="sslversion",1,3 OK

----------- AT+CSSLCFG="convert",2,"httpbin-ca.cer" ----------- Success (0.03secs): AT+CSSLCFG="convert",2,"httpbin-ca.cer" OK

----------- AT+SHSSL=1, "httpbin-ca.cer" ----------- Success (0.04secs): AT+SHSSL=1, "httpbin-ca.cer" OK

----------- AT+SHSSL? ----------- Success (0.08secs): AT+SHSSL? +SHSSL: 1,"httpbin-ca.cer","" OK

----------- AT+SHSTATE? ----------- Success (0.09secs): AT+SHSTATE? +SHSTATE: 0 OK

----------- AT+SHCONN ----------- Error (0.04secs): AT+SHCONN +CME ERROR: operation not allowed

...

[alibahba]

You can try: AT+SHSSL=1,"" This will make a connection on HTTPS but don't validate the certificate. I have the same problem and an outstanding question about this by SIMCOM. My modem is a SIM7000G on firmware 1529B05SIM7000G.

Some of my examples:

AT+SHSSL=1,"" OK AT+SHCONF="URL","https://google.nl" OK AT+SHCONF="BODYLEN",1024 OK AT+SHCONF="HEADERLEN",350 OK AT+SHCONN OK

AT+SHSSL=1,"" OK AT+SHCONF="URL","https://nu.nl" OK AT+SHCONF="BODYLEN",1024 OK AT+SHCONF="HEADERLEN",350 OK AT+SHCONN ERROR

AT+SHSSL=1,"" OK AT+SHCONF="URL","https://myqtthub.com" OK AT+SHCONF="BODYLEN",1024 OK AT+SHCONF="HEADERLEN",350 OK AT+SHCONN OK

AT+CSSLCFG=convert,2,ca.crt OK AT+SHSSL=1,"ca.crt" OK AT+SHCONF="URL","https://myqtthub.com" OK AT+SHCONF="BODYLEN",1024 OK AT+SHCONF="HEADERLEN",350 OK AT+SHCONN ERROR

[cshunt]

I had a similar issue when running code at 115200 and even some at 9600 baud. The beginning of strings were missing. I believe same as @CoolNamesAllTaken second post on this, when "with" is used, it re-initiates the serial port and takes a moment and potentially drops some characters from the incoming stream. One potential fix is below, the send and receive commands are initiated with the same "with" serial command. Another option would be to move the serial setup as part of a first run command.

Also note that I was able to remove all of the sleep commands and received data came through as expected!!

Thank you for sharing your code, I have adopted some to SIM7080, the AT command and return does work well.

#def send(data):
#    with serial.Serial(PORT, BAUD, timeout=1) as ser:
#        ser.write(data)

#def send_cmd(cmd):
#    send(cmd.encode('utf-8') + CMD_LINEBREAK)

#def send_cmd(cmd):
#    with serial.Serial(PORT, BAUD, timeout=1) as ser:
#        ser.write((cmd.encode('utf-8') + CMD_LINEBREAK))

def send_cmd(cmd, timeout=10, success=None, failure=None, echo_cmd=None):
#    with serial.Serial(PORT, BAUD, timeout=1) as ser:
#       ser.write((cmd.encode('utf-8') + CMD_LINEBREAK))

#def watch(timeout=10, success=None, failure=None, echo_cmd=None):
    with serial.Serial(PORT, BAUD, timeout=1) as ser:
        t_start = time.time()
        ser.write((cmd.encode('utf-8') + CMD_LINEBREAK))
        reply = list()
        while True:
            #time.sleep(.2)
            if ser.in_waiting:
                line = ser.readline()
                echo = False
                if echo_cmd:
                    echo = line.decode('utf-8').strip().endswith(echo_cmd)
                if line != CMD_LINEBREAK and not echo:
                    if line != "\r\n":
                        line = line.decode('utf-8').strip()
                        reply.append('\t' + line)
                    if success and line.startswith(success):
                        return ("Success", reply, time.time()-t_start)
                    if failure and line.startswith(failure):
                        return ("Error", reply, time.time()-t_start)
            if (time.time()-t_start) > timeout:
                return ("Timeout", reply, time.time()-t_start)
            #time.sleep(0.1)

def AT(cmd="", timeout=10, success="OK", failure="+CME ERROR"):
    cmd = 'AT' + cmd
    print("----------- ", cmd, " -----------")
    #send_cmd(cmd)
    #reply = watch(echo_cmd=cmd, timeout=timeout, success=success, failure=failure)
    reply = send_cmd(cmd, echo_cmd=cmd, timeout=timeout, success=success, failure=failure)
    print("{0} ({1:.2f}secs):".format(reply[0], reply[2]))
    print(*reply[1], sep='\n')
    print('')
    return reply