search

botletics / SIM7000-LTE-Shield

Botletics SIM7000 LTE CAT-M1/NB-IoT Shield for Arduino
https://www.botletics.com/products/sim7000-shield
GNU General Public License v3.0
477 stars 215 forks source link

Do you have Amazon AWS Tutorial..? #58

Open kittinook opened 5 years ago

kittinook commented 5 years ago

Hi, I've interested this board. Do you have the example for communication between the board and AWS IoT cloud?

botletics commented 5 years ago

Not currently, no. However, there are example AT command logs I have from SIMCom for Microsoft Azure which theoretically should work for other platforms like AWS IoT and anything that requires certificates. Basically you store the certificate in the SIM7000's EFS (electronic file system) and use the SSL commands to connect.

kittinook commented 5 years ago

Thanks for your fast response, you can share example AT command logs for Microsoft Azure. I can't find the example AT command in this web http://www.simcomm2m.com/En/module/detail.aspx?id=175.

botletics commented 5 years ago

I can't share it publicly so you would have to order a Botletics shield before I could share it.

bradleytompkins commented 5 years ago

Can you share these with me? We have purchased several botletics shields to experiment with, and are trying to get them talking to Azure IoT Hub.

botletics commented 5 years ago

Please email me, botletics "at" gmail "dot" com.

slipiduche commented 5 years ago

Not currently, no. However, there are example AT command logs I have from SIMCom for Microsoft Azure which theoretically should work for other platforms like AWS IoT and anything that requires certificates. Basically you store the certificate in the SIM7000's EFS (electronic file system) and use the SSL commands to connect.

Hi, there are some example where shows how store and read data from SIM7000's EFS?

botletics commented 5 years ago

This should help but also check the related AT command manual.

slipiduche commented 5 years ago

This should help but also check the related AT command manual.

oh thanks you. i'm realy confused, i don't know from where extract the .cer, i mean a SD? a web server? or the download from pc?. where i could put the .cer to apply these commands.

thanks so much.

botletics commented 5 years ago

That depends on what platform you're using (Azure, AWS, etc.) and that file would be on your computer and sent to the SIM7000 via USB with AT commands.

slipiduche commented 5 years ago

i have a doubt, the certificate must be in what format? i try this -----BEGIN CERTIFICATE----- MIIDQTCCAimgAwIBAgITBmyfz5m/jAo54vB4ikPmljZbyjANBgkqhkiG9w0BAQsF ADA5MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRkwFwYDVQQDExBBbWF6 b24gUm9vdCBDQSAxMB4XDTE1MDUyNjAwMDAwMFoXDTM4MDExNzAwMDAwMFowOTEL MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEZMBcGA1UEAxMQQW1hem9uIFJv b3QgQ0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALJ4gHHKeNXj ca9HgFB0fW7Y14h29Jlo91ghYPl0hAEvrAIthtOgQ3pOsqTQNroBvo3bSMgHFzZM 9O6II8c+6zf1tRn4SWiw3te5djgdYZ6k/oI2peVKVuRF4fn9tBb6dNqcmzU5L/qw IFAGbHrQgLKm+a/sRxmPUDgH3KKHOVj4utWp+UhnMJbulHheb4mjUcAwhmahRWa6 VOujw5H5SNz/0egwLX0tdHA114gk957EWW67c4cX8jJGKLhD+rcdqsq08p8kDi1L 93FcXmn/6pUCyziKrlA4b9v7LWIbxcceVOF34GfID5yHI9Y/QCB/IIDEgEw+OyQm jgSubJrIqg0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMC AYYwHQYDVR0OBBYEFIQYzIU07LwMlJQuCFmcx7IQTgoIMA0GCSqGSIb3DQEBCwUA A4IBAQCY8jdaQZChGsV2USggNiMOruYou6r4lK5IpDB/G/wkjUu0yKGX9rbxenDI U5PMCCjjmCXPI6T53iHTfIUJrU6adTrCC2qJeHZERxhlbI1Bjjt/msv0tadQ1wUs N+gDS63pYaACbvXy8MWy7Vu33PqUXHeeE6V/Uq2V8viTO96LXFvKWlJbYK8U90vv o/ufQJVtMVT8QtPHRh8jrdkPSHCa2XV4cdFyQzR1bldZwgJcJmApzyMZFo6IQ6XU 5MsI+yMRQ+hDKXJioaldXgjUkK642M4UwtBV8ob2xJNDd2ZhwLnoQdeXeGADbkpy rqXRfboQnoZsG4q5WTP468SQvvG5 -----END CERTIFICATE-----

and not connect

at log is that <--- OK --->AT+CFSgfis=3,"root_ca.pem" <--- +CFSGFIS: 1189

OK +CNACT: 1,"100.100.197.199" OK --->AT+SMCONF="URL",a5xpqsmvbu9sq-ats.iot.us-west-2.amazonaws.com,8883 <--- OK --->AT+SMCONF="CLIENTID",device2 <--- OK --->AT+SMCONF="KEEPTIME",60 <--- OK --->AT+SMCONF="CLEANSS",0 <--- OK --->AT+SMCONF="QOS",0 <--- OK --->AT+CSSLCFG? <--- OK --->AT+CSSLCFG="sslversion",0,3 <--- OK --->AT+CSSLCFG=0,1,0 <--- ERROR --->AT+CSSLCFG=convert,2,root_ca.pem <--- OK --->AT+CSSLCFG=convert,1,my_client.pem,my_key.pem <--- OK --->AT+CSSLCFG? <--- OK --->AT+CIPSTATUS <--- OK STATE: IP GPRSACT --->AT+CIFSR <--- 100.100.197.199 --->AT+CIPSTATUS <--- OK STATE: IP STATUS --->AT+SMSSL=1,root_ca.pem,my_client.pem <--- OK --->AT+SMSSL? <--- +SMSSL: 1,"root_ca.pem","my_client.pem" OK --->AT+CSSLCFG? <--- OK --->AT+CGATT? <--- +CGATT: 1 OK --->AT+SMCONN <--- ERROR When i try not secure connection with cloudmqtt these commands works but not with AWS
brunokruse commented 5 years ago

Not currently, no. However, there are example AT command logs I have from SIMCom for Microsoft Azure which theoretically should work for other platforms like AWS IoT and anything that requires certificates. Basically you store the certificate in the SIM7000's EFS (electronic file system) and use the SSL commands to connect.

This issue should be open. I saw the azure example in your AT Command Logs; thank you for that. However, it seems AWS only supports Https. The firmware on some of the shields support SSL only via TCP. Is there info on specific firmware releases and features to confirm? I am using B017000G.

botletics commented 5 years ago

Sorry, I'm not sure if there's anything on specific firmware versions.

jefflikesbagels commented 5 years ago

I apologize in advance for my ignorance, as this is my first time programming a SIM7000. I am trying to perform the same task as above but using hologram.io. I created a new function in the Adafruit_FONA.cpp library and called it postDataHTTPS:

boolean Adafruit_FONA::postDataHTTPS(const char *request_type, const char *URL, const char *body, const char *token, uint32_t bodylen) {
  // NOTE: Need to open socket/enable GPRS before using this function
  // char auxStr[64];

    sendCheckReply(F("AT+GMR"), ok_reply, 10000);
    sendCheckReply(F("AT+CNACT=1,\"hologram\""), ok_reply, 10000);
    sendCheckReply(F("AT+CNACT?"), ok_reply, 10000);
    sendCheckReply(F("AT+CSSLCFG=\"convert\",2,\"hologram.cer\""), ok_reply, 10000);
    sendCheckReply(F("AT+SHSSL=1,\"hologram.cer\""), ok_reply, 10000);

    char urlBuff[strlen(URL) + 22];
    sprintf(urlBuff, "AT+SHCONF=\"URL\",\"%s\"", URL);
    if (! sendCheckReply(urlBuff, ok_reply, 10000))
        return false;

    sendCheckReply(F("AT+SHCONF=\"BODYLEN\",100"), ok_reply, 10000);
    sendCheckReply(F("AT+SHCONF=\"HEADERLEN\",100"), ok_reply, 10000);
    sendCheckReply(F("AT+SHCONN"), ok_reply, 10000);

    char dataBuff[strlen(body) + 22];
    sprintf(dataBuff, "AT+SHBOD=\"%s\",100", body);

    //if (! sendCheckReply(dataBuff, ok_reply, 10000))
    //  return false;
    sendCheckReply(dataBuff, ok_reply, 10000);
    //sendCheckReply(F("AT+SHBOD=\"TEST\",100"), ok_reply, 10000);

    sendCheckReply(F("AT+SHAHEAD=\"Content-Length\",\"120\""), ok_reply, 10000);
    sendCheckReply(F("AT+SHSTATE?"), ok_reply, 10000);
    sendCheckReply(F("AT+SHREQ=3"), ok_reply, 10000);
    sendCheckReply(F("AT+SHREAD=0,227"), ok_reply, 10000);
    sendCheckReply(F("AT+SHDISC"), ok_reply, 10000);

  return true;
}

I've been reading the SIM7000 documentation for the HTTPS commands, and I am struggling to get it working as I am sure I have mistakes somewhere. In my Arduino sketch I have the following:

        // Post data to website via 2G or LTE CAT-M/NB-IoT
        // Create char buffers for the floating point numbers for sprintf
        // Make sure these buffers are long enough for your request URL
        char URL[150];
        char body[100];
        char deviceID[] = "######";
        char tagID[] = "[\"_RESTAPI_\", \"WATER_LOW\"]";
        char message[] = "\"Water_Low\"";

        // POST request
        sprintf(URL, "https://dashboard.hologram.io/api/1/csr/rdm");
        sprintf(body, "{\"deviceid\": %s, \"tags\": %s, \"data\": %s}", deviceID, tagID, message);

        Serial.println(F("Attempting to perform HTTPS POST..."));
        Serial.print("URL: ");
        Serial.print(URL);
        Serial.println();
        Serial.print("Body: ");
        Serial.print(body);
        Serial.println();
        if (!fona.postDataHTTPS("POST", URL, body)){
          Serial.println(F("Failed to complete HTTPS POST!"));
        } else {
          Serial.println(F("Successfully performed HTTPS POST!"));
        }

When the sketch runs, I get the following on the serial monitor:

Attempting to perform HTTPS POST...
URL: https://dashboard.hologram.io/api/1/csr/rdm
Body: {"deviceid": ######, "tags": ["_RESTAPI_", "WATER_LOW"], "data": "Water_Low"}
    ---> AT+GMR
    <--- Revision:1351B03SIM7000A
    ---> AT+CNACT=1,"hologram"
    <--- ERROR
    ---> AT+CNACT?
    <--- +CNACT: 1,"###.###.###.###"
    ---> AT+CSSLCFG="convert",2,"hologram.cer"
    <--- ERROR
    ---> AT+SHSSL=1,"hologram.cer"
    <--- OK
    ---> AT+SHCONF="URL","https://dashboard.hologram.io/api/1/csr/rdm"
    <--- OK
    ---> AT+SHCONF="BODYLEN",100
    <--- OK
    ---> AT+SHCONF="HEADERLEN",100
    <--- OK
    ---> AT+SHCONN
    <--- ERROR
    ---> AT+SHBOD="{"deviceid": ######, "tags": ["_RESTAPI_", "WATER_LOW"], "data": "Water_Low"}",100
    <--- ERROR
    ---> AT+SHAHEAD="Content-Length","120"
    <--- ERROR
    ---> AT+SHSTATE?
    <--- +SHSTATE: 0
    ---> AT+SHREQ=3
    <--- ERROR
    ---> AT+SHREAD=0,227
    <--- ERROR
    ---> AT+SHDISC
    <--- ERROR

I used the LTE_Demo example sketch as a building block, so I have all of the other associated code in place and working well to set up the SIM7000. It is also getting a proper IP address when I issue the AT+CNACT? command.

The first problem is the error on the AT+CSSLCFG command, I think that is preventing the AT+SHCONN and AT+SHBOD commands from working. I am also not sure how I should be handling the quotation marks and commas inside the body for the AT+SHBOD command, do I simply prefix them with a back slash?. Any ideas on what I could be doing wrong? I downloaded the top-level Starfield Class 2 Certification Authority key, which is below:

-----BEGIN CERTIFICATE-----
MIIEDzCCAvegAwIBAgIBADANBgkqhkiG9w0BAQUFADBoMQswCQYDVQQGEwJVUzEl
MCMGA1UEChMcU3RhcmZpZWxkIFRlY2hub2xvZ2llcywgSW5jLjEyMDAGA1UECxMp
U3RhcmZpZWxkIENsYXNzIDIgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMDQw
NjI5MTczOTE2WhcNMzQwNjI5MTczOTE2WjBoMQswCQYDVQQGEwJVUzElMCMGA1UE
ChMcU3RhcmZpZWxkIFRlY2hub2xvZ2llcywgSW5jLjEyMDAGA1UECxMpU3RhcmZp
ZWxkIENsYXNzIDIgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwggEgMA0GCSqGSIb3
DQEBAQUAA4IBDQAwggEIAoIBAQC3Msj+6XGmBIWtDBFk385N78gDGIc/oav7PKaf
8MOh2tTYbitTkPskpD6E8J7oX+zlJ0T1KKY/e97gKvDIr1MvnsoFAZMej2YcOadN
+lq2cwQlZut3f+dZxkqZJRRU6ybH838Z1TBwj6+wRir/resp7defqgSHo9T5iaU0
X9tDkYI22WY8sbi5gv2cOj4QyDvvBmVmepsZGD3/cVE8MC5fvj13c7JdBmzDI1aa
K4UmkhynArPkPw2vCHmCuDY96pzTNbO8acr1zJ3o/WSNF4Azbl5KXZnJHoe0nRrA
1W4TNSNe35tfPe/W93bC6j67eA0cQmdrBNj41tpvi/JEoAGrAgEDo4HFMIHCMB0G
A1UdDgQWBBS/X7fRzt0fhvRbVazc1xDCDqmI5zCBkgYDVR0jBIGKMIGHgBS/X7fR
zt0fhvRbVazc1xDCDqmI56FspGowaDELMAkGA1UEBhMCVVMxJTAjBgNVBAoTHFN0
YXJmaWVsZCBUZWNobm9sb2dpZXMsIEluYy4xMjAwBgNVBAsTKVN0YXJmaWVsZCBD
bGFzcyAyIENlcnRpZmljYXRpb24gQXV0aG9yaXR5ggEAMAwGA1UdEwQFMAMBAf8w
DQYJKoZIhvcNAQEFBQADggEBAAWdP4id0ckaVaGsafPzWdqbAYcaT1epoXkJKtv3
L7IezMdeatiDh6GX70k1PncGQVhiv45YuApnP+yz3SFmH8lU+nLMPUxA2IGvd56D
eruix/U0F47ZEUD0/CwqTRV/p2JdLiXTAAsgGh1o+Re49L2L7ShZ3U0WixeDyLJl
xy16paq8U4Zt3VekyvggQQto8PT7dL5WXXp59fkdheMtlb71cZBDzI0fmgAKhynp
VSJYACPq4xJDKVtHCN2MQWplBqjlIapBtJUhlbl90TSrE9atvNziPTnNvT51cKEY
WQPJIrSPnNVeKtelttQKbfi3QBFGmh95DmK/D5fs4C8fF5Q=
-----END CERTIFICATE-----

However I am not sure how to include that on the AT+CSSHCFG command.

Edit: I was able to successfully install the QPST software and upload the CA key to the 'customer' directory in the alternate file system of the SIM7000. I now get an "OK" response on the AT+CSSHCFG command. However, I am still getting "ERROR" on the AT+SHCONN command. I wonder if it is related to the AT+SNACT command erroring out too? The following tool was helpful in getting some additional insight on how the certificates work: https://github.com/tmcadam/sim7000-tools As well as this previous issue: #71

Thanks!

TimRoadley commented 5 years ago

I'm having the same issues as @jefflikesbagels

In addition, I'm concerned that if I release these IoT devices to our customers then we will have to recall them all to update the root CA when those certificates expire.

I wonder if there's a way to force the device to trust the root CA regardless of who it is and then just continue with the HTTPS POST.

I think this would be ok, since the configuration on what we're sending to will be baked in to the firmware of our devices.

Thoughts?

TimRoadley commented 5 years ago

In case it helps anyone, I managed to upload a cert with the following procedure (not using the EFS Explorer)

Copy a CA Root Cert (for the site you're trying to connect to) to the SIM7000:

1) Go to ssltools.com 2) Enter the website you want the root cert for (e.g. https://putsreq.com) 3) Download the root certificate PEM, which should give you a .cer file 4) Strip out all carriage returns and then count the characters in the cert file (look at the bottom of VSCode for a character count)

Use these commands to load the cert onto the SIM7000:

AT+CFSINIT AT+CFSWFILE=3,"AddTrust.crt",0,1496,10000 <-- 1496 is the char length that noted before paste the cert data into the terminal within 10 seconds AT+CFSTERM

This puts the cert in the 'customer' directory on the SIM7000. Tomorrow I'll see if this lets me make that https post, try your luck with this approach in the mean time.

jefflikesbagels commented 5 years ago

@TimRoadley Thanks for the info. I've been thinking about how I can integrate all of this into the Arduino so I can remotely push a new certificate to the EEPROM using some creativity with hologram.io's tools, then use the AT+CFSWFILE command to write the certificate, but unfortunately I'm using an UNO which does not have enough EEPROM space to store the entire certificate. For my use case, I'm just doing a simple DIY project for a friend, so if it lasts until 2034 when the CA expires that's good enough for me haha. I guess I could expand the EEPROM with an additional chip, but at that point for all the extra work involved I might as well just switch to SMS alerts and pay the $0.20 per message instead.

What's strange is the ssltools site is giving me a 502 bad gateway when I tried to download the root certificate, but all of the others work. I originally just used Chrome to export it anyways. I think I am past the certificate part, so now I have to figure out why the AT+SHCONN command is failing.

jefflikesbagels commented 5 years ago

Well this is frustrating. I did some more digging, and found another issue that is preventing me from making progress. According to the SIMCOM technical documentation, the max string size for the URL on the AT+SHCONF and AT+SHREQ commands is 64 bytes. For sending a data route through Hologram, the Arduino needs to do an HTTPS POST to the following URL: https://dashboard.hologram.io/api/1/csr/rdm?apikey=############################## I was not adding the API key before (duh - hologram was rejecting the API call), but now that I am adding it, the length of the URL is 81 bytes. I tried setting up an HTTP redirect on my personal web server to shorten the length, but the redirect prevents the hologram REST API from parsing the data properly.

One thing that helped me immensely was using the Restlet Client Chrome Extension. Between that and sifting through the Hologram REST API documentation again helped me figure out what format it's actually expecting.

Getting back to the issue at hand, it could be possible that the AT+SHCONN command fails because the Hologram API is rejecting the connection due to the incorrect URL (without the API key). Surely I am missing something here? The 64 byte URL limit is going to completely break the SIM7000's ability to do HTTPS POST commands to activate Hologram data routes. The next option may be using a TCP socket connection to Hologram Cloud: Socket API, Device Key. It looks like that would be the better solution anyways.

Sorry to derail a bit from the original intention of working with AWS, but I believe the procedure will be very similar to Hologram, so this development will still be beneficial. If I should create a separate issue tracker just let me know. It looks like AWS supports both HTTPS and MQTT calls, while Hologram supports HTTPS and TCP socket calls.

TimRoadley commented 5 years ago

@jefflikesbagels out of interest what firmware version are you running (and what chip)?

My testing has paused since I blew up my SIM7000E with a firmware update. Be careful with firmware over the air (FOTA)!

jefflikesbagels commented 5 years ago

@TimRoadley I have a SIM7000A running 1351B03SIM7000A firmware.

TimRoadley commented 5 years ago

@jefflikesbagels I wonder if https://github.com/botletics/SIM7000-LTE-Shield/blob/master/SIM7000%20Documentation/Firmware/1351B04SIM7000A.rar would help (I have no idea)

jefflikesbagels commented 5 years ago

@TimRoadley Thanks, I went ahead and updated to B04 just for good measure.

I finally got the Arduino sending data to Hologram via the Socket API!!! The issue I found is very silly too. For the FONA library commands, a lot of them are used in the following (or a similar) fashion:

        // Connect to TCP server
        if (!fona.TCPconnect(host, port)) {
          Serial.println(F("Failed to connect to server!"));
          delay(5000);
          break;
        } else {
          Serial.println(F("Successfully connected to server!"));
        }
        delay(5000);
        // Send TCP payload
        if (!fona.TCPsend(TCPpayload,sizeof(TCPpayload))) {
          Serial.println(F("Failed to send TCP payload!"));
          delay(5000);
          break;
        }

Where there is an if statement checking whether the function returned false or true. With this code it was not working properly at all. However, on a whim I decided to try and simplify the code as much as possible, and removed all of these checks down to the following:

        fona.TCPconnect(host, port);
        fona.TCPsend(TCPpayload,sizeof(TCPpayload));
        fona.TCPclose();

And all of a sudden it started working! One thing I noticed before was that I would get the "failed to connect" message on the serial monitor, but would continue receiving responses from the SIM7000, almost like the code is getting ahead of itself. I know I've deviated really far from the original goal of using HTTPS POST, but give this a shot and see if it helps. It's possible that removing all of the extra if statements and logic will allow the SIM7000 to send data properly. Here's the final snippet of code for my TCP socket connection:

        // Send TCP payload to server via LTE CAT-M/NB-IoT
        char host[] = "cloudsocket.hologram.io";
        uint32_t port = 9999;
        char devicekey[] = "xxxxxxxx";
        char data[] = "Water_Low";
        char topics[] = "WATER_LOW";
        char TCPpayload[strlen(devicekey)+strlen(data)+strlen(topics)+24];
        sprintf(TCPpayload, "{\"k\":\"%s\",\"d\":\"%s\",\"t\":\"%s\"}", devicekey, data, topics);
        Serial.println(TCPpayload);

        // Connect to GPRS
        fona.enableGPRS(true);

        // Connect to TCP server
        fona.TCPconnect(host, port);

        // Send TCP payload
        fona.TCPsend(TCPpayload,sizeof(TCPpayload));

        // Disconnect from TCP server
        fona.TCPclose();

        // Disconnect from GPRS
        fona.enableGPRS(false);
botletics commented 4 years ago

Hey guys, there is now a Botletics community forum that makes it easier to post questions and things. Feel free to join!

sethivansh6 commented 3 years ago
  1. ssltools.com

In case it helps anyone, I managed to upload a cert with the following procedure (not using the EFS Explorer)

Copy a CA Root Cert (for the site you're trying to connect to) to the SIM7000:

  1. Go to ssltools.com
  2. Enter the website you want the root cert for (e.g. https://putsreq.com)
  3. Download the root certificate PEM, which should give you a .cer file
  4. Strip out all carriage returns and then count the characters in the cert file (look at the bottom of VSCode for a character count)

Use these commands to load the cert onto the SIM7000:

AT+CFSINIT AT+CFSWFILE=3,"AddTrust.crt",0,1496,10000 <-- 1496 is the char length that noted before paste the cert data into the terminal within 10 seconds AT+CFSTERM

This puts the cert in the 'customer' directory on the SIM7000. Tomorrow I'll see if this lets me make that https post, try your luck with this approach in the meantime.

@TimRoadley Hey I used all the things you stated from downloading and remove carriage return to sending. But I GOT an error while writing the command

AT+CFSWFILE=3,"dweet.crt",0,1901,10000 DOWNLOAD

ERROR

Do you know any reason why it happened? do I have to place that file in a certain folder? or do I have to remove Begin certificate and end certificate line?

blazczak commented 3 years ago

@sethivansh6 ERROR during AT+CFSWFILE points at the module not receiving the (correct) file contents within the self-imposed time (you specified 10000 = 10 seconds). Perhaps there is a mismatch on the number of bytes (you specified 1901). No other content validation is performed in this step, it's just a straight EFS put.

Btw, when working in *nix command line, one can just wc -c (or even ls -l) the local file to get the exact number of bytes when preparing the transfer.

botletics commented 3 years ago

Hey guys, please see this AWS AT command log from SIMCom. Hope it helps!

tomlankhorst commented 3 years ago

@botletics Thanks, that helps a bit. I'm looking to configure the module to just validate the server cert. The client will be authenticated via username/password so there is no client cert.

Did anyone achieve this?

botletics commented 3 years ago

Please see the addRootCA() and TCPconnect() functions here. Please also see this set of AT commands that another user tried that worked for him.

tomlankhorst commented 3 years ago

Thank you. In addition, I got a response from SIMCOM:

The chipset for SIM7000E need CA\client crt\client key , 3 files, can not support only CA. But you can input dummy client crt and client key file, just to "cheat" the stack that it already has 3 file totally. BTW before connection please update date and time by NTP function, AT+CCLK? should return correct data and time. [...]

davegravy commented 3 years ago

@botletics I tried following the AWS IoT command log you posted however I am getting ERROR at AT+SMCONN.

How can I debug this? The same certificates and key allowed me to use their Python SDK to publish successfully.

AT+CCLK?
+CCLK: "21/05/31,04:30:24+00"

OK
AT+CPIN?
+CPIN: READY

OK
AT+CSQ
+CSQ: 29,99

OK
AT+CGREG?
+CGREG: 0,1

OK
AT+COPS?
+COPS: 0,0,"ROGERS ROGERS",7

OK
AT+CGNAPN
+CGNAPN: 1,"ciot"

OK
AT+CNACT=1,"ciot"
OK

+APP PDP: ACTIVE
AT+CNACT?
+CNACT: 1,"10.237.129.141"

OK
at+csslcfg=convert,2,AmazonRootCA1.pem
OK
at+csslcfg=convert,1,my_thing.cert.pem,my_thing.private.key
OK
AT+CSSLCFG="sslversion",0,3
OK
AT+SMSSL=1,AmazonRootCA1.pem,my_thing.cert.pem
OK
AT+SMCONF=url,##############-ats.iot.us-east-1.amazonaws.com,8883
OK
AT+SMCONF="clientid","basicPubSub"
OK
AT+SMCONF="KEEPTIME",60
OK
AT+SMCONF?
+SMCONF
CLIENTID: "basicPubSub"
URL: "##############-ats.iot.us-east-1.amazonaws.com:8883"
KEEPTIME: 60
USERNAME: ""
PASSWORD: ""
CLEANSS: 0
QOS: 0
TOPIC: ""
MESSAGE: ""
RETAIN: 0

OK
AT+SMCONN
ERROR

I also tried setting a topic and message (same topic that the python client successfully publishes to) with no luck.

davegravy commented 3 years ago

I finally figured this out!

I now have AT+SMCONN succeeding with AWS (haven't tested anything beyond this yet)

There's 2 important non-obvious steps.

  1. Whatever url AWS gives you for your IoT Core endpoint, you have to strip out the "-ats" from it. So "a1k9ecig9j720o-ats.iot.us-east-1.amazonaws.com" becomes "a1k9ecig9j720o.iot.us-east-1.amazonaws.com"
  2. You need to use the legacy root certificate provided by AWS here under "VeriSign Endpoints (legacy)". NOTE some regions apparently don't support legacy certs, so it seems the SIM7000 won't work in those.

I also updated my SIM7000A to the B04 firmware from B03, not sure if that mattered.

Contrary to what @tomlankhorst posted, it doesn't seem to matter if your RTC is sync'd.

davegravy commented 3 years ago

Not quite there yet... subscriptions work but publishing doesn't. 

2019-06-26 14:08:06:424[Send->]AT+SMPUB="ryan","8",1,1

2019-06-26 14:08:06:428[Recv<-]AT+SMPUB="ryan","8",1,1

> test626
2019-06-26 14:08:21:365[Send->]test626
2019-06-26 14:08:21:398[Recv<-]test626
2019-06-26 14:08:27:766[Recv<-]
+SMSTATE: 0

OK

The connection drops immediately after publish, and in the AWS (Cloudwatch) logs is this:

{
    "timestamp": "2021-06-01 04:56:38.276",
    "logLevel": "ERROR",
    "traceId": "6c4ca615-12d0-93d3-6434-f5f85365cc66",
    "accountId": "1234567890123",
    "status": "Failure",
    "eventType": "Publish-In",
    "protocol": "MQTT",
    "topicName": "ryan",
    "clientId": "basicPubSub",
    "principalId": "82b899e4bcb6bfc158f83be904d9a305e2b21ee255f0d7062cbe6ad3eda05f7a",
    "sourceIp": "xxx.xxx.xxx.xxx",
    "sourcePort": 29573
}
davegravy commented 3 years ago

Publishing works with this:

AT+SMPUB="test-topic",8,0,0
> testing1
OK
AT+SMPUB="test-topic",8,1,0
> testing2
OK

fails with:

AT+SMPUB="test-topic",8,2,0
> testing3
+SMSTATE: 0

OK

fails with:

AT+SMPUB="test-topic",8,0,1
> testing3
+SMSTATE: 0

OK

In summary, server retain can't be used and qos=2 can't be used. This is an AWS limitation.

davegravy commented 3 years ago

Can't seem to get persistent connections working with AWS

  1. After AT+SMCONF="CLEANSS",0 I connect, then subscribe. Confirm I receive messages on the subscribed topic.

  2. Then I kill the power on the shield, publish a new (qos=1) message to the subscribed topic from another client, power the shield back on, reconnect (SMCONN). Queued message not received. Even new messages on the topic are not received.

  3. Resubscribe to topic (SMSUB), queued messages still not received, but new messages are received.

After power up and reconnect in step 2 I see this message in AWS Cloudwatch:

{
    "timestamp": "2021-06-02 13:54:53.996",
    "logLevel": "ERROR",
    "traceId": "8133c8d9-621f-d392-0b22-5f55b4c22f3e",
    "accountId": "144349053222",
    "status": "Failure",
    "eventType": "Disconnect",
    "protocol": "MQTT",
    "clientId": "ARMS-GF-D01",
    "principalId": "82b899e4bcb6bfc158f83ac904d9a305e2b21ee255f0d7062cbe6ad3eda05f7a",
    "sourceIp": "74.198.90.117",
    "sourcePort": 16187,
    "reason": "DUPLICATE_CLIENT_ID",
    "details": "A new connection was established with the same client ID",
    "disconnectReason": "DUPLICATE_CLIENTID"
}

I think this suggests it is not rejoining the existing persistent session but starting a new one? Anyone know what I'm doing wrong?

Also the last will and testament message isn't getting published to the configured topic at any point after disconnect.

davegravy commented 3 years ago

While I've been successful getting connections to complete over MQTTS, HTTPS is still not working with ERROR response from SHCONN.

If I use port 443 for the URL, SHCONN succeeds, but then I get 403 errors to all my requests because they lack SigV4 signing. If I use port 8443, then SHCONN times out. Note that I'm using the same CA , client cert and key for MQTTS, HTTPS (port 443) and HTTPS (port 8443). I can also curl the endpoint with no problem using these certs, so there does appear to be a bug somewhere with HTTPS.

I've had no response from SIMCOM technical support. in over a week.

JamesHillman87 commented 2 years ago

@davegravy Thanks for all your updates on getting MQTT working with AWS IoT. Can you please share your final commad log? We currently have an error when we connect. Thanks in advance.

psshiner commented 2 years ago

@davegravy Thanks for all your updates on getting MQTT working with AWS IoT. Can you please share your final commad log? We currently have an error when we connect. Thanks in advance.

Hi James, I have been playing around with this today and below is the list of commands i used to both publish and subscribe.

//connecting the device to data network

//Is the SIM ready? does it require a pin AT+CPIN?

+CPIN: READY OK //check network strength Signal Quality report lower the first number the better 99 = unknown AT+CSQ +CSQ: 14,99 OK //Check network registration status, 2nd number should be 1 to show that its registered AT+CGREG? +CGREG: 0,1 OK //show operator selection AT+COPS? +COPS: 0,0,"O2 - UK giffgaff",7 //Get Network APN in CAT-M Or NB-IOT AT+CGNAPN +CGNAPN: 0,"" OK //Set prefered modem selection 2 Automatic,13 GSM only,38 LTE only,51 GSM and LTE only AT+CNMP=13 //select the APN 0,deactive,1 active, 2 auto active AT+CNACT=1,"giffgaff" OK +APP PDP: ACTIVE //Get ip address of modem AT+CNACT? +CNACT: 1,"100.71.118.22" OK //Set the verisign ca cert into device at+csslcfg=convert,2,verisignca.pem //set the device connection to use the device certs at+csslcfg=convert,1,cert.pem,privatekey.pem //set ssl version to use at least 1.2 AT+CSSLCFG="sslversion",0,3 //view current connection AT+SMCONF? //Set up the certs for the SecureMqtt connection //rootca and device cert AT+SMSSL=1,verisignca.pem,cert.pem AT+SMCONF="URL",axxxxxxxxxu.iot.eu-west-1.amazonaws.com,8883 //Make sure you set the clientid AT+SMCONF="clientid","SIM7000" //Connect to the MQTT broker AT+SMCONN //Publish a message //default policy only allows you to publish on clientid topics,number of characters,QOS(only 0 works),Retain (only 0 works) AT+SMPUB="SIM7000/test",8,0,0 > 12345678 OK AT+SMSUB="SIM7000/#",0 AT+SMSUB="SIM7000",1
chillenberger commented 2 years ago

Here is a quick list of things I had to do to get the SIM7000A to work with MQTTS:

I hope this helps someone.

reddy9698 commented 2 years ago

I've been meaning to implement the process mentioned by @davegravy in a program. This is the program that I'm using in the void loop part.

`modem.sendAT("+SMCONN"); if (modem.waitResponse(1000L, res) == 1) { res.replace(GSM_NL "OK" GSM_NL, ""); Serial.println(res); Serial.println("connection!!!!"); }

res="";

modem.sendAT("+SMPUB = \"BasicPubSub\", 425,0,0");
    if (modem.waitResponse(10000L, res) == 1) 
    {
        res.replace(GSM_NL "OK" GSM_NL, "");
        Serial.println(res);
        Serial.println("datssent");
        res="";

        SerialAT.print(jsonBuffer);

    }

modem.sendAT("+SMDISC");
    if(modem.waitResponse(1000L,res) == 1)
    {
        Serial.println(res);
        res="";
        Serial.println("disconnectedf");

    }   

    res="";

}`

and this is the message that arrives on the topic

`AT+SMCONN AT+SMPUB = "BasicPubSub", 425,0,0 AT+SMDISC AT+SMCONN

AT+SMPUB = "BasicPubSub", 425,0,0

AT+SMDISC

AT+SMCONN

AT+SMPUB = "BasicPubSub", 425,0,0

AT+SMDISC

AT+SMCONN

AT+SMPUB = "BasicPubSub", 425,0,0

AT+SMDISC

AT+SMC `

What is the mistake that I am doing ?

Scrts commented 1 year ago

I have succeeded MQTTS to AWS using SIM7070G (Firmware B11) and hologram network. Follow @davegravy flow, BUT be sure to update module time manually or using NTP to the current time! It did not work without updating to the correct time!

polpol commented 1 year ago

Hoping someone can give me a 101 on generating / collecting the Certs because that's where I'm stuck.

I'm using a HiveMQ broker that's an AWS server. Can someone point me to a detailed how-to on creating / saving the certificates? I think I pulled one, put it in the customer folder, but no matter what I do, I can't get at+csslcfg=convert,2,AWSCert.pem to get accepted by the SIM7000. Always sends an ERROR code back. I can't tell if it's because the file itself is an issue or something else - like wrong file location.

I've opened up the board I have using QPST, and it doesn't contain any of the folders we see in the CFSWFILE documentation: 0 "/custapp/" 1 "/fota/" 2 "/datatx/" 3 "/customer/" Tried to load a Cert using CFSWFILE but Putty sucks. Anyone got a better tool to use? I know I should probably not be using Win OS.

I created my own >>customer<< folder and put the Cert there. Also put the Cert in about 10 other folders to see if I could find the location it needs to be in, didn't work. I'm using B04 firmware.

reddy9698 commented 1 year ago

Once you open up the QPST tool, open start clients> EFS Explorer. The EFS Explorer will show you the primary file system by default. On the tool bar you will find an option for an "Alternative File System". You will find the required folders in this and inside the customer folder you can just drag drop the cert files that you want. (just make sure that the names and extensions are right when you are running AT commands).

polpol commented 1 year ago

The EFS Explorer will show you the primary file system by default. On the tool bar you will find an option for an "Alternative File System".

This worked beautifully, thank you! I thought I was going crazy.

Now I just need the right certificates...

erocam2012 commented 1 year ago

@botletics I tried following the AWS IoT command log you posted however I am getting ERROR at AT+SMCONN.

How can I debug this? The same certificates and key allowed me to use their Python SDK to publish successfully.

AT+CCLK?
+CCLK: "21/05/31,04:30:24+00"

OK
AT+CPIN?
+CPIN: READY

OK
AT+CSQ
+CSQ: 29,99

OK
AT+CGREG?
+CGREG: 0,1

OK
AT+COPS?
+COPS: 0,0,"ROGERS ROGERS",7

OK
AT+CGNAPN
+CGNAPN: 1,"ciot"

OK
AT+CNACT=1,"ciot"
OK

+APP PDP: ACTIVE
AT+CNACT?
+CNACT: 1,"10.237.129.141"

OK
at+csslcfg=convert,2,AmazonRootCA1.pem
OK
at+csslcfg=convert,1,my_thing.cert.pem,my_thing.private.key
OK
AT+CSSLCFG="sslversion",0,3
OK
AT+SMSSL=1,AmazonRootCA1.pem,my_thing.cert.pem
OK
AT+SMCONF=url,##############-ats.iot.us-east-1.amazonaws.com,8883
OK
AT+SMCONF="clientid","basicPubSub"
OK
AT+SMCONF="KEEPTIME",60
OK
AT+SMCONF?
+SMCONF
CLIENTID: "basicPubSub"
URL: "##############-ats.iot.us-east-1.amazonaws.com:8883"
KEEPTIME: 60
USERNAME: ""
PASSWORD: ""
CLEANSS: 0
QOS: 0
TOPIC: ""
MESSAGE: ""
RETAIN: 0

OK
AT+SMCONN
ERROR

I also tried setting a topic and message (same topic that the python client successfully publishes to) with no luck.

You have a working code sample that connects to AWS MQTT via SIM70XX and ESP32 AND listens to a topic?

Scrts commented 1 year ago

AT+CCLK? +CCLK: "21/05/31,04:30:24+00"

Fix the time. It has to be synchronized. You can use NTP for that.

URL: "##############-ats.iot.us-east-1.amazonaws.com:8883"

Remove '-ats' from the URL. You also need to use legacy certificate. See "Supported legacy endpoints" section here: https://docs.aws.amazon.com/general/latest/gr/greengrass.html

the '-ats' has been stripped from the endpoint address.

You have a working code sample that connects to AWS MQTT via SIM70XX and ESP32 AND listens to a topic?

My example here works for me when time is synchronized with NTP: https://iot.stackexchange.com/questions/6347/connecting-cellular-module-sim7070g-to-aws-mqtt

botletics commented 1 year ago

Hey guys, figured out how to connect with SSL without verifying certs! Not sure if this would work with AWS though... Please do the following:

zibyan123 commented 1 year ago

I finally figured this out!

I now have AT+SMCONN succeeding with AWS (haven't tested anything beyond this yet)

There's 2 important non-obvious steps.

  1. Whatever url AWS gives you for your IoT Core endpoint, you have to strip out the "-ats" from it. So "a1k9ecig9j720o-ats.iot.us-east-1.amazonaws.com" becomes "a1k9ecig9j720o.iot.us-east-1.amazonaws.com"
  2. You need to use the legacy root certificate provided by AWS here under "VeriSign Endpoints (legacy)". NOTE some regions apparently don't support legacy certs, so it seems the SIM7000 won't work in those.

I also updated my SIM7000A to the B04 firmware from B03, not sure if that mattered.

Contrary to what @tomlankhorst posted, it doesn't seem to matter if your RTC is sync'd.

I finally figured this out!

I now have AT+SMCONN succeeding with AWS (haven't tested anything beyond this yet)

There's 2 important non-obvious steps.

  1. Whatever url AWS gives you for your IoT Core endpoint, you have to strip out the "-ats" from it. So "a1k9ecig9j720o-ats.iot.us-east-1.amazonaws.com" becomes "a1k9ecig9j720o.iot.us-east-1.amazonaws.com"
  2. You need to use the legacy root certificate provided by AWS here under "VeriSign Endpoints (legacy)". NOTE some regions apparently don't support legacy certs, so it seems the SIM7000 won't work in those.

I also updated my SIM7000A to the B04 firmware from B03, not sure if that mattered.

Contrary to what @tomlankhorst posted, it doesn't seem to matter if your RTC is sync'd.

I am using sim7000g.I have used RSA 2048 bit key: [VeriSign Class 3 Public Primary G5 root CA certificate] but still I am unable to connect with aws. AT+SMCONN returns error. my firmware version is also upgraded.Need help.?

botletics commented 1 year ago

Please see this

wenjun1972 commented 1 year ago

I was stuck at the same situation of my 7080G that AT+SMCONN was failed by error,

I followed the guidance from https://github.com/botletics/SIM7000-LTE-Shield/issues/58 but still go nowhere.

"Whatever url AWS gives you for your IoT Core endpoint, you have to strip out the "-ats" from it. So "a1k9ecig9j720o-ats.iot.us-east-1.amazonaws.com" becomes "a1k9ecig9j720o.iot.us-east-1.amazonaws.com" You need to use the legacy root certificate provided by AWS here under "VeriSign Endpoints (legacy)". NOTE some regions apparently don't support legacy certs, so it seems the SIM7000 won't work in those."

I also checked AT+CCLK and saw the time was correct.

Any suggestion ?

thanks !

mrbacani commented 10 months ago

@wenjun1972 I'm having the exact same issue with the 7080G. Were you able to figure out how to correct it?

Scrts commented 10 months ago

@wenjun1972 I'm having the exact same issue with the 7080G. Were you able to figure out how to correct it?

Please copy the whole AT command sequence after you power up.