search

botletics / SIM7000-LTE-Shield

Botletics SIM7000 LTE CAT-M1/NB-IoT Shield for Arduino
https://www.botletics.com/products/sim7000-shield
GNU General Public License v3.0
477 stars 215 forks source link

Acquire and add SSL certificates to SIM7000 memory #156

Open JGLJGL opened 4 years ago

JGLJGL commented 4 years ago

Nedd to add SSL certificates to SIM7000. I want to access to https://www.atesens.com, how can I get and add the certificates to the device memory ?

botletics commented 4 years ago

You can add certificates to the SIM7000's EFS memory by using the EFS Explorer tool after installing the QPST tool from here.

RobertMouncer commented 4 years ago

@botletics Where can we download the tool from? That link doesn't exist anymore. Many thanks.

botletics commented 4 years ago

I just tried clicking on the link and it works.

RobertMouncer commented 4 years ago

@botletics Copy link address -> "https://www.dropbox.com/home/SIMCom%20Firmware%20Update%20Files" this isn't a sharing link - this is just your link to your dropbox. Please right click on the folder/file and click share to get a link :) Thanks!

botletics commented 4 years ago

Oh sorry, my bad. You can actually find the correct link on the firmware update wiki or go there directly

JGLJGL commented 4 years ago

Ok thanks, I allready have EFS explorer. But how can I get the certificates needed?

botletics commented 4 years ago

In Chrome there's a little padlock icon next to the URL. Click that and view the certificate.

JGLJGL commented 4 years ago

This is what I get in the QPST EFS Explorer, the icons aren't shown. qpst captura

botletics commented 4 years ago

Make sure your SIM7000 module is powered on with a LiPo battery and connected via USB (make sure the USB cable you're using has data wires, not just power).

JGLJGL commented 4 years ago

I have the battery and also the right USB, I get this error: This port works fine with other applications like Realterm, AT Command Tester,... puerto

botletics commented 4 years ago

Make sure all the USB drivers are installed from the device manager. I just tried with a random board and it detects it just fine.

JGLJGL commented 4 years ago

Ok, now it detects the port, I was trying to use the AT port instead of Diagnostics port

JGLJGL commented 4 years ago

Now the problem is, how do I add the certificates? I supposed something had to be in "customer" folder. EFS

botletics commented 4 years ago

With the folder selected (like you have it in the screenshot above), right click on the right side and click "New" then "File..."

JGLJGL commented 4 years ago

Ok, now I have the certificates, but in .cer format, how could I convert them to .crt format ? I use AT+CSSLCFG="convert",2,"atesens.cer" but gives ERROR.

botletics commented 4 years ago

Try saving it as "atesens.crt" to begin with.

sethivansh6 commented 3 years ago

Try saving it as "atesens.crt" to begin with.

@botletics Hey I tried all the things you stated in sim7000 issues and now I have copied .crt files to flash memory of SIM7000E and I followed the AT COMMANDS as yours in AT commands log 1 and 2. I tried all this on PUTTY. But while following those AT LOGS my Sim7000 got hang on AT+SHCONN for like 10-15 minutes or most of the times it gave an error.

Once it showed OK pr then it showed AT + SHSTATE SHSTATE 0 ......which should be 1 for making an https request right.

Can you tell me the probable reason for all this and any solution if possible.

botletics commented 3 years ago

Try these AT commands which worked for a user.

jugkrit commented 2 years ago

Ok, now it detects the port, I was trying to use the AT port instead of Diagnostics port

Hi, How did you change port from Diagnostics port to AT port in QPST?

stbrie commented 2 years ago

Hi, How did you change port from Diagnostics port to AT port in QPST?

Did you update all of the device drivers in device manager that are associated with the sim7000?

botletics commented 1 year ago

Hey guys, figured out how to connect with SSL without verifying certs! Please do the following:

baconcheese113 commented 1 year ago

@botletics can you point me to the commands you're using to connect without uploading client certs?

I'm unable to get past AT+SHCONN with HTTPS, works fine with HTTP

Here are my logs

AT
OK
AT+CFSINIT
OK
AT+CFSWFILE=3,"httpbin_root_ca.cer",0,1077,10000
DOWNLOAD

OK
AT+CFSTERM
OK
AT+CSSLCFG="convert",2,"httpbin_root_ca.cer"
OK
at+csslcfg="sslversion",1,3
OK
AT+SHSSL=1,"httpbin_root_ca.cer"
OK
AT+SHCONF="URL","https://httpbin.org"
OK
AT+SHCONF="BODYLEN",1024
OK
AT+SHCONF="HEADERLEN",350
OK
AT+SHCONN
ERROR
botletics commented 1 year ago

So the code has recently migrated to another repo for Arduino Library Manager compatibility (you can install the Botletics SIM7000 library directly from the library manager now). Set BOTLETICS_SSL to 1 in the .h file, then in the LTE_Demo change the HTTP_connect() URL to "https://", then comment out the lines like this (line 1026):

// #ifdef SIMCOM_7070 // Use this line if you have the SIM7000G because the 1529B01SIM7000G firmware doesn't seem to run the commands below well
#if defined(SIMCOM_7000) || defined(SIMCOM_7070) // Use this if you have SIM7000A, especially with SSL

and then run the 'G' command, followed by the '2' command. You will see it use AT+CNACT to enable data, then use AT+SHSSL=1,"" without using CFS to load the cert.

BUT I will say that for some reason it seems to work on the SIM7000A and SIM7070G, but not on the SIM7000G... Seems like a firmware thing because on SIM7000G, even AT+SHCONF="BODYLEN",1024 throws me an error. If you can figure it out on yours, let me know!

baconcheese113 commented 1 year ago

I actually got HTTPS working on the SIM7000G last night!

; save this file as 'conn.ttl' or something and run it from tera term Control -> Macro
timeout = 5

sendln 'AT'
wait 'OK'
sendln 'AT+CCLK?'
wait 'OK'
sendln 'AT+CCLK="22/12/01,00:01:00-12"'
wait 'OK'
sendln 'AT+CNACT=1,"hologram"'
wait 'OK'
wait '+APP PDP'
sendln 'AT+CSSLCFG="sslversion",1,3'
wait 'OK'
sendln 'AT+SHSSL=1,""'
wait 'OK'
sendln 'AT+SHCONF="BODYLEN",1024'
wait 'OK'
sendln 'AT+SHCONF="HEADERLEN",350'
wait 'OK'
sendln 'AT+SHCONF="URL","https://httpbin.org"'
wait 'OK'
sendln 'AT+SHCONN'
wait 'OK' 'ERROR'
if result=2 then
    goto ERROR
endif
sendln 'AT+SHSTATE?'
wait 'OK'
sendln 'AT+SHCHEAD'
wait 'OK'
sendln 'AT+SHAHEAD="User-Agent","curl/7.47.0"'
wait 'OK'
sendln 'AT+SHAHEAD="Cache-control", "no-cache"'
wait 'OK'
sendln 'AT+SHAHEAD="Connection", "keep- alive"'
wait 'OK'
sendln 'AT+SHAHEAD="Accept", "*/*"'
wait 'OK'
sendln 'AT+SHREQ="/get?user=jack&password=123", 1'
wait 'OK'
wait '+SHREQ'
sendln 'AT+SHREAD=0, 300'
wait 'OK'
wait '+SHREAD'
sendln 'AT+SHDISC'
wait 'OK'
goto ERROR

:ERROR
    sendln 'AT+CNACT=0'
    wait 'OK'

The main issue is that the modem clock defaults to year 2080 and I believe this is used with verifying the server certs You can do AT+CCLK? to check the time and `AT+CCLK="......" to set it, see this for more info.

I also finally heard back from SIMCOM about the firmware situation on the SIM7000G, it has different versions than the other variants. You can see the release notes here 1529SIM7000G Firmware Release Note.pdf I'm currently using 1529B06SIM7000G

The issue remaining is that I'm unable to connect to specific websites, including my own backend. I'm struggling to figure out why some sites can be connected to and others can't. Here's my kitchen sink macro file, you'll need to rename the extension to .ttl. You should be able to run it immediately after a cold boot. conn.txt

baconcheese113 commented 1 year ago

So even after setting the time, around 10% of sites would fail to connect through HTTPS, this is due to the servers hosting those domains having multiple certificates and/or the default certificate not being correct for the requested domain.

I recommend this video to understand the issue

To solve it you have to enable SNI for the domain you're connecting to before calling AT+SHCONN with this command

AT+CSSLCFG="sni",1,"example.com"
baconcheese113 commented 1 year ago

If you don't care to check the time validity of the cert it can be ignored with

AT+CSSLCFG="ignorertctime",1,0

With this command it's no longer necessary to set the time with AT+CCLK=...

Mark-Wills commented 1 year ago

Guys,

This is some really great information you are laying out here. Firstly, I want to thank you for solving this, and sharing the results with the community. Secondly, I feel like this should be written up somewhere because this issue will come up again and again. I wonder if someone can write a small program that demonstrates connecting to a web site using https and place it in a repo somewhere? I'll be sure to subscribe and star your repo. :-)

With thanks

Mark

On Thu, Dec 1, 2022 at 11:08 PM baconcheese113 @.***> wrote:

If you don't care to check the time validity of the cert it can be ignored with

AT+CSSLCFG="ignorertctime",1,0

With this command it's no longer necessary to set the time with AT+CCLK=...

— Reply to this email directly, view it on GitHub https://github.com/botletics/SIM7000-LTE-Shield/issues/156#issuecomment-1334562943, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFAGDCXFV6ZP4VUHU7FAYW3WLEVXRANCNFSM4KMQDRFA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

baconcheese113 commented 1 year ago

@Mark-Wills How's this https://gist.github.com/baconcheese113/1f0264727fce3fa51a5bb06fa031aed2

botletics commented 1 year ago

I'll include this link in the wiki, thank you!

Mark-Wills commented 1 year ago

This looks fantastic! Only one question at the moment - which modem(s) does this sequence apply to? I couldn't see a reference in the text.

Starred :-)

Mark

On Sat, Dec 3, 2022 at 12:41 AM baconcheese113 @.***> wrote:

@Mark-Wills https://github.com/Mark-Wills How's this https://gist.github.com/baconcheese113/1f0264727fce3fa51a5bb06fa031aed2

— Reply to this email directly, view it on GitHub https://github.com/botletics/SIM7000-LTE-Shield/issues/156#issuecomment-1335989351, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFAGDCRPVU7NQCD33YSQCXDWLKJM3ANCNFSM4KMQDRFA . You are receiving this because you were mentioned.Message ID: @.***>

baconcheese113 commented 1 year ago

@Mark-Wills I only have the SIM7000G to test with, and it should definitely work with all regional variants of the SIM7000. I'd also bet that it works for all SIM7xxxx modems, but I'm much less confident in that. I'm hoping that users with other SIMCOM modems will provide feedback and let me know!

Mark-Wills commented 1 year ago

Okay great - I've literally just ordered the SIM7000E board from the Botletics website (I'm in the UK) so we can perform some tests here in the UK using your technique. Thanks once again for your efforts on everyone's behalf with this issue. I've been on this mailing list for over a year, and have not ordered a board - not because there is anything wrong with it - on the contrary, it's a fantastic little design. My issue was that there didn't seem to be a solution for secure HTTP comms, either due to modem firmware issues, or just the really poor standard of the SIMCOM documentation. None was really sure. The work done here on this list is a tremendous step forward. Thanks again.

Mark

On Sun, Dec 4, 2022 at 9:31 PM baconcheese113 @.***> wrote:

@Mark-Wills https://github.com/Mark-Wills I only have the SIM7000G to test with, and it should definitely work with all regional variants of the SIM7000. I'd also bet that it works for all SIM7xxxx modems, but I'm much less confident in that. I'm hoping that users with other SIMCOM modems will provide feedback and let me know!

— Reply to this email directly, view it on GitHub https://github.com/botletics/SIM7000-LTE-Shield/issues/156#issuecomment-1336522393, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFAGDCQG2WWXWHPKGDJXM5TWLUETJANCNFSM4KMQDRFA . You are receiving this because you were mentioned.Message ID: @.***>

jatayu86 commented 1 year ago

@baconcheese113 I have been trying secure comms through SIM7000C ,but in vain. Unfortunately there are some modem firmware issues I guess in their latest 1351B05SIM7000 firmware. AT+SHCONF="BODYLEN",1024 and HEADERLEN throws error, it cannot go past 60 in size. Maybe for that reason and due to incomplete header info ,my SHCONN is always ERROR. If anyone have figured a way out of this for SIM7000C, I will buy you a drink ! ;)

baconcheese113 commented 1 year ago

@jatayu86 Not sure if it'd be a firmware issue, but can you try resetting the modem with ATZ and then following through the steps I outlined in my gist? It'd help to know exactly where your output starts to deviate. Also, you should try sending a plain HTTP request without specifying certificate data.

jatayu86 commented 1 year ago

@baconcheese113 Plain http requests and mqtt works fine. Here are the steps as followed from the gist for https for SIm7000C Rev 1351B05SIM7000 (latest sent by simcom)

AT+CNACT=1,"airtelgprs.com" <--- OK <--- +APP PDP: ACTIVE

AT+CSSLCFG="sslversion",1,3 OK

AT+CSSLCFG="sni",1,"httpbin.org" +CME ERROR: operation not allowed

AT+SHSSL=1,"" OK

AT+SHCONF="BODYLEN",1024 +CME ERROR: operation not allowed

AT+SHCONF="BODYLEN",64 OK

AT+SHCONF="HEADERLEN",350 +CME ERROR: operation not allowed

AT+SHCONF="HEADERLEN",64 OK

AT+SHCONF="URL","https://www.httpbin.org" OK

AT+SHCONN OK

AT+SHCHEAD OK

AT+SHAHEAD="Content-type","application/json" OK

AT+SHAHEAD="Connection","keep-alive" +CME ERROR: operation not allowed. //going beyond headerlen configured

AT+SHBOD="{\"query\":\"query getMySensors{hubViewer{sensors{serial}}}\",\"variables\":{}}",73 +CME ERROR: operation not allowed. //going beyond bodylen configured

AT+SHBOD="SIMCOMBODY",10 OK

AT+SHREQ="/",1 +CME ERROR: operation not allowed - Somehow my firmware doesn't support giving more than one argument for SMREQ.

AT+SHREQ=1 OK +SHREQ: "GET",400,122. (probably because headers are incomplete)

AT+SHREAD=0,122

400 Bad Request

400 Bad Request

This seems consistent to the findings @botletics had as per below https://github.com/botletics/SIM7000-LTE-Shield/issues/156#issuecomment-1332651957

https://github.com/botletics/SIM7000-LTE-Shield/blob/master/SIM7000%20Documentation/AT%20Command%20Logs/SIM7000_SSL_Log_1.txt

Thusaraka2000 commented 2 months ago

AT+CFSINIT

ERROR Command error..

Get the size of the file Device.crt in custapp

AT+CFSGFIS=0,"Device.crt"

+CFSGFIS: 1224

OK AT+CFSRFILE=0,"Device.crt",0,1224,10000 Free the data buffer

+CFSRFILE: 1224 -----BEGIN CERTIFICATE----- MIIDWjCCAkKgAwIBAgIVAMPrDSrvoyofjTSWegB09RhO/TLMA0GCSqGSIb3DQEB CwUAME0xSzBJBgNVBAskFtYXpvbiXZWIgU2VydmljZXMgTz1BbWF6b24uY29t IEluYy4gTD1TZW0dGxlIFNUPdhc2hpbmd0b24gQz1VUzAeFw0yNDA0MTAxMTQ -----EN

OK

i uploaded the ssl certificate using the AT command tester i was able to load the Device .crt file using FS at commands(it loads sucessfully and shows me that the file exsist inside the custapp folder), but the problem is i am not able to config the ssl certificate and the following error occurs(for security purpose i have removed some parts of my ssl certicates in the comment).can you please help me with this?

AT+CSSLCFG="convert",2,"Device.crt"

ERROR Command error.. file1 file2