Adding support for Nordic nRF9160 NB-IoT/LTE-M modem

[aldras]

[X] I have read the Troubleshooting section of the ReadMe

What type of issues is this?

[(x)] Request to support a new module

[ ] Bug or problem compiling the library [ ] Bug or issue with library functionality (ie, sending data over TCP/IP) [X] Question or request for help

What are you working with?

Modem: NRF9160 Main processor board: Nordic nRF9160DK https://www.digikey.com/en/products/detail/nordic-semiconductor-asa/NRF9160-DK/9740721 TinyGSM version: github master

Question

Hello, I am currently working on trying to add support for the nRF9160 by Nordic LTE NB-IoT/LTE-M modem to TinyGSM. I have been stumble through adding basic HTTP support, but now I am trying to add SSL/HTTPS support through BearSSL (https://github.com/arduino-libraries/ArduinoBearSSL) and this is where I am running into issues. Due to the way I added support for the nRF9160 it does not work correctly with BearSSL. BearSSL keeps trying to request more data from the nRF9160 even though it has no more to give.

Basic description, uses XSEND and XRECV to send and receive data. The nRF9160 does buffer data in some way but it does not read the response from the server until you issue the XRECV command by requesting a number of bytes. It returns as many bytes as it can from the server, up to the number requested. If it has no more bytes to return it returns zero.

I am not familiar enough with how TinyGSM works to resolve some of these issues and would appreciate some help. I can provide full project files, etc. Including a basic Arduino INO sketch SSL script I am using to test with as well.

The non-HTTPS sketch includes full support for the camera, etc. But this simple HTTPS functionality test sketch should run fine on a Feather M0, or Arduino Zero board, perhaps others. nrf9160_ssl_client_working.ino.zip nrf9160_ssl_client_working.ino.zip nrf9160_ssl_client_working.ino.zip

The issue is BearSSL keeps trying to request more data even though it has no more to give, I am attaching the clunky .h file to here. The nRF9160 does have SSL built in but it requires setting a certificate for each website, etc. I am trying to get a more broad approach that uses BearSSL and can hold a certificate store, perhaps on an SD card.

Current platform I am using to develop is a platform to upload/download BMP/JPEG images, easy to confirm if the image is readable when it uploads/downloads.

Also attaching the firmware used, I had to change one thing in the XRECV logic so it reported how many bytes it received BEFORE the received data instead of after. This firmware can be used with the Actinius Programmer to upload via the bootloader. No nRF9160 development software is required.

Pins for the nRF9160 Icarus are as follows... tx-pin = 3 rx-pin = 1 921600 BAUD

Using the following hardware: Adafruit M4 Grand Central https://www.adafruit.com/product/4064 Adafruit 1.8" Color TFT Shield w/microSD and Joystick - v 2 https://www.adafruit.com/product/802 Icarus IoT Board - nRF9160 https://www.adafruit.com/product/4753 OV7670 camera Can be searched off Amazon

Used this guide as a starting point for camera support and went from there… https://learn.adafruit.com/adafruit-ov7670-camera-library-samd51

Thanks for your help,

aldras

tinygsm_work_20210128.zip

[aldras]

Here is an output with DUMP_AT_COMMANDS enabled.

Resetting modem (6s)...
Initializing modem...AT
Ready
OK
AT#XRESET
OK
[13290] ### TinyGSM Version: 0.10.9
[13290] ### TinyGSM Compiled Module:  TinyGsmClientNRF9160
AT
Ready
OK
AT+CGDCONT=0,"IP","iijmio.jp"
OK
AT+CEREG=1
OK
AT+CFUN=1
OK
AT+CMEE=2
ERROR
AT+CGMM
nRF9160-SICA
OK
[13366] ### Modem: nRF9160-SICA
[13366] ### Modem: nRF9160-SICA
AT+CPIN?
ERROR
AT+CPIN?
ERROR
AT+CPIN?
+CPIN: READY
OK
 [OK]
ATI
ERROR
Modem Info: 
AT+CPIN?
+CPIN: READY
OK
Waiting for network...AT+CEREG?
+CEREG: 1,4
OK
AT+CEREG?
+CEREG: 1,4
OK
AT+CEREG?
+CEREG: 2
+CEREG: 1,2
OK
AT+CEREG?
+CEREG: 1,2
OK
AT+CEREG?
+CEREG: 1,2
OK
AT+CEREG?
+CEREG: 1,2
OK
AT+CEREG?
+CEREG: 1,2
OK
AT+CEREG?
+CEREG: 1
+CEREG: 1,1
OK
 [OK]
Connecting to APN YourAPN... [OK]
AT+CGPADDR=0
+CGPADDR: 0,"100.72.62.220"
OK
Local IP: 100.72.62.220

Please enter a website address (www.blah.com) to connect to: https://www.google.com/
Trying: www.google.com:443...AT#XSOCKET=0,1,0
ERROR
AT#XSOCKET=1,1,0
#XSOCKET: 1, 1, 0, 6
OKAT#XSOCKETOPT=1,20,2

OKAT#XCONNECT="www.google.com",443

#XCONNECT: 1
OKgetTime
AT+CCLK?

+CCLK: "21/01/27,07:30:39+36"
OK
AT#XSEND=0,"16030100D6010000D2030300000000A4C1898B5576611F40DEB416BA3D7ACC2A7EA8427EEF2A7C1FAB56E000005ACCA9CCA8C02BC02FC02CC030C0ACC0ADC0AEC0AFC023C027C024C028C009C013C00AC014C02DC031C02EC032C025C029C026C02AC004"
#XSEND: 100
OK
AT#XSEND=0,"C00EC005C00F009C009DC09CC09DC0A0C0A1003C003D002F0035C008C012C003C00D000A0100004FFF0100010000000013001100000E7777772E676F6F676C652E636F6D0001000101000D00160014040303030503060302030401030105010601020100"
#XSEND: 100
OK
AT#XSEND=0,"0A000A0008001700180019001D000B00020100"
#XSEND: 19
OK
Connected!
-------
AT#XRECV=576
#XRECV: 0, 1152
1603030057020000530303601116A02D19F92E401E20BC024EAA02606A44B470A551D7444F574E4752440120CE9B5BF17DF345A75DD670E5DD2119C08C3A2EAC3F63C216E8065F3317C5179ECCA900000BFF01000100000B0002010016030309260B00092200091F0004CB308204C7308203AFA003020102021034A32DA46BCEC9CC05000000008591AF300D06092A864886F70D01010B05003042310B3009060355040613025553311E301C060355040A1315476F6F676C65205472757374205365727669636573311330110603550403130A47545320434120314F31301E170D3231303130353132313330305A170D3231303333303132313235395A3068310B3009060355040613025553311330110603550408130A43616C69666F726E6961311630140603550407130D4D6F756E7461696E205669657731133011060355040A130A476F6F676C65204C4C43311730150603550403130E7777772E676F6F676C652E636F6D3059301306072A8648CE3D020106082A8648CE3D0301070342000487FBA6E3C537A464276C1B948179A642D821CAADA34A816843EAA06C60CDD96915FBAFCAC17D4BEE43D5A9EEC8A4AD6E6FD0DE58AA8623D30EA17615721A2564A382025C30820258300E0603551D0F0101FF04040302078030130603551D25040C300A06082B06010505070301300C0603551D130101FF04023000301D0603551D0E04160414DB9BB8C1122A60E46C3027E3CB9CF9007EA17835301F0603551D2304183016801498D1F86E10EBCF9BEC609F18901BA0EB7D09FD2B306806
OK
AT#XRECV=576
#XRECV: 0, 1152
082B06010505070101045C305A302B06082B06010505073001861F687474703A2F2F6F6373702E706B692E676F6F672F677473316F31636F7265302B06082B06010505073002861F687474703A2F2F706B692E676F6F672F677372322F475453314F312E63727430190603551D1104123010820E7777772E676F6F676C652E636F6D30210603551D20041A30183008060667810C010202300C060A2B06010401D67902050330330603551D1F042C302A3028A026A0248622687474703A2F2F63726C2E706B692E676F6F672F475453314F31636F72652E63726C30820104060A2B06010401D6790204020481F50481F200F00075009420BC1E8ED58D6C88731F828B222C0DD1DA4D5E6C4F943D61DB4E2F584DA2C200000176D2ADE76A000004030046304402205F1EB8AD9A22271B8F1C1425181886A39900856F0616B71B099103E42B9E5186022022B91181A44AE5BA5A26638A91341CD9A4DA4EB62899002239F053F695898D28007700F65C942FD1773022145418083094568EE34D131933BFDF0C2F200BCC4EF164E300000176D2ADE6FE0000040300483046022100A267668425F44340707A39C9EA6C005A1B1C6F1F2E3200DC7233A296B077A29B022100C258712B33EF9E7B20FEFCC32D14699A30879D63C393CF2B687592E720A6400C300D06092A864886F70D01010B050003820101005DCAC09D37C3ECFAB7FF1E1E4EC55702D7CAF35317A1BE560D38A9CCB01F347F8159747D7D52D588B76FE067E1911B438DB90B6CFCE5DEAD76DB802FA24985C91064994647D6AA541ECD
OK
AT#XRECV=576
#XRECV: 0, 1152
940D705A1AD883526A4C729D86455127A1FDF05035AE97F60BA5546403020AB664EDCF391B7B20CBA8681FE98D6C74A38603A16CDA0E3E57818955F6DBD55167928CA386AECEE4BC55E8B944B35F635E8B2009B69BBDEEF7FD528F140BCC7CBA8C2CB736855EAA3D6B09155D316CA5F105556B604B53862971754F0E4FB408A961713E32AB15CD081BB5D354246D3C0552B72380EA8F4421D5BFE7547E967A16B1607514FFDCE8264FA63AB52D94C34FD227FDB4847700044E3082044A30820332A003020102020D01E3B49AA18D8AA981256950B8300D06092A864886F70D01010B0500304C3120301E060355040B1317476C6F62616C5369676E20526F6F74204341202D20523231133011060355040A130A476C6F62616C5369676E311330110603550403130A476C6F62616C5369676E301E170D3137303631353030303034325A170D3231313231353030303034325A3042310B3009060355040613025553311E301C060355040A1315476F6F676C65205472757374205365727669636573311330110603550403130A47545320434120314F3130820122300D06092A864886F70D01010105000382010F003082010A0282010100D018CF45D48BCDD39CE440EF7EB4DD69211BC9CF3C8E4C75B90F3119843D9E3C29EF500D10936F0580809F2AA0BD124B02E13D9F581624FE309F0B747755931D4BF74DE1928210F651AC0CC3B222940F346B981049E70B9D8339DD20C61C2DEFD1186165E7238320A82312FFD2247FD42FE7446A5B4DD75066B0AF9E426305FBE01CC46361AF9F6A33
OK
AT#XRECV=576
#XRECV: 0, 1152
FF6297BD48D9D37C1467DC75DC2E69E8F86D7869D0B71005B8F131C23B24FD1A3374F823E0EC6B198A16C6E3CDA4CD0BDBB3A4596038883BAD1DB9C68CA7531BFCBCD9A4ABBCDD3C61D7931598EE81BD8FE264472040064ED7AC97E8B9C05912A1492523E4ED70342CA5B4637CF9A33D83D1CD6D24AC070203010001A38201333082012F300E0603551D0F0101FF040403020186301D0603551D250416301406082B0601050507030106082B0601050507030230120603551D130101FF040830060101FF020100301D0603551D0E0416041498D1F86E10EBCF9BEC609F18901BA0EB7D09FD2B301F0603551D230418301680149BE20757671C1EC06A06DE59B49A2DDFDC19862E303506082B0601050507010104293027302506082B060105050730018619687474703A2F2F6F6373702E706B692E676F6F672F6773723230320603551D1F042B30293027A025A0238621687474703A2F2F63726C2E706B692E676F6F672F677372322F677372322E63726C303F0603551D20043830363034060667810C010202302A302806082B06010505070201161C68747470733A2F2F706B692E676F6F672F7265706F7369746F72792F300D06092A864886F70D01010B050003820101001A803E3679FBF32EA946377D5E541635AEC74E0899FEBDD13469265266073D0ABA49CB62F4F11A8EFC114F68964C742BD367DEB2A3AA058D844D4C20650FA596DA0D16F86C3BDB6F0423886B3A6CC160BD689F718EEE2D583407F0D554E98659FD7B5E0D2194F58CC9A8F8D8F2ADCC0F1AF39AA7A90427F9A3
OK
AT#XRECV=576
#XRECV: 0, 530
C9B0FF02786B61BAC7352BE856FA4FC31C0CEDB63CB44BEAEDCCE13CECDC0D8CD63E9BCA42588BCC16211740BCA2D666EFDAC4155BCD89AA9B0926E732D20D6E6720025B10B090099C0C1F9EADD83BEAA1FC6CE8105C085219512A71BBAC7AB5DD15ED2BC9082A2C8AB4A621AB63FFD7524950D089B7ADF2AFFB50AE2FE1950DF346AD9D9CF5CA16030300740C00007003001D20778F582E2B6CA1F2D41D11B1C91F48CA9E929DEC131112D0E5927F990F45F35F040300483046022100B3140F44DA002DFB971C8C88C1BE1F06AC53102F8F31946D6B8BB87090867F9502210096811B20BCB1B975DEA53940D4A5A5FFC7FE683C2CB2E79E3116C330EA7A6D6E16030300040E000000
OK
AT#XRECV=265
#XSOCKET: -116
ERROR

[SRGDamia1]

Are these TCP/socket commands now part of the "standard" modem-level firmware or is this depending on someone having programmed the user portion of the nRF9160 with a specific firmware (like one of the examples provided by Nordic)? Last time I looked at the nRF9160 the modem AT commands were all lower level commands and you had to use some sort of additional firmware to provide the AT-like access to sockets.

I have an nRF9160 dev board somewhere in my office, but I didn't ever get too far with it because of the lack of solid support for socket-level AT commands.

[SRGDamia1]

Nordic doesn't even have the example with socket level AT commands available any more.

[aldras]

There is a project example I am working off called "serial lte modem" that has the AT commands, but yeah, it handles the sockets itself it seems, since you can only open one socket at a time it looks like.

Since it is possible to write your own code for this, and all compiling tools are free, it is possible to custom write your own full AT set, but this is a lot of work so am using "serial lte modem" as a starting point. I only had to modify the "AT#XREVC" command so it reported the data length BEFORE the it spit out the data. Before the modification it would spit out the data first, then at the end report the data length. I found that harder to work with.

I think this is would be a great thing to get TinyGSM support for. Let me get more organized and I will upload more items and a full description of a development setup. I have a Adafruit Feather M0 Basic development platform setup and a Adafruit M4 Grand Central development platform working now. The main idea behind the M4 is so I can have fun uploading/downloading small 180 x 160 JPEG images using the camera and TFT screen attached to the M4.

[aldras]

Are these TCP/socket commands now part of the "standard" modem-level firmware or is this depending on someone having programmed the user portion of the nRF9160 with a specific firmware (like one of the examples provided by Nordic)? Last time I looked at the nRF9160 the modem AT commands were all lower level commands and you had to use some sort of additional firmware to provide the AT-like access to sockets.

I have an nRF9160 dev board somewhere in my office, but I didn't ever get too far with it because of the lack of solid support for socket-level AT commands.

Yes, the commands appear to have changed and evolved over the time, was a mess getting started with it at first.

[SRGDamia1]

The "serial lte example" doesn't even exist in the Nordic Playground repo any more. I would love to have this supported, but I'm hesitant to put in any effort when it would necessarily depend on custom and possibly unstable user firmware.

While all the compiling tools are free and there are instructions available, I found that just getting the tool chain set up to program the nRF9160 was A LOT harder than the tool chain for, say, an Arduino-style board.

[aldras]

Ah it is in the 1.4.99-dev1 SDK under nrf/applications/serial_lte_modem I was going to post the git here but its actually a combination of repositories using west (I think its called) to combine multiple git repositories. https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/gs_assistant.html It is much quicker to just install their application and let it download and setup the source tree, then you can build and such with "west" as opposed to "make".

There is some other info here, one of the boards I am using is from this cool guy.

https://docs.jaredwolff.com/nrf9160-getting-started.html

It does take a minute to get the build environment setup though. I can provide some assistance along the way, what platform are you on? I setup MacOS (a couple times).

[aldras]

I would recommend using the nRF Connect application and let it setup everything for you, don't know about Linux support, but it set it up to the point that all I really had to do is say "make" and it worked.

However the make command is more something like this in their SDK, for three different platforms I have worked with so far.

west build -b nrf9160dk_nrf9160ns -p west build -b actinius_icarus_ns -p west build -b circuitdojo_feather_nrf9160ns -p

Then to actually load it it would be something like this... /usr/local/bin/nrfjprog -f NRF91 --program build/zephyr/merged.hex --sectorerase

[aldras]

I am still working on getting an easily duplicatable environment setup.

Can either go with an ESP32-CAM (because of the 4MB PSRAM its easy to make 100k of JPEG data), or with the Adafruit Feather M0 basic. Both are Arduino sketches. Planning on getting a zip file put together with all the libraries, sketches, and wire hookups.

The nRF9160, a completely different setup using the Zephyr RTOS is a little more tricky to setup, but everything is available for free, helps if you have a nRF9160DK or J-Link programmer of some sorts. The bootloaders can be hit or miss sometimes.

[aldras]

I think I have got a working dev environment setup with instructions and such. I am not sure how to get a branch up here, so I forked it and the branch with some nRF9160 support is here.

https://github.com/aldras/TinyGSM/tree/nrf9160_dev

I would like to get that branch into this repository if there are no objections, just so everything is centralized and everyone can work on it easily.

In addition to the branch, a collection of files, firmware, etc, I have uploaded to a VPS server, it is rather large so it is not in the repository. It comes in at around 713MB, but includes rebuilt firmware, documentation, and (partial) source code for the nRF9160 firmware that is being used with TinyGSM. For a full nRF9160 build environment nRF connect along with the "Toolchain Manager" and the "v1.4.99-dev1" SDK from Nordic will need to be installed, after installed the source code can be added in the appropriate place in the source code. Instructions are inside the zip file.

https://www.drassal.net/filestore/tinygsm_nrf9160_dev_20210202.zip

There are two examples added, one is HttpClient_nrf9160, which is basically the HttpClient example with some changes for the nRF9160 and ESP32-CAM platform. The other is esp32_nrf9160_http which is a JPEG image uploader which uses the ESP32-CAM camera to take a picture then upload the picture.

I think I forgot to add one more instruction with the esp32_nrf9160_camera_http, that is that GPIO14 is used to trigger the camera to take a picture and upload it, just short this to ground to automatically take pictures back to back.

The main goal here is to get full TinyGSM support so I can get HTTPS support through BearSSL.

The nRF9160 also has a GPS feature, which I have not added yet into TinyGSM, and I think I need to enable some things in the nRF9160 firmware. Since it is connected to the LTE network it also knows the current time, that feature is implemented in a limited way, there is a function in the esp32_nrf9160_camera_http example that retrieves the current date/time and displays it. For this the time.h "Time" library will be required.

I am continuing to work on this and will try to keep the branch current after I get to a milestone that is stable enough to push. Let me know anything I can do to help resolve issues with those who would like to help out with development.

[SRGDamia1]

The only way to create a branch here would be for me to create one and then you to issue a pull request against it.

You mentioned that you modified the firmware for the 9160. I see in your zip several (6) different firmwares. Which of those are you using? Do you have a separate (forked or not) repo with the src for your firmware?

[aldras]

I have updated the branch here with an example that is based on the "HttpClient" example with only necessary changes. The repository is here. I have done a lot of cleanup and fixes for the nRF9160. The branch below is up to date as of this post.

I have done a lot of cleanup and hope it is a lot easier to work with. I have tested everything with the nRF9160DK that I have here, since it is most likely the nRF9160 board that most developers will have on hand.

https://github.com/aldras/TinyGSM/tree/nrf9160_dev

A note about the examples/HttpClient_nrf9160_adafruit_m0_m4 folder, this might not be a good thing to do, and only for development so all files are easily accessible, but the firmware (except the firmware source) is located in the nRF9160 example at examples/HttpClient_nrf9160_adafruit_m0_m4

This should be cleaned up after it is somewhat stable, but now considering where to put the required prebuilt firmware. I would not recommend merging this in its current state, will most likely better to create a clean branch once support is getting near ready to merge. Perhaps a separate repository for the nRF9160 support files and such would be a good way to go.

About the firmware source, the way that Nordic does it is pull in a number of git repos and merge them, due to that mess I have zipped up the changes that I made to the "serial_lte_modem" application located at "v1.4.99-dev1/nrf/applications" and provide them in this zip file, install the v1.4.99-dev1 SDK normally through the nRF Connect application and then just copy what is in the below repository into the v1.4.99-dev1/nrf/applications folder, build commands and such are located in the "readme.txt" file inside the zip file. I don't know a better way to deal with at the moment, I am open to options. The source is rather large because I left the build folders intact with all the resulting .hex files.

https://github.com/aldras/TinyGSM_nrf9160_firmware

Feel free to fork or copy the above repository.

I am open to options on how to keep this better organized and ease development for those who can help contribute. Since the nRF9160 is somewhat unique in a way that it is actually a LTE modem with a main Cortex-M33 MCU in the same package, this presents another step in making sure the nRF9160 is running the correct firmware (AT command) version for TinyGSM.

[aldras]

I spent some time and finally got up to date, along with using the new v1.8.0 SDK. With version 1.8.0 there is one catch, the method to send/receive HEX data was removed, so I made a small addition to the nRF9160 firmware to accomplish this. Other than this, it is running the stock serial_lte_modem sample application by Nordic. The repo is now up to date (as of this post) with one commit for nRF9160 support.

https://github.com/aldras/TinyGSM

The firmware and source code is in the extras folder. There is an example "HttpClientNrf9160" that explains the details of how to get started and program the nRF9160 board. I have updated the repo above, feel free to fork it and review the code. There are many things not cleaned up, but it does, for the most part, function. I have added the firmware and source under the "extras" folder, which seemed like the best place for it at the moment.

This time around, a nRF9160-DK is not required since programming is possible with the bootloader. A Circuit Dojo nRF9160 Feather and Adafruit Feather M0 was used for development, of course other options are available as well, all written up in the sample sketch. This can also run on the nRF9160-DK development board, but a firmware specifically targeting that board would be ideal, which I can provide if desired. The least expensive way to get started is to grab the nRF9160 Circuit Dojo Feather off DigiKey or off the maker's website.

With this addition, I hope we can clean up the code and get things smoothed out to have it in TinyGSM.

[aldras]

Added firmware for the nRF9160-DK since it seems more people have it on hand than expected at first.

One issue I found with using the nRF9160-DK was that there was no way to connect directly to the nRF9160 RESET pin, I ended up soldering a wire onto the RESET button. However, if you time it right, you can press the RESET button while the MCU is attempting reset.

The pinout information is located in the extras folder also in the following files, depending on which firmware file is programmed: nrf9160_circuitdojo_feather_1.8.0_serial_lte_modem_uart.txt nrf9160_dk_1.8.0_serial_lte_modem_uart.txt

These files have been added in the extras folder for the time being.