search

efento / NB-IoT-sensors-integration

This quick tutorial will show you how to set up a simple CoAP server, which receives the data from Efento NB-IoT sensors and saves it to a PostgreSQL database.
Apache License 2.0
7 stars 2 forks source link

readme

NB-IoT-sensors-integration

This quick tutorial will show you how to set up a simple CoAP server, which receives the data from Efento NB-IoT sensors and saves it to a PostgreSQL database. In this tutorial we are using Python and PostgreSQL database, but the same idea can be easily implemented in other programming languages / with different databases. Should you have any issues or questions, feel free to drop us a line at help.efento.io

How does it work?

Efento NB-IoT sensors send the data in Protobuf format using CoAP protocol. This guarantees fast transmissions and small size of data, which results in up to 10 years battery life time. Moreover, as both CoAP and Protobuf are popular standards, it’s easy to integrate Efento wireless sensors with any cloud platform or custom application. To learn more about CoAP and Protobuf, please visit our Knowledge Library. The Python Script we are going to write sets up the CoaP server. The server is constantly listening for data sent by Efento NB-IoT sensors. Once a new message arrives, the server parses the data, saves it in the PostgreSQL database and responds to the sensor with confirmation that the message has been received (code 2.01 “CREATED”). This means that the message has been successfully parsed and saved in the database. If anything goes wrong (e.g. database is down), the sensor will receive a response with code 5.00 “INTERNAL_SERVER_ERROR” . In that case, the NB-IoT sensor will retry to send the same data after a while.

Important! The response time from the server to the sensor should be as short as possible. In real-life integrations, the data received by the server should be pushed to a queue, and server should reply with 2.01 CREATED as quickly as possible. Saving the data to the database before responding to the sensor decreases the sensor’s battery life time.

On top of the messages with the measurements, the sensor sends three other types of messages:

Before you start

Before you start this, you will need to install and configure:

You will also need:

PostgreSQL database

After downloading and installing PostgreSQL you will need to create the first database. This is one of the steps during the PostgreSQL installation. By default, the database will be created with the following credentials:

DATABASE_HOST = 'localhost' DATABASE_USER = 'postgres'; DATABASE_PASSWORD = 'Your password'; DATABASE_NAME = 'postgres';

If you want to, you can change the names / credentials. Write them down, as they will be needed in the next steps. If you want to check database credentials, open pgAdmin in the PostgreSQL folder. Next open Object -> Properties -> General

Create table

To save the measurements coming from Efento Gateway in your database, you need to create a table. In this example, we are creating a very simple table to store all the data from the sensors, no matter what the sensor type. The table will have 6 columns, all of them of “text” type. Please note that this architecture of the database is only for the demonstration purposes. Database structure should be selected according to your project requirements.

You can create the table manually, using pgAdmin’s interface or using a SQL query. In pgAdmin select your database, open Tools menu: Tools -> Query Tools. Copy the request below into the Query Editor and click Execute (▶) :

CREATE TABLE measurements (
measured_at text ,
serial_number text ,
battery_ok text ,
type text,
channel text,
value text);

CREATE TABLE will create a new, initially empty table in the current database. The table will be owned by the user issuing the command.

CoAP Server

Before you start

The script uses a bunch of libraries. Before you start, you will need to download and install the following libraries:

Compiling Protocol Buffers

Protocol buffers (or Protobuf) are a method of serializing the data that can be transmitted between microservices, applications or used in communication between IoT devices and servers. Protocol Buffers have the same function as well known and widely used JSON and XML. Like JSON and XML, the Protobufs are language-neutral and platform-neutral. Moreover, Protobuf is optimised to be fast and use as little network bandwidth as possible by minimizing the size of transmitted data. This makes Protobuf a perfect choice for serializing the data sent by the battery powered IoT devices.

Unlike JSON and XML, the data and context are separated in Protobuf. Context is defined in the configuration files called proto files (.proto). These files contain field names along with their types and identifiers (eg. string first_name = 1; string surname = 2;), based on the configuration fields Protobuf data is being serialized. The proto files can be compiled to generate code in the user’s selected programming language – a class with setters and getters for all the fields defined in the proto file. Currently, Google provides a code generator for multiple languages including C++, C#, Dart, Go, Java and Python under an open source license.

In order to compile the proto file to a Python class, you will require a protobuf compiler. If you don’t have the compiler yet, download the compiler from its official Github repository: https://github.com/protocolbuffers/protobuf/releases. You need to download the compiler dedicated for the operating system you are using (eg. for Windows download protoc-21.7-win64.zip and unzip it)

Once you have the protbuf compiler, download the proto files, unzip the folder and place the files in the same directory as the protobuf compiler (if you are using Windows .../protoc-3.17.1-win64/bin/). Run the protocol buffer compiler protoc on your .proto files - open a terminal window and enter:

protoc --python_out=.. proto_measurement_types.proto
protoc --python_out=.. proto_measurements.proto
protoc --python_out=.. proto_device_info.proto
protoc --python_out=.. proto_config.proto
protoc --python_out=.. proto_rule.proto

This will generate:

in your specified destination directory. Move the files to your project directory (your_python_project/protos/protos_files). Note! The repository with the sample code already contains classes resulting from compiling the proto files. If you use the sample code, you can skip this step.

NB-IoT sensor configuration

Configuration of Efento sensors is done with a free mobile application for Android. Application can be downloaded from Google Play. Once you download and install the application select “Nearby sensors” mode and unlock the power user mode: open the application menu and quickly tap the Efento logo five times.

Before you start the configuration, make sure the sensor is able to register in the NB-IoT network and the APN settings are right and the APN you use allows the device to connect to the server. Detailed user manual of Efento NB-IoT sensors and Efento mobile application can be found in the support section of our website.

Using the mobile application, connect to the sensor -> click on the menu (three dots in the upper right corner) -> Cellular network status. Mare sure that the field “Registration status” value is either “REGISTERED” or “REGISTERED_ROAMING”

Set the sensor to send the data to your server. Connect to the sensor -> click on the menu (three dots in the upper right corner) -> Power user -> Server configuration. Select “Other” and fill in the IP address and the port of the server. Note! The IP address used for setting up the CoAP server must be a static, public IP. If you are running the script on your computer, make sure you set the port forwarding on your router right.

Results

When you run the script, all the data coming from the sensor will be saved in the database. To view the measurements open pgAdmin 4, select your database, then open Tools > Query Tools. Enter the request below into the Query Editor and select Execute (▶) :

SELECT * FROM measurements;

Moving on - adding two way communication

All parameters of Efento NB-IoT sensors can be changed remotely, from a server. The new configurations or server requests are sent in the responses (ACK) to the messages sent by the sensor.

We are going to modify the script from the example above to add a “device info” request sent from the server to the sensor. Device info is used by the sensor to send detailed information about the sensor's operations and radio-related statistics to the server. In order to increase the battery lifetime, this information is sent by the sensor only at the server's request. The server can request "device info" by sending a proper flag in the response to any of the sensor's confirmable message.Once the server receives the device info frame from the sensor, it will parse it and log to a file.

Device info message sent by Efento sensor contains information about:

How does it work?

The CoAP server is constantly listening for data sent by Efento NB-IoT sensors. Once a new message arrives, the server parses the data, saves it in the PostgreSQL database and responds to the sensor with request device info and confirmation that the message has been received (code 2.01 “CREATED”). The response sent by the server will also contain a “Device info flag” - once the sensor receives it, it will send a second message with the detailed statistics (device info). The same approach can be applied to change the sensor's settings.

Results

When you run the script, all the data coming from the gateway will be saved in the database. To view the measurements open pgAdmin 4, select your database, then open Tools > Query Tools. Enter the request below into the Query Editor and select Execute (▶) :

SELECT * FROM measurements;

On top of that, after each confirmable message, the sensor will send a “Device Info” message which will be saved in the Deviceinfo.txt file