This is a guide for building an application or integration for The Things Stack.
🚧 This guide is work in progress.
This guide follows a number of best practices that will help you build scalable, reliable and yet simple applications:
The architecture of the application we'll build looks as follows:
placeholder.example.com
as a placeholder, and integration.example.com
as "your domain".Connect to the server over SSH:
% ssh username@integration.example.com
Make sure the system is up to date:
% sudo apt-get update
% sudo apt-get dist-upgrade
% sudo snap install core
% sudo snap refresh core
ℹ️ You may want to reboot your server at this point.
Next, install some packages that we'll need to install the rest of the server:
% sudo apt-get install ca-certificates curl git gnupg lsb-release
It will be useful to have this guide and example files on your server, so let's download the entire cookbook.
% git clone https://github.com/htdvisser/lorawan-stack-application-cookbook cookbook
% curl -fsSL https://nginx.org/keys/nginx_signing.key \
| sudo gpg --dearmor -o /usr/share/keyrings/nginx-archive-keyring.gpg
% echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \
http://nginx.org/packages/ubuntu `lsb_release -cs` nginx" \
| sudo tee /etc/apt/sources.list.d/nginx.list > /dev/null
% sudo apt-get update
% sudo apt-get install nginx
% sudo systemctl enable nginx --now
% sudo snap install --classic certbot
We'll set up NGINX as a reverse proxy, with HTTPS certificates from Let's Encrypt that we'll request with Certbot.
ℹ️ tl;dr if you are already familiar with NGINX:
- Proxy
https://integration.example.com/post/*
tohttp://127.0.0.1:4195
Copy the NGINX configuration from the cookbook directory to the NGINX config directory:
% sudo cp cookbook/nginx/nginx.conf /etc/nginx/nginx.conf
Copy the HTTP configuration from the cookbook directory to the NGINX config directory:
% sudo cp cookbook/nginx/conf.d/integration-http.conf \
/etc/nginx/conf.d/integration-http.conf
Replace occurrences of placeholder.example.com
in the HTTP configuration with your domain name (integration.example.com
):
% sudo sed -i 's/placeholder.example.com/integration.example.com/g' \
/etc/nginx/conf.d/integration-http.conf
Test the config, and reload NGINX:
% sudo nginx -t && sudo systemctl reload nginx
Use Certbot to request a Let's Encrypt certificate.
⚠️ Make sure to replace
integration.example.com
andhello@example.com
before running this command:
certbot certonly \
-n --agree-tos --force-renewal \
--webroot -w /var/www/_letsencrypt \
-d integration.example.com \
--email hello@example.com --no-eff-email
Now we can copy the HTTPS configuration from the cookbook directory to the NGINX config directory:
% sudo cp cookbook/nginx/conf.d/integration-https.conf \
/etc/nginx/conf.d/integration-https.conf
Replace occurrences of placeholder.example.com
in the HTTPS configuration with your domain name (integration.example.com
):
% sudo sed -i 's/placeholder.example.com/integration.example.com/g' \
/etc/nginx/conf.d/integration-https.conf
Generate Diffie-Hellman keys on your server:
% sudo openssl dhparam -out /etc/nginx/dhparam.pem 2048
Again, we test the config, and reload NGINX:
% sudo nginx -t && sudo systemctl reload nginx
If everything worked, you'll see:
Finally, in order to make Certbot automatically reload NGINX when it updates certificates, copy reload.sh
from the cookbook directory to the Certbot hooks directory:
% sudo cp cookbook/nginx/reload.sh /etc/letsencrypt/renewal-hooks/post/nginx-reload.sh
% curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
| sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
% echo "deb [signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] \
https://download.docker.com/linux/ubuntu `lsb_release -cs` stable" \
| sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
% sudo apt-get update
% sudo apt-get install docker-ce docker-ce-cli containerd.io
% sudo systemctl enable docker containerd --now
% sudo usermod -aG docker $USER && newgrp docker
% mkdir -p ~/.docker/cli-plugins/
% curl -fsSL https://github.com/docker/compose/releases/download/v2.1.1/docker-compose-linux-x86_64 \
-o ~/.docker/cli-plugins/docker-compose
% chmod +x ~/.docker/cli-plugins/docker-compose
If everything worked, you can now run docker run --rm hello-world
and you'll see something like:
[...]
Hello from Docker!
This message shows that your installation appears to be working correctly.
[...]
The ingest pipeline is responsible for accepting Webhooks from The Things Stack, doing some basic filtering, and then publishing the messages on Pub/Sub.
The ingest pipeline consists of two files:
ingest/benthos.yaml
defines the configuration for our Benthos pipeline.ingest/compose.yaml
defines the containers that will run in the deployment: Benthos, Redis, RabbitMQ and Mosquitto.Each of these files contains blocks that can be commented out if you only want to use one pub/sub system.
Before we can deploy the pipeline, we need to generate an API key that we'll give to The Things Stack to authenticate webhooks:
% openssl rand -base64 24 | tee cookbook/ingest/apikey.txt
Now let's deploy the ingest pipeline:
% cd cookbook/ingest
% docker compose up -d
The processing pipelines are responsible for subscribing to Pub/Sub, and actually processing each message.
There are currently three processing pipelines:
process-stdout
: Writes messages to standard output. This pipeline also shows how to subscribe to RabbitMQ and MQTT. The other pipelines only use Redis Streams.process-clickhouse
: Writes messages to a Clickhouse database.process-mongo
: Writes messages to a Mongo database.The processing pipelines consist of two files:
process-*/benthos.yaml
defines the configuration for our Benthos pipeline.process-*/compose.yaml
defines the containers that will run in the deployment: for now just Benthos.Let's deploy the processing pipeline that writes to standard output:
% cd cookbook/process-stdout
% docker compose up -d
You can now start following the logs to see messages come in when you enable the webhook integration in The Things Stack:
% docker compose logs -f
In The Things Stack, go to your application, then Integrations and Webhooks.
Click Add Webhook, then Custom webhook and configure it as follows:
https://integration.example.com/post
.X-APIKey
with as value the API key that we generated into cookbook/ingest/apikey.txt
earlier./up
.Now save the webhook configuration.
If you have active end devices in your application, you'll now start seeing messages come in. Otherwise you can go to a device in the Console of The Things Stack and simulate uplinks from the Messaging tab.
See enhancement issues.