paulbartell
commented
1 year ago
- Connect a device to the AWS IoT Core and set up a rule to retransmit/republish these base64 payloads to S3. Then send base64-encoded binary data piece by piece. It comes at a cost of ~30% extra payload length (base64) and extra decoding work on the backend, but brings benefits of integration with Lambdas, SageMaker studio, visualization, etc.
@dizcza : Sending the data via MQTT may be the best choice for your application. Batching samples will be particularly important for your use case. In other words, you may want to buffer and send data for 1, 5, or 10 seconds at a time. Keep in mind that the maximum payload size for an MQTT publish is 128KB on AWS IoT Core and messages are metered in 5KB increments.
The examples in this repository are not optimized for high throughput, so you may need to make some optimizations to the way responses from the server are received to reach your desired throughput.
You might consider Kinesis Data Streams and/or Kinesis Data Firehose and their associated AWS IoT Core rules which allow you to access these APIs via MQTT in a more optimal way without having to base64 encode all of your data.
A time-series database like AWS timestream may also be an option, although it may not be well-suited to high sampling rates.
Regarding the esp32 platform, consider using the esp-aws-iot collection from espressif which includes some of the same libraries included in the repository optimized for the esp-idf platform.
To address the questions from your post directly:
- Connect a device to the AWS IoT Core and use the secure connection to get an authentication token (JWT) that can be passed to API Gateway to secure the endpoint. Then use your S3 upload demo to upload a binary audio stream piece by piece.
The best reference for this is the HTTP S3 Download Demo which demonstrates using the AWS SigV4 library and AWS IoT Credential Provider MQTT api. You will need to modify it as you see fit to use Upload API calls rather than Download API calls. A similar demo is available in the FreeRTOS repository as well.
A minor offset question: I cannot append incoming data in a file located on an S3 bucket, can I?
AWS S3 supports multi-part uploads. While this isn't appending to an existing file, it may fit your use case. Please take a look at the CreateMultipartUpload, UploadPart, and CompleteMultipartUpload API calls for more info.
dizcza
DarshakDev
I've found neither forum nor Github discussion page of yours so I'm asking it here.
I'd like to upload custom data (1024 Hz sampling rate, 10 bytes each) to an S3 bucket. For the purpose of the question, think of it as an audio stream.
One thing that came to my mind is websockets, but it seems you don't support them as there is a number of related issues that have not been addressed.
Natively, the reasoning boiled down to HTTP S3 upload demo. But it sends the data in a non-secure fashion without authentication.
Could someone elaborate on how to for the following two alternatives:
or maybe something else? There are plenty of AWS services with an exponentially-many number of ways of their interactions. I cannot pick the best case to suit this scenario.
For a start, I just want to store the binary data from many devices (one for a start) in a database or S3. Later on, I'd like to visualize the stream of incoming data (a real-time graph) on AWS IoT page. Eventually, I'll apply machine learning stuff on top of that (but that's way too far from the current scope).
Either way, I'll be using ATECC608A module to secure the communication. The data will be sent from ESP32 boards if this matters.
A minor offset question: I cannot append incoming data in a file located on an S3 bucket, can I?