tbec
commented
5 years ago Okay the answer is symmetric key encryption, where device encrypts with a key and server decrypts with the same key. Then you can just verify the file integrity with something arbitrary, like all the column headers are in place, the filename is accurate, the file has at least two lines (header and a data line)
We need to set up a system that stores data while the device is offline (but still powered on) and when it reconnects to a network it will upload offline data to the server. Data is stored in SD card in the same format it is saved in the database (same columns) and is also stored every minute. The names of the SD card files should be something like YYYY-MM-DD.csv. We store about 120B of data every minute, so the maximum file size will be less than
60min * 24hr * 120B =200KB. Transfer speed for HTTPS with TCP socket should be around 1 MB/s so hopefully file transfers are instant. We'll need to track when it goes offline and when it comes back online, or how many offline entries we store in the SD card, or something. When it comes back online it willl upload the entire file in its own thread. If the board is constantly going offline, will this put a load on the server? Maybe limit it to only sending files once an hour or once a minute or something.A few difficulties here. We'll have to track if the file was uploaded. For instance, say we come back online, start to upload a file, then lose connection, and this happens over the course of several days (several csv files). We'll have to make sure all files gets uploaded. We'll also need to keep this info across power failures. For instance, say we're offline, then the device loses power, reboots, and goes back online. We need to send the missed data.
Serverside is the difficult part. Lots of questions. Can you insert packets at arbirary timestamps in Influx? I want to say no... If not then we would have to sort the data based on the device's POSIX timestamp, meaning the Influx timestamp would be completely useless. This would make the CS people very unhappy. It also makes queries a lot more difficult. The answer is probably to just store this data in another database, then combine the data when we want. Another issue is securely sending packets. I'm guessing we'll just POST files to a specific route on the server, which means anyone can post files. What security do we use to stop this? Maybe some kind of hash so that the server can verify the file actually came from the board, and throw it out if not. Will this end up being a heavy load on the server?
One more note. I compressed a 173KB file into a 800B file (200x!). Can you hash and compress? It would be great to send <1KB files