sparkfun / phant

the data logging engine behind data.sparkfun.com
700 stars 99 forks source link

Data burst #203

Closed megavolts closed 6 years ago

megavolts commented 7 years ago

I am running my own server, and I was wondering how I can send data by burst (every 15 minutes) that I collected every 30 seconds ? rather than sending every 30 seconds ? In the future I have the idea to save the data if no connection is available and then retry to post later on.

astroeng commented 7 years ago

I am going to be doing a similar thing. I ended up adding a local date stamp (generated on my end) to the stream. At the moment though my code is still just uploading every 2 minutes. Internally my theory is to just queue up the messages and once enough are in the buffer send them all with one or two seconds between messages.

https://data.sparkfun.com/easy_weather2

utdrmac commented 7 years ago

@astroeng via the HTTP API, how are you blast/bulk POSTing? What's your URL string for multiple data sets?

davidcie commented 7 years ago

Tried to submit data simply adding an ISO-formatted timestamp parameter &timestamp=2017-03-10T10%3A44%3A00.000Z (%3A being the url-encoded colon) but sadly that did not work.

This is important from a remote battery powered device point of view, where enabling RF, establishing connection, getting an IP and finally communicating with the server are by far the most energy consuming activites (~80-90mA over 5s) compared to data gathering which is almost instantaneous and needs very little with the sensors we use (~15mA over less than 1s). This capability would translate into months more use before we need to change batteries.

davidcie commented 7 years ago

Turns out phant simply overwrites timestamp on this line.

In case anyone's interested I have a fork that tries to parse user submitted timestamp here and does new Date() only if none has been found or could not be parsed successfully. It is probably best to submit ISO-formatted timestamp (example above) but any Javascript compatible format will do. Just beware that if you do timestamp=2010,0,30,12,13,14 it will create a date of January 30, 2010 at 12:13:14 in your server's timezone, not UTC. If there's interest I might try to add a special path that does a new Date(Date.UTC(y,m,d,h,m,s)) if string contains six numbers separated by five commas. Perhaps looks a little easier on the eyes than url-encoded colons.

bboyho commented 6 years ago

Phant is No Longer in Operation

Unfortunately Phant, our data-streaming service, is no longer in service and will be discontinued. The system has reached capacity and, like a less-adventurous Cassini, has plunged conclusively into a fiery and permanent retirement. There are several other maker-friendly, data-streaming services and/or IoT platforms available as alternatives. The three we recommend are Blynk, ThingSpeak, and Cayenne. You can read our blog post [ https://www.sparkfun.com/news/2413 ] on the topic for an overview and helpful links for each platform.