read reliability on data.sparkfun.com :(

Sequoia commented 9 years ago

I was getting lots of errors during some times of the day when attempting to read a data stream so I wrote a script to check reliability of reads of data.sparkfun.com. Below are the results of 100 reads. Basically, response times are slow & frequently time out.

Is this a known issue?
Is this being addressed or is it a wontfix? I know data.sparkfun is free so I'd understand if this were the case.

Is phant running on multiple instances/cores? Seems like this is either due to too much load or something blocking the event loop. Anyway here's the results & I'll attach my script

test output from 10:13am EST Oct 16

req 0...200   7.018s
req 1...504   30.279s    Gateway Time-out
req 2...200   0.412s
req 3...200   4.324s
req 4...504   30.281s    Gateway Time-out
req 5...200   0.157s
req 6...200   8.461s
req 7...200   0.156s
req 8...504   31.329s    Gateway Time-out
req 9...504   30.122s    Gateway Time-out
req 10..200   1.077s
req 11..200   0.631s
req 12..200   22.509s
req 13..200   0.218s
req 14..504   30.29s     Gateway Time-out
req 15..200   25.434s
req 16..200   0.609s
req 17..200   19.648s
req 18..504   30.478s    Gateway Time-out
req 19..200   15.351s
req 20..200   15.598s
req 21..200   0.82s
req 22..200   14.13s
req 23..504   30.114s    Gateway Time-out
req 24..200   16.902s
req 25..200   0.151s
req 26..200   0.15s
req 27..200   11.271s
req 28..504   30.116s    Gateway Time-out
req 29..504   30.112s    Gateway Time-out
req 30..504   30.112s    Gateway Time-out
req 31..200   8.877s
req 32..200   0.244s
req 33..200   0.179s
req 34..504   30.117s    Gateway Time-out
req 35..200   3.654s
req 36..200   4.053s
req 37..504   30.126s    Gateway Time-out
req 38..200   0.276s
req 39..504   30.139s    Gateway Time-out
req 40..200   3.507s
req 41..200   0.157s
req 42..200   3.392s
req 43..200   0.538s
req 44..200   3.321s
req 45..200   0.181s
req 46..504   30.269s    Gateway Time-out
req 47..504   30.305s    Gateway Time-out
req 48..200   3.931s
req 49..200   1.424s
req 50..504   30.131s    Gateway Time-out
req 51..200   0.661s
req 52..504   30.207s    Gateway Time-out
req 53..200   0.181s
req 54..200   4.373s
req 55..200   0.156s
req 56..200   0.324s
req 57..200   0.17s
req 58..200   0.161s
req 59..200   5.13s
req 60..200   4.686s
req 61..504   30.286s    Gateway Time-out
req 62..200   1.872s
req 63..200   1.853s
req 64..200   0.181s
req 65..200   3.301s
req 66..504   30.271s    Gateway Time-out
req 67..200   0.174s
req 68..200   0.892s
req 69..504   30.112s    Gateway Time-out
req 70..200   1.172s
req 71..200   0.539s
req 72..504   30.189s    Gateway Time-out
req 73..200   0.343s
req 74..504   115.985s   Gateway Time-out
req 75..200   3.312s
req 76..504   30.253s    Gateway Time-out
req 77..504   30.132s    Gateway Time-out
req 78..200   0.159s
req 79..200   18.951s
req 80..504   30.116s    Gateway Time-out
req 81..504   30.35s     Gateway Time-out
req 82..504   30.17s     Gateway Time-out
req 83..200   12.644s
req 84..504   30.163s    Gateway Time-out
req 85..200   0.166s
req 86..200   10.558s
req 87..200   0.17s
req 88..200   12.68s
req 89..200   0.155s
req 90..504   30.111s    Gateway Time-out
req 91..200   11.07s
req 92..504   30.143s    Gateway Time-out
req 93..200   0.235s
req 94..200   0.169s
req 95..504   30.298s    Gateway Time-out
req 96..200   10.272s
req 97..504   30.276s    Gateway Time-out
req 98..200   0.164s
req 99..200   0.485s
done

test script

#!/usr/bin/env node
//npm i async superagent string chalk
//node testdata.sparkfun.com.js
var async = require('async'),
    request = require('superagent'),
    S = require('string'),
    chalk = require('chalk'),
    boo = chalk.red.bold,
    yay = chalk.green;
    publicKey = 'OGzpoQN9bAir9gEj5NDE';

var streamurl = 'https://data.sparkfun.com/output/'+publicKey+'.json';

async.timesSeries(100, getStreamData, theEnd);

function getStreamData(n, done){
  process.stdout.write(S('req ' + n).padRight(8, '.').s);
  var start  = Date.now();
  request.get(streamurl)
    .end(getResHandler(start, done));
}

function getResHandler(start, done){
  return function(err, res){
    var time = ((Date.now() - start)/1000).toString() + 's';
    if(err){
      console.log(
        boo(err.status), ' ',
        S(time).padRight(10).s,
        S(err.message).truncate(30).s
      );
      return done();
    }
    console.log(
      yay(res.status), ' ',
      S(time).padRight(10).s
    );
    done();
  };
}

function theEnd(){
  console.log('done');
}

peruginim commented 8 years ago

I haven't run any test but I can definitely say that requests are sometimes slow. I'm pulling data from my stream to graph with google charts, and completion times can be inconsistent and sometimes very long. My query isn't just pulling the first page, I'm using a filter (data: 'gt[timestamp]=now-1day'), so that will slow down the response time, but I still wish my request could complete faster.

bboyho commented 6 years ago

Phant is No Longer in Operation

Unfortunately Phant, our data-streaming service, is no longer in service and will be discontinued. The system has reached capacity and, like a less-adventurous Cassini, has plunged conclusively into a fiery and permanent retirement. There are several other maker-friendly, data-streaming services and/or IoT platforms available as alternatives. The three we recommend are Blynk, ThingSpeak, and Cayenne. You can read our blog post [ https://www.sparkfun.com/news/2413 ] on the topic for an overview and helpful links for each platform.

sparkfun / phant