Possible size limitation

[B504now]

I am testing a data stream and I noticed something odd. My original stream had 10 fields, with 111 bytes of content. When I added a field, the content went to 118 and the stream failed in update. The additional field caused the packet to be malformed (400).

I recreated the database but with much shorter names. I was able to update all the way to the 15 field limit. The content was 87. When I add real values, this will be closer to 111 but for now, I can proceed.

Can you look to see if there is an issue around 300 ? The header itself has 190 so either phant is having a memory issue, or the Xbee is but something is corrupting the stream.

[B504now]

Attached is a program that demonstrates the issue. Around line 452 in the subroutine buildPhant, is a commented set of lines. //***** phant.add("drn", 1); // phant.add("drn", 12);' //*

Swap the comment and the stream fails.

[B504now]

https://gist.github.com/885391842189cd48689e.git

[toddtreece]

@B504now where are you seeing that there is a 15 field limit? The number of fields that you can define for a stream is 30: https://github.com/sparkfun/phant/blob/master/lib/validator.js#L52

It's very possible that the phant-arduino client has a memory leak, but phant (the server) can accept 50KB of post data with no issues. How much memory is available on your arduino when you add that extra field?

[toddtreece]

@jimblom Could you take a look at this? I don't have the hardware to test.

[B504now]

and this is the message that appears when you try to create a stream over 15

Creating stream failed - too many tags sent. 15 tags max   Bill Dickenson

302 521 3565

On Sunday, August 24, 2014 6:05 AM, Todd Treece notifications@github.com wrote:

@jimblom Could you take a look at this? I don't have the hardware to test. — Reply to this email directly or view it on GitHub.

[toddtreece]

Tags are just used for categorization of streams, and aren't required. The field list is the list that defines the schema for your stream.

Stream creation docs: http://phant.io/docs/management/create

[B504now]

Ahh. Thanks. I was treating them as field names. Thanks - That helps .. now if I can solve my other problem my weather station is ready to go.

Thanks again. Sorry to be asking stupid questions.

  Bill Dickenson

302 521 3565

On Sunday, August 24, 2014 8:32 AM, Todd Treece notifications@github.com wrote:

Tags are just used for categorization of streams, and aren't required. The field list is the list that defines the schema for your stream. Stream creation docs: http://phant.io/docs/management/create — Reply to this email directly or view it on GitHub.

[toddtreece]

They aren't stupid questions at all, I just didn't know where the 15 field limit came from. Hopefully one of our engineers will be able to help sort out the issue soon. I don't have the hardware, or I would give it a shot.

[B504now]

This is not hardware related. Not sure why I didnt think of this earlier, but run this sketch. Around content length 129 things get dicey. The stream (%label=data) vanishes and then around 15 elements the end of the text starts to vanish.

I left you a few things you can switch to see,

include <avr/pgmspace.h>

include

// This is the set of variables we will feed to PHANT --- we have // iData which is the count of data elements // WeatherData which is the float table for all the results. Yes, I waste a bit of memory // lValt has the table of labels for phant so it doesn't stay in memory //
int iData = 1; float WeatherData[] = { 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19};

prog_char wlbl_0[] PROGMEM = "avgwindspeed"; prog_char wlbl_1[] PROGMEM = "cloudcover"; prog_char wlbl_2[] PROGMEM = "gustdir"; prog_char wlbl_3[] PROGMEM = "gustspeed"; prog_char wlbl_4[] PROGMEM = "humi"; prog_char wlbl_5[] PROGMEM = "humidity"; prog_char wlbl_6[] PROGMEM = "interval"; prog_char wlbl_7[] PROGMEM = "lightlvl"; prog_char wlbl_8[] PROGMEM = "lightningdistance"; prog_char wlbl_9[] PROGMEM = "powerlvl"; prog_char wlbl_10[] PROGMEM = "pressure"; prog_char wlbl_11[] PROGMEM = "pressurei"; prog_char wlbl_12[] PROGMEM = "raindaily"; prog_char wlbl_13[] PROGMEM = "rainfallrate"; prog_char wlbl_14[] PROGMEM = "temp"; prog_char wlbl_15[] PROGMEM = "tempi"; prog_char wlbl_16[] PROGMEM = "uv"; prog_char wlbl_17[] PROGMEM = "winddir"; prog_char wlbl_18[] PROGMEM = "windi"; prog_char wlbl_19[] PROGMEM = "windspeed";

PROGMEM const char *weatherLabel[] = // Table of labels {
wlbl_0, wlbl_1, wlbl_2, wlbl_3, wlbl_4, wlbl_5, wlbl_6, wlbl_7, wlbl_8, wlbl_9, wlbl_10, wlbl_11, wlbl_12, wlbl_13, wlbl_14, wlbl_15, wlbl_16, wlbl_17, wlbl_18, wlbl_19 };

char buffer[20]; // make sure this is large enough for the largest string it must hold String tempstring;

define AVGWINDSPEED 0

define CLOUDCOVER 1

define GUSTDIR 2

define GUSTSPEED 3

define HUMI 4

define HUMIDITY 5

define INTERVAL 6

define LIGHTLVL 7

define LIGHTNINGDISTANCE 8

define POWERLVL 9

define PRESSURE 10

define PRESSUREI 11

define RAINDAILY 12

define RAINFALLRATE 13

define TEMP 14

define TEMPI 15

define UV 16

define WINDDIR 17

define WINDI 18

define WINDSPEED 19

// https://data.sparkfun.com/input/rooqN2WA3pSDWaKjLjYA?private_key=jkk7wvr2nZsb5VwPdPyr&baro=28.14&blvl=17.16&cld=3.83&drn=12.09&hum=4.53&lght=22.50&lng=26.22&rn=0.12&tmp=20.69&uv=26.26&was=10.00&wd=6.04&wg=3.76&wgd=26.83&ws=26.81 char response[12]; int returncode = 0;

///////////////// // Phant Stuff // ///////////////// Phant phant("data.sparkfun.com", "rooqN2WA3pSDWaKjLjYA", "jkk7wvr2nZsb5VwPdPyr");

/////////// // Setup // /////////// void setup() { delay(5000); // Set up serial ports: Serial.begin(9600); Serial.print(F("Memory Available is :")); Serial.println(availableMemory()); }

////////// // Loop // ////////// void loop() { Serial.print(F("Start Phant.add with ")); Serial.println(iData); buildPhant(); // Serial.println(F("Start Phant2")); // buildPhant2(); delay(5000); iData++;

};

// *** Utilities *** // free RAM check for debugging. SRAM for ATmega328p = 2048Kb. int availableMemory() { // Use 1024 with ATmega168 int size = 2048; byte buf; while ((buf = (byte ) malloc(--size)) == NULL); free(buf); return size; }

void buildPhant() { for (int i=0;i<iData;i++) { strcpy_P(buffer, (char*)pgm_read_word(&(weatherLabel[i]))); // Necessary casts and dereferencing, just copy. tempstring = buffer; phant.add(tempstring,WeatherData[i]); // Serial.println(tempstring); } Serial.println(phant.post()); }

void buildPhant2() { phant.add("avgwindspeed",1); phant.add("cloudcover",1); phant.add("gustdir",1); phant.add("gustspeed",1); phant.add( "humi",1); phant.add("humidity",1); phant.add("interval",1); phant.add("lightlvl",1); phant.add("lightningdistance",1); phant.add("powerlvl",1); phant.add("pressure",1); phant.add("pressurei",1); phant.add("raindaily",1); phant.add("rainfallrate",1); phant.add("temp",1); phant.add("tempi",1); phant.add("uv",1); phant.add("winddir",1); phant.add("windi",1); phant.add( "windspeed",1); Serial.println(phant.post()); delay(5000); // cleanup(); }

[B504now]

include <avr/pgmspace.h>

include

not sure why they didnt copy

[ToniCorinne]

Again, I think there's something funny going on with your code + your hardware configuration. Trying swapping your XBee over to a digital serial port, and use the Serial monitor to debug as your code runs. My guess is you're running right up to the edge of the phant update rate, and that using delay() instead of the update rate variable as shown in this example is causing the problems you are running into. I would really recommend downloading the example sketch we have on the tutorial and adding in your particular variables to there and see how it works. It may at least give you some more explicit errors that help you track down exactly where the problem is.

[B504now]

Thank you. I actually did just that. Thats in fact how the whole thread started. I took that sketch, added the 20 variables and we started from there. That was the original error. When we added Serial.println(phant,post()); the result looked curiously incomplete. In fact large chunks of the outgoing stream disappeared.  So it throws off a 400 error for a misformed data request. I have had the second Xbee hooked up and tried the xbee to xbee communiations and got exactly the same data stream that the example shows. 

I also have a version with only 8 data element which ran for 42 hours with an occasional 502, but nothing that caused me concern.

The pre-production code has no delay. It relies on the RTC to change state and the updates are every 2 minutes in the example. (now.seconds % SENDINTERVAL) and the production code will update every 10. So i don't think I am running updates too quickly. 

I also posted a new program with NO XBEE and it does exactly the same thing. Since I wasn't transmitting, I simply built the data stream and printed it. Phant.post clears the stream so it was a nice way to check. And without the Xbee, it has no latency.

That stream adds a data variable every iteration (5 second loop) and prints phant.post. Below content 111, it all looks good. After 111, it loses the data stream ( and would produce a 400 error if sent because the content variable shows content but there may not be any. A few more elements, and the content variable seems to vanish also and I have no idea what would be the result. That program is posted. 

This points to phant arduino client having an issue with data streams. Something seems to muck up around 111

Thanks

  Bill Dickenson

302 521 3565

On Wednesday, August 27, 2014 12:25 PM, Toni Klopfenstein notifications@github.com wrote:

Again, I think there's something funny going on with your code + your hardware configuration. Trying swapping your XBee over to a digital serial port, and use the Serial monitor to debug as your code runs. My guess is you're running right up to the edge of the phant update rate, and that using delay() instead of the update rate variable as shown in this example is causing the problems you are running into. I would really recommend downloading the example sketch we have on the tutorial and adding in your particular variables to there and see how it works. It may at least give you some more explicit errors that help you track down exactly where the problem is. — Reply to this email directly or view it on GitHub.

[ToniCorinne]

That stream adds a data variable every iteration (5 second loop) and prints phant.post. Below content 111, it all looks good. After 111, it loses the data stream ( and would produce a 400 error if sent because the content variable shows content but there may not be any. A few more elements, and the content variable seems to vanish also and I have no idea what would be the result. That program is posted.

So the code you have posted above in this issue is simply running off of a single Arduino connected via serial port to your computer, with no other hardware, correct? And this is the complete code that you are running into the issue you are attempting to address here, correct?

If this is the case, then we can dig into the code here. If you do actually have an XBee or additional hardware hooked up and are running into additional issues, then you need to send an email to techsupport@sparkfun, with a full description of your circuitry. A large circuit will need to be more thoroughly debugged than we can really do properly on the GitHub issues.

[B504now]

Nothing is hooked up. Single Uno, USB port. Complete code. At this stage I don't have any other issues. May not be the case after this works but this makes any other issues moot.

Thanks

  Bill Dickenson

302 521 3565

On Wednesday, August 27, 2014 4:03 PM, Toni Klopfenstein notifications@github.com wrote:

That stream adds a data variable every iteration (5 second loop) and prints phant.post. Below content 111, it all looks good. After 111, it loses the data stream ( and would produce a 400 error if sent because the content variable shows content but there may not be any. A few more elements, and the content variable seems to vanish also and I have no idea what would be the result. That program is posted. So the code you have posted above in this issue is simply running off of a single Arduino connected via serial port to your computer, with no other hardware, correct? And this is the complete code that you are running into the issue you are attempting to address here, correct? If this is the case, then we can dig into the code here. If you do actually have an XBee or additional hardware hooked up and are running into additional issues, then you need to send an email to techsupport@sparkfun, with a full description of your circuitry. A large circuit will need to be more thoroughly debugged than we can really do properly on the GitHub issues. — Reply to this email directly or view it on GitHub.

[ToniCorinne]

Ok, cool. We'll try digging into the code in that basic set up then and see if we can duplicate your problem. Thanks!

[B504now]

Any luck reproducing the problem ?

[jimblom]

I get the same behavior as you with your example sketch: after the sample post with a "Start Phant.add with 7" (content length 113), the following posts don't include any of the content keys/values.

I'm just not sure we're going to track this down to anything other than a lack of SRAM in the Arduino. In your example sketch from this comment, if (just to test) you set the public and private keys in the Phant declaration to null (the quickest way I could think of to test and conserve some SRAM), the POSTs look like they'll succeed all the way up to "Phant.add with 14" (content length 244). I think that points to the problem being independent of the implementation of the Phant library.

There's no hard limit within the Phant library for keys and values. The limit will fluctuate with available memory. If you need to implement more fields than the Arduinio is allowing, I think you'll just have to find places to conserve memory.

[B504now]

I added a println for available memory and there is a lot left but I suspect it's running out elsewhere. So you are right, when I cut down the variables to 3 letters, the data value to int, all 20 mangled variables fit.

Since we started with 900 of 1024 free, and we ran out on the original at around 300, it's not the memory most of us can effect. Mine had loaded a massive amount into PROGMEM for that reason. I don't have many programs that run that light.

Guess I can make a run at the HTTP post. Is there a phant.get to pull just the contents out without the headers ? Otherwise this is a bit of a dead end.

Thanks

Bill Dickenson

302 521 3565

On Friday, August 29, 2014 12:59 PM, Jim Lindblom notifications@github.com wrote:

I get the same behavior as you with your example sketch: after the sample post with a "Start Phant.add with 7" (content length 113), the following posts don't include any of the content keys/values. I'm just not sure we're going to track this down to anything other than a lack of SRAM in the Arduino. In your example sketch from this comment, if (just to test) you set the public and private keys in the Phant declaration to null (the quickest way I could think of to test and conserve some SRAM), the POSTs look like they'll succeed all the way up to "Phant.add with 14" (content length 244). I think that points to the problem being independent of the implementation of the Phant library. There's no hard limit within the Phant library for keys and values. The limit will fluctuate with available memory. If you need to implement more fields than the Arduinio is allowing, I think you'll just have to find places to conserve memory. — Reply to this email directly or view it on GitHub.

[toddtreece]

@B504now HTTP responses always have headers. They are part of the protocol, so there isn't really a way around them.

Have you tried printing available memory right before & after the Serial.println(phant.post());? It looks like you are only printing it during setup currently.

[B504now]

sorry - keyboard error. That last not was incomprehensible.

I JUST posted a note about this. Yes, I added a Serial.println(availableMemory()) before phant.post and one after. They give the same result. available memory shrinks proportional to how much is added by phant.add.The numbers are identical. I don't think it's leaking. But as I said in my note, the issue here is that it starts with 900 or 1024 free, and by the time the phant.adds are done there is only a little room. And since it STILL shows 400 free even when phant is long corrupted, it's not a problem we can solve. The stream is 500 characters long.

You misunderstood my comment. I do know http responses need headers... BUT there are three obvious enhancements that could be made to this,

1) override phant.post with a method that does not clear the stream after. Something so you can print the contents of the stream without destroying it. So you could do Serial.println(phant.post(keep)) and follow it with xb.print(phant.post(flush)). Yes you could build it again but thats not great for debugging. 2) the format of the stream datalabel=value& is used by a lot of programs. My muse was that it's too bad that a method that JUST gave you back the data stream without the headers so it could be manipulated. So Serial.println(phant.get()) would return "data1=val1&data2=val2........" 3)Add a check so when the memory is exhausted, it says so instead of just canabalizing the stream.

Thanks

Bill Dickenson

302 521 3565

On Friday, August 29, 2014 2:31 PM, Bill Dickenson b504now@yahoo.com wrote:

I added a println for available memory and there is a lot left but I suspect it's running out elsewhere. So you are right, when I cut down the variables to 3 letters, the data value to int, all 20 mangled variables fit.

Since we started with 900 of 1024 free, and we ran out on the original at around 300, it's not the memory most of us can effect. Mine had loaded a massive amount into PROGMEM for that reason. I don't have many programs that run that light.

Guess I can make a run at the HTTP post. Is there a phant.get to pull just the contents out without the headers ? Otherwise this is a bit of a dead end.

Thanks

Bill Dickenson

302 521 3565

On Friday, August 29, 2014 12:59 PM, Jim Lindblom notifications@github.com wrote:

I get the same behavior as you with your example sketch: after the sample post with a "Start Phant.add with 7" (content length 113), the following posts don't include any of the content keys/values. I'm just not sure we're going to track this down to anything other than a lack of SRAM in the Arduino. In your example sketch from this comment, if (just to test) you set the public and private keys in the Phant declaration to null (the quickest way I could think of to test and conserve some SRAM), the POSTs look like they'll succeed all the way up to "Phant.add with 14" (content length 244). I think that points to the problem being independent of the implementation of the Phant library. There's no hard limit within the Phant library for keys and values. The limit will fluctuate with available memory. If you need to implement more fields than the Arduinio is allowing, I think you'll just have to find places to conserve memory. — Reply to this email directly or view it on GitHub.

[toddtreece]

@B504now

1) You can assign the output of phant.post() to variable if you need a copy of the string for debugging. i.e. String debugpost = phant.post(); 2) I added phant.queryString() so you can get a copy of the url encoded params. 3) Memory management is outside of the scope of this very simple library, and it's left up to the user to handle that. That may change in the future, but for now I think that's best.

[B504now]

That sounds like a useful change.

The memory issue made the arduino client unusable. So I have rewritten the code to eliminate the phant calls and it's now much more usable. But thanks for tracking it down

Bill Dickenson

302 521 3565

On Tuesday, September 2, 2014 3:23 PM, Todd Treece notifications@github.com wrote:

@B504now 1) You can assign the output of phant.post() to variable if you need a copy of the string for debugging. i.e. String debugpost = phant.post(); 2) I added phant.queryString() so you can get a copy of the url encoded params. 3) Memory management is outside of the scope of this very simple library, and it's left up to the user to handle that. That may change in the future, but for now I think that's best. — Reply to this email directly or view it on GitHub.