Invalid Response from WUnderground server!

[namsral]

Cannot connect to the WUnderground API, clicking "WUnderground Diagnostics" from the Advanced menu produces the following error:

Invalid Response from WUnderground server!

This is because the WUnderground API servers do not respond to HTTP requests without a Host header.

Possible solution is to remove the WUnderground IP option from the Settings page and have Sprinklers Pi connect to api.wunderground.com instead.

[rszimm]

Hmmm, It seems to be working for me and for other people who are using the API as is. I'll look into this...

[rszimm]

OK, try this IP: 38.102.136.138. That one works for me. That's what api.wunderground.com "used to" resolve to. That server still seems to operate without the host header. Apparently wunderground (which very recently did a complete rebuild on their platforms) rehosted the api.wunderground.com and now it points to 23.11.88.120. The new one requires the host key.

So, short term use the old IP and it should work fine. Long term I'm going to fix this so I resolve the domain name and modify the code to insert the host header in the request.

[namsral]

I wanted to confirm the IP 38.102.136.138 works.

Resolving the host api.wunderground.com returns an alias of the Akamai CDN which can change over time so having Sprinkler resolving the domain would fix the issue, thanks.

[maknij]

This solution partially works for me, I'm still having intermittent 'Invalid Response from WUnderground server!' messages when using IP 38.102.136.138.

[Agh42]

This feature is SO awesome - unfortunately I'm having the same problem.

I'm trying to resolve a personal weather station. These two queries work using wget but not with the settings in sprinklers_pi:

http://api.underground.com/api//conditions/q/pws:IBERLINB44.json http://38.102.136.138/api//conditions/q/pws:IBERLINB44.json

The IP that "api.wunderground.com" currently resolves to is "23.209.152.120" and this only works when using the hostname.

[Agh42]

I removed my API key in the examples above. It needs to be inserted between the //

[maknij]

Some feedback - I compiled the changes proposed by nhorvath, and using an IP resolved from api.wunderground.com still fails intermittently

[nhorvath]

I'm going to guess that it probably has something to do with akamai then. As a high availability CDN hard coded IPs really shouldn't be used against their servers. It's much more complicated change to do the DNS resolution. Maybe if I have some time I'll look into it.

On Thu, May 29, 2014 at 1:53 AM, maknij notifications@github.com wrote:

Some feedback - I compiled the changes proposed by nhorvath, and using an IP resolved from api.wunderground.com still fails intermittently

Reply to this email directly or view it on GitHubhttps://github.com/rszimm/sprinklers_pi/issues/18#issuecomment-44496948 .

[maknij]

Great, thanks :)

[ardupi]

How about trying to fetch data several times instead of falling back to default times after the first invalid response form wunderground? Say, 5 retries, each separated by 20 secs. This should get rid of the intermittent invalid response problem.

[johnwick]

This works from a browser using:

http://38.102.136.138/api/keygoeshere/yesterday/conditions/q/pws:KCAOJAI22.json

Also can wget http://38.102.136.138/api/keygoeshere/yesterday/conditions/q/pws:KCAOJAI22.json

But no go from the sprinkler_pi interface:

2015/05/13 08:29:11 Page:json/wcheck 2015/05/13 08:29:12 Connected 2015/05/13 08:29:13 Bad WUnderground Response

[nhorvath]

do any other PWS urls work for you? If you find one that does i suggest diffing the response and looking for missing values that might be important to the rainfall/humidity/temperature calculation.

On Wed, May 13, 2015 at 11:31 AM, johnwick notifications@github.com wrote:

This works from a browser using:

http://38.102.136.138/api/keygoeshere/yesterday/conditions/q/pws:KCAOJAI22.json

Also can wget http://38.102.136.138/api/keygoeshere/yesterday/conditions/q/pws:KCAOJAI22.json

But no go from the sprinkler_pi interface:

2015/05/13 08:29:11 Page:json/wcheck 2015/05/13 08:29:12 Connected 2015/05/13 08:29:13 Bad WUnderground Response

— Reply to this email directly or view it on GitHub https://github.com/rszimm/sprinklers_pi/issues/18#issuecomment-101715942 .

[johnwick]

It fails if I go by zip code in settings as well. The interesting thing is that it worked for 3 weeks using the pws, just stopped yesterday.

Sent from my iPhone

On May 13, 2015, at 8:40 AM, nhorvath notifications@github.com wrote:

do any other PWS urls work for you? If you find one that does i suggest diffing the response and looking for missing values that might be important to the rainfall/humidity/temperature calculation.

On Wed, May 13, 2015 at 11:31 AM, johnwick notifications@github.com wrote:

This works from a browser using:

http://38.102.136.138/api/keygoeshere/yesterday/conditions/q/pws:KCAOJAI22.json

Also can wget http://38.102.136.138/api/keygoeshere/yesterday/conditions/q/pws:KCAOJAI22.json

But no go from the sprinkler_pi interface:

2015/05/13 08:29:11 Page:json/wcheck 2015/05/13 08:29:12 Connected 2015/05/13 08:29:13 Bad WUnderground Response

— Reply to this email directly or view it on GitHub https://github.com/rszimm/sprinklers_pi/issues/18#issuecomment-101715942 .

— Reply to this email directly or view it on GitHub.

[johnwick]

Is there a temp file the sprinkler_pi creates that I can look at?

Sent from my iPhone

On May 13, 2015, at 8:40 AM, nhorvath notifications@github.com wrote:

do any other PWS urls work for you? If you find one that does i suggest diffing the response and looking for missing values that might be important to the rainfall/humidity/temperature calculation.

On Wed, May 13, 2015 at 11:31 AM, johnwick notifications@github.com wrote:

This works from a browser using:

http://38.102.136.138/api/keygoeshere/yesterday/conditions/q/pws:KCAOJAI22.json

Also can wget http://38.102.136.138/api/keygoeshere/yesterday/conditions/q/pws:KCAOJAI22.json

But no go from the sprinkler_pi interface:

2015/05/13 08:29:11 Page:json/wcheck 2015/05/13 08:29:12 Connected 2015/05/13 08:29:13 Bad WUnderground Response

— Reply to this email directly or view it on GitHub https://github.com/rszimm/sprinklers_pi/issues/18#issuecomment-101715942 .

— Reply to this email directly or view it on GitHub.

[nhorvath]

you would have to enable the debugging that will output the response to the logs.

in Weather.cpp you'll see this section, you need to uncomment the trace(getstring) line:

if (client.connect(ip, 80)) { char getstring[90]; trace(F("Connected\n")); if (usePws) snprintf(getstring, sizeof(getstring), "GET /api/%s/yesterday/conditions/q/pws:%s.json HTTP/1.1\r\n", key, pws); else snprintf(getstring, sizeof(getstring), "GET /api/%s/yesterday/conditions/q/%ld.json HTTP/1.1\r\n", key, (long) zip); //trace(getstring); client.write((uint8t) getstring, strlen(getstring)); //send host header snprintf(getstring, sizeof(getstring), "Host: api.wunderground.com\r\nConnection: close\r\n\r\n"); client.write((uint8t) getstring, strlen(getstring));

            ParseResponse(client, &vals);
            client.stop();
            if (!vals.valid)
            {
                    if (vals.keynotfound)
                            trace("Invalid WUnderground Key\n");
                    else
                            trace("Bad WUnderground Response\n");
            }
    }

While you are in there you should also uncomment this line in the first function on that file ParseResponse //trace("%s:%s\n", key, val);

after you save your changes you need to compile again: make (if it doesn't build anything do "make clean" then make) install your changes with: sudo make install restart sprinklers_pi: sudo /etc/init.d/sprinklers_pi restart

if you don't see those key:value lines printed in your logs after that it means there is something wrong with the whole json response causing it not to be parsed correctly. if you see them printing it means there is a value missing but i would think you would get the invalid wunderground key message if that was happening.

On Wed, May 13, 2015 at 12:17 PM, johnwick notifications@github.com wrote:

Is there a temp file the sprinkler_pi creates that I can look at?

Sent from my iPhone

On May 13, 2015, at 8:40 AM, nhorvath notifications@github.com wrote:

do any other PWS urls work for you? If you find one that does i suggest diffing the response and looking for missing values that might be important to the rainfall/humidity/temperature calculation.

On Wed, May 13, 2015 at 11:31 AM, johnwick notifications@github.com wrote:

This works from a browser using:

http://38.102.136.138/api/keygoeshere/yesterday/conditions/q/pws:KCAOJAI22.json

Also can wget

http://38.102.136.138/api/keygoeshere/yesterday/conditions/q/pws:KCAOJAI22.json

But no go from the sprinkler_pi interface:

2015/05/13 08:29:11 Page:json/wcheck 2015/05/13 08:29:12 Connected 2015/05/13 08:29:13 Bad WUnderground Response

— Reply to this email directly or view it on GitHub < https://github.com/rszimm/sprinklers_pi/issues/18#issuecomment-101715942>

.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/rszimm/sprinklers_pi/issues/18#issuecomment-101732439 .

[johnwick]

Thank you for all your help with this.  Here is what I found...

Made the changes and ran the diagnostics and got the full result in the logs and a successful calculation of the adjustment when setup to run by zip code, but it does not work with the pws still which would be useful since our school is in a microclimate that is dryer than the results from the zip (though I suppose most times it is not a huge difference)

Odd part is that the zip functionality was not working earlier today either.

I did try two other pws entries and they both failed with bad wunderground response

2015/05/13 14:34:03 Got a client 2015/05/13 14:34:03 Page:json/wcheck 2015/05/13 14:34:03 Connected 2015/05/13 14:34:03 GET /api/effa2c3a4b217c15/yesterday/conditions/q/pws:KCAOJAI22.json HTTP/1.0

2015/05/13 14:34:04 Bad WUnderground Response

Then for grins I tried changing back to the pas restarting sprinkler_pi and trying again.  Now I get a bad wunderground response

2015/05/13  14:38:42 Page:json/wcheck 2015/05/13 14:38:42 Connected 2015/05/13 14:38:42 GET /api/effa2c3a4b217c15/yesterday/conditions/q/pws:KCAOJAI22.json HTTP/1.0

2015/05/13 14:38:43 Bad WUnderground Response

I hope this is helpful in some way.  For now the zip method is working so that will be OK.  I wrote a bash script that pulls the ET data straight from our wview installation and adjusts the watering based on that for the internal program.  It was pretty easy to do because I could just load a url with the run times embedded using a cron job.  

Is there a straight forward way that I could pass the percent that I calculate from my script to the sprinklers_pi, and have it replace either the seasonal adjust or the underground based adjustment?

My coding skills are functional, but very elementary.

On May 13, 2015, at 09:30 AM, nhorvath notifications@github.com wrote:

you would have to enable the debugging that will output the response to the logs.

in Weather.cpp you'll see this section, you need to uncomment the trace(getstring) line:

if (client.connect(ip, 80)) { char getstring[90]; trace(F("Connected\n")); if (usePws) snprintf(getstring, sizeof(getstring), "GET /api/%s/yesterday/conditions/q/pws:%s.json HTTP/1.1\r\n", key, pws); else snprintf(getstring, sizeof(getstring), "GET /api/%s/yesterday/conditions/q/%ld.json HTTP/1.1\r\n", key, (long) zip); //trace(getstring); client.write((uint8t) getstring, strlen(getstring)); //send host header snprintf(getstring, sizeof(getstring), "Host: api.wunderground.com\r\nConnection: close\r\n\r\n"); client.write((uint8t) getstring, strlen(getstring));

ParseResponse(client, &vals); client.stop(); if (!vals.valid) { if (vals.keynotfound) trace("Invalid WUnderground Key\n"); else trace("Bad WUnderground Response\n"); } }

While you are in there you should also uncomment this line in the first function on that file ParseResponse //trace("%s:%s\n", key, val);

after you save your changes you need to compile again: make (if it doesn't build anything do "make clean" then make) install your changes with: sudo make install restart sprinklers_pi: sudo /etc/init.d/sprinklers_pi restart

if you don't see those key:value lines printed in your logs after that it means there is something wrong with the whole json response causing it not to be parsed correctly. if you see them printing it means there is a value missing but i would think you would get the invalid wunderground key message if that was happening.

On Wed, May 13, 2015 at 12:17 PM, johnwick notifications@github.com wrote:

Is there a temp file the sprinkler_pi creates that I can look at?

Sent from my iPhone

On May 13, 2015, at 8:40 AM, nhorvath notifications@github.com wrote:

do any other PWS urls work for you? If you find one that does i suggest diffing the response and looking for missing values that might be important to the rainfall/humidity/temperature calculation.

On Wed, May 13, 2015 at 11:31 AM, johnwick notifications@github.com wrote:

This works from a browser using:

http://38.102.136.138/api/keygoeshere/yesterday/conditions/q/pws:KCAOJAI22.json

Also can wget

http://38.102.136.138/api/keygoeshere/yesterday/conditions/q/pws:KCAOJAI22.json

But no go from the sprinkler_pi interface:

2015/05/13 08:29:11 Page:json/wcheck 2015/05/13 08:29:12 Connected 2015/05/13 08:29:13 Bad WUnderground Response

— Reply to this email directly or view it on GitHub < https://github.com/rszimm/sprinklers_pi/issues/18#issuecomment-101715942>

.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/rszimm/sprinklers_pi/issues/18#issuecomment-101732439 .

— Reply to this email directly or view it on GitHub.

[nhorvath]

if you can get your data point onto a webpage you could replace the code in weather.cpp to get your adjustment and replace the calculation in there to just use the straight result without parsng it. in confguration change the wundergound ip to your server ip. If you do go off changing the code I recommend you forking on github and checking out your own fork... that way you can at least manage you changes in case you break something.

If you forward me the log with the raw data output I'm willing to take a look and see if we can see anything wrong (or perhaps a bug in the parser).

On Wed, May 13, 2015 at 5:45 PM, johnwick notifications@github.com wrote:

Thank you for all your help with this. Here is what I found...

Made the changes and ran the diagnostics and got the full result in the logs and a successful calculation of the adjustment when setup to run by zip code, but it does not work with the pws still which would be useful since our school is in a microclimate that is dryer than the results from the zip (though I suppose most times it is not a huge difference)

Odd part is that the zip functionality was not working earlier today either.

I did try two other pws entries and they both failed with bad wunderground response

2015/05/13 14:34:03 Got a client 2015/05/13 14:34:03 Page:json/wcheck 2015/05/13 14:34:03 Connected 2015/05/13 14:34:03 GET /api/effa2c3a4b217c15/yesterday/conditions/q/pws:KCAOJAI22.json HTTP/1.0

2015/05/13 14:34:04 Bad WUnderground Response

Then for grins I tried changing back to the pas restarting sprinkler_pi and trying again. Now I get a bad wunderground response

2015/05/13 14:38:42 Page:json/wcheck 2015/05/13 14:38:42 Connected 2015/05/13 14:38:42 GET /api/effa2c3a4b217c15/yesterday/conditions/q/pws:KCAOJAI22.json HTTP/1.0

2015/05/13 14:38:43 Bad WUnderground Response

I hope this is helpful in some way. For now the zip method is working so that will be OK. I wrote a bash script that pulls the ET data straight from our wview installation and adjusts the watering based on that for the internal program. It was pretty easy to do because I could just load a url with the run times embedded using a cron job.

Is there a straight forward way that I could pass the percent that I calculate from my script to the sprinklers_pi, and have it replace either the seasonal adjust or the underground based adjustment?

My coding skills are functional, but very elementary.

On May 13, 2015, at 09:30 AM, nhorvath notifications@github.com wrote:

you would have to enable the debugging that will output the response to the logs.

in Weather.cpp you'll see this section, you need to uncomment the trace(getstring) line:

if (client.connect(ip, 80)) { char getstring[90]; trace(F("Connected\n")); if (usePws) snprintf(getstring, sizeof(getstring), "GET /api/%s/yesterday/conditions/q/pws:%s.json HTTP/1.1\r\n", key, pws); else snprintf(getstring, sizeof(getstring), "GET /api/%s/yesterday/conditions/q/%ld.json HTTP/1.1\r\n", key, (long) zip); //trace(getstring); client.write((uint8t) getstring, strlen(getstring)); //send host header snprintf(getstring, sizeof(getstring), "Host: api.wunderground.com\r\nConnection: close\r\n\r\n"); client.write((uint8t) getstring, strlen(getstring));

ParseResponse(client, &vals); client.stop(); if (!vals.valid) { if (vals.keynotfound) trace("Invalid WUnderground Key\n"); else trace("Bad WUnderground Response\n"); } }

While you are in there you should also uncomment this line in the first function on that file ParseResponse //trace("%s:%s\n", key, val);

after you save your changes you need to compile again: make (if it doesn't build anything do "make clean" then make) install your changes with: sudo make install restart sprinklers_pi: sudo /etc/init.d/sprinklers_pi restart

if you don't see those key:value lines printed in your logs after that it means there is something wrong with the whole json response causing it not to be parsed correctly. if you see them printing it means there is a value missing but i would think you would get the invalid wunderground key message if that was happening.

On Wed, May 13, 2015 at 12:17 PM, johnwick notifications@github.com wrote:

Is there a temp file the sprinkler_pi creates that I can look at?

Sent from my iPhone

On May 13, 2015, at 8:40 AM, nhorvath notifications@github.com wrote:

do any other PWS urls work for you? If you find one that does i suggest diffing the response and looking for missing values that might be important to the rainfall/humidity/temperature calculation.

On Wed, May 13, 2015 at 11:31 AM, johnwick notifications@github.com wrote:

This works from a browser using:

http://38.102.136.138/api/keygoeshere/yesterday/conditions/q/pws:KCAOJAI22.json

Also can wget

http://38.102.136.138/api/keygoeshere/yesterday/conditions/q/pws:KCAOJAI22.json

But no go from the sprinkler_pi interface:

2015/05/13 08:29:11 Page:json/wcheck 2015/05/13 08:29:12 Connected 2015/05/13 08:29:13 Bad WUnderground Response

— Reply to this email directly or view it on GitHub < https://github.com/rszimm/sprinklers_pi/issues/18#issuecomment-101715942

.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub < https://github.com/rszimm/sprinklers_pi/issues/18#issuecomment-101732439>

.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/rszimm/sprinklers_pi/issues/18#issuecomment-101825418 .

[DanJoeStone]

I've been having the same issue of late where I've used IP: 38.102.136.138 for years but recently it has failed.

I fixed it by googling a "nslookup" service ( i used http://www.kloth.net/services/nslookup.php) and looked up "api.wunderground.com". The IP I got was 95.101.244.53

I'm not sure often this will change but it is now working.