Weather site not updating

[gmyenni]

http://157.230.136.69/weather-data.html seems to have stopped updating at 2019-02-12 19:00:00.

There is new data at http://166.153.133.121.

[ethanwhite]

I'm looking into this this morning, but don't see anything promising yet.

@gmyenni - can you email the weather station company and ask if their modem setup would firewall IPs that access the weather station repeatedly?

[ethanwhite]

Also, since we're checking daily now, is it OK for me to reduce the number of records we're grabbing for both tables? Currently it's 1000 for weather and 2500 for storms.

[ethanwhite]

Looks like the new server is timing out on anything (even just requesting 10 records per table). I did some testing on my local machine this and things were working OK for a while but now looks like I'm having the same problems as the server (even in my browser... though apparently this is idiosyncratic because sometimes I'm getting things to work in both). The most likely culprit at this point is the modem blocking IPs that regularly access the data.

So, let's contact the weather station maker and go from there. @gmyenni - Cc on those emails and I'll keep an eye on them in case I see something I might usefully contribute to the conversation.

[ethanwhite]

I've setup a new droplet based on our conversation during the staff meeting, but it looks like the weather station has been down for the last few hours so I haven't been able to test it yet. If anyone notices that it's back up let me know and I'll say if the new IP address takes care of things.

[gmyenni]

I talked to someone at Campbell Sci and he said, no, the modem shouldn't be blocking any IP addresses, unless we've configured an allowable IP range ourselves (which we haven't, obviously, unless the rodents did it themselves on Feb 12).

[gmyenni]

He thought it was more likely that the modem is just never stable long enough to access the dynamic tables, since that takes so long.

[ethanwhite]

Interesting. So the idea is that it was more stable, so things were working on Travis. It got less stable, which stopped Travis's from working consistently, so we set up our digital ocean system. This worked pretty well for a while but then things got less stable, and now it's so bad that it hasn't succeed in 30 attempts in a row. I guess the next question is is this the modem decaying our our cell coverage.

Regardless, it seems like the right strategy from the cyberinfrastructure end is to increase the number of checks and reduce the number of records that we get. By increasing the number of checks we increase the chance of hitting the modem at a good time, by decreasing the number of records we decrease how good we need "good time" to be. Does that sound right to you?

[ethanwhite]

OK, can confirm. I just got a run to work on the original droplet.

[gmyenni]

OK, can confirm. I just got a run to work on the original droplet.

Oh wow, nice.

Yeah, let's just try more often. And fewer records should be fine, once we're back up to date.

[ethanwhite]

... OK, there's a bit of a complexity here with how often we run the cron job for PortalData, because we need the new data table to go back that far, so if we're only running this repo once a week, but we need to go back at least 7 days (7 * 24 = 168 records in the weather table).

What is your thinking on how many we need in the storms table to safely cover a week?

[gmyenni]

It's highly variable. Usually there will be zero new records. But the most we've ever gotten is 1622 new records in a week.

[ethanwhite]

Hmmm. I wonder if we need to incrementally build up the tables. That's a pretty heavy revision, but it's probably the right solution.

So, basically download in smallish chunks but put the chunks together (removing duplicates) to produce a big table that's stored for PortalData to access.

What do you think?

[ethanwhite]

OK, I opened a PR with the simpler solution #312. See what you think. Maybe we can run this for the next few days and see how it's doing (it should update every 6 hours). If it's working well we can go with it. If not we can undertake a more substantive rewrite, where we download less data and incrementally expand the table.

[ethanwhite]

I've been periodically checking the weather station and it's pretty clear that we can rarely connect to it at all at this point. I suspect we are either seeing modem that is failing or a change in connectivity at the site.

[gmyenni]

I talked to a tech at Verizon and he was surprised we ever had any service. Though he didn't really have any explanation for why it would have gotten worse. Someone from Campbell Sci recommended we install a directional antenna, pointed at a tower. So I've ordered that. The modem can have two antennas, so we can leave the omnidirectional one up as well.

[ethanwhite]

Looks like our new strategy may be paying off: http://157.230.136.69/weather-data.html http://157.230.136.69/storms-data.html

[gmyenni]

👍 👍

[ethanwhite]

Current status: data not updated since March 14th.

[gmyenni]

I guess it decided to do phenocam this week. We have images since the 12th. 🤷‍♀️

[ethanwhite]

:laughing: :sob:

[gmyenni]

Update: Apparently the datalogger OS had 'connectivity issues.' Adding a unidirectional antenna improved signal strength. Updating to the latest OS (https://www.campbellsci.com/downloads/cr1000-os) seems to have improved reliability. ??