WNV updates are failing

[tomschenkjr]

The updates failed on 2017-07-08 and on 2017-07-10. There doesn't seem to be an attempt to run the predictions on 2017-07-09 (a Sunday).

The 2017-07-08 provided this error

FAIL - R/10_calculate_idtable.R
FAIL - R/21_create_features.R
FAIL - R/40_upload_predictions_ROracle.R

And for 2017-07-10:

SUCCESS - R/10_calculate_idtable.R
FAIL - R/21_create_features.R
FAIL - R/40_upload_predictions_ROracle.R

[geneorama]

The main bug right now is that libcurl can't access ftp://ftp.ncdc.noaa.gov/pub/data/noaa/isd-lite/2017/725300-94846-2017.gz. This error occurs in download_noaa_hourly which was introduced in the NOAA download issue discussed in #34.

I think that the problem is that the FTP port is blocked, although I'm not sure how to test this. Based on what I've read FTP is mainly port 21. I tried manually transferring the files, but ftp isn't installed on the analytics servers.

[tomschenkjr]

Use telnet over port 21 from the command line in the linux server. If it’s a firewall issue, then you’ll get a network time out.

[geneorama]

I tried telnet

$ telnet ftp://ftp.ncdc.noaa.gov/ 21
telnet: ftp://ftp.ncdc.noaa.gov/: Name or service not known
ftp://ftp.ncdc.noaa.gov/: Unknown host

[tomschenkjr]

Try without the “ftp://” prefix.

[geneorama]

Tried that too

$ telnet ftp.ncdc.noaa.gov/ 21
telnet: ftp.ncdc.noaa.gov/: Name or service not known
ftp.ncdc.noaa.gov/: Unknown host

[geneorama]

After some extensive troubleshooting, we discovered that it is a firewall issue. It was confusing that the telnet ftp.ncdc.noaa.gov 21 was successful, but wget and curl failed. As I now understand it, we have different virtual firewalls, and the telnet command was probably being handled differently (not blocked or blocked at a different level).

I will put in a firewall request into the tufin system.

[geneorama]

It took longer to upload the predictions manually.

Some lessons learned:

• RODBC is no longer a good way to update the predictions, there have been some significant departures from the original "proof of concept" uploads. One big change is that the time zone information is stored as an environmental variable that the Oracle drivers use.
• The database info in the "untracked" folder contains four lines now, the server name, the service name, the username, and the password. Previously the database info files didn't contain the first line. The RODBC connection didn't use the server name, so my local database connection info files were out of sync with the ones I needed on the analytics servers.

I ran the data acquisition files locally, uploaded the two .Rds files that are produced, then ran the model and database update code on the analytics servers.

[tomschenkjr]

RODBC or ROracle? I thought you were using the latter.

[geneorama]

I'm using ROracle on the linux servers.

RODBC is no longer a good way to update the predictions

Also, it was good to have a dev database to test the update first. When I tried updating the database with RODBC it would drop the tables, but then wouldn't write new ones. So, I would have messed up the prod server (temporarily) if I wouldn't have had access to the dev database.

[geneorama]

I put in the tufin request earlier this afternoon.

[geneorama]

Firewall has been opened.