search

weecology / PortalData

Official Repo of the Portal Project Data
Other
46 stars 30 forks source link

Issues with regional weather station data #278

Closed ethanwhite closed 5 years ago

ethanwhite commented 5 years ago

The issues with the local weather station data not updating were fixed in 240038e614ce3bdfa53afa3d94e3d98a9eead7e9. However the resulting deploy commit has issues with the regional weather station data:

https://github.com/weecology/PortalData/commit/57f0cf9f3b4af2ecc38f34ffdccaad7078f75868

The following lines were added to Portal4sw_regional_weather.csv:

"          <",,,2,"ong ",87726,"7","2","5",
"          <",,,3,"ong ",53834,"1","5","2",
"          <",,,6,"ong ",66123,"2","7","2",
"    <td ali",,,8,"nter",-3,"7","a","6",
"    <td ali",,,9,"nter",66267,"\"",">","<",

And the following lines were added to Sansimon_regional_weather.csv:

"          <",,,2,"ong ",87726,"7","2","5",
"          <",,,3,"ong ",53834,"1","5","2",
"          <",,,6,"ong ",66123,"2","7","2",
"    <td ali",,,8,"nter",-3,"7","a","6",
"    <td ali",,,9,"nter",66267,"\"",">","<",

This seems unrelated to the changes in 240038e614ce3bdfa53afa3d94e3d98a9eead7e9, which should only have touched the local weather station data.

ethanwhite commented 5 years ago

Also, it seems like this should have been caught by a test, so we may need a bit more testing on the regional weather station data.

ethanwhite commented 5 years ago

I removed this bad data in 560414b7b5036c8913175f88c7a4b4f6f49b1115. Now we'll see if it comes back...

ethanwhite commented 5 years ago

Same bad data came back, so this needs exploring.

skmorgane commented 5 years ago

This seems to be a government shutdown issue. The website address we use to get poart4sw now redirects to a page that says this: The website you are trying to access is not available at this time due to a lapse in appropriation.

NOAA.gov and specific NOAA websites necessary to protect lives and property are operational and will be maintained during this partial closure of the U.S. Government.

See weather.gov for forecasts and critical weather information.

ethanwhite commented 5 years ago

Good find @skmorgane. We should add a check on the validity of the data before integrating it. For these datasets I don't think we want to stop the build if there is bad data, just skip it and keep going with other tasks.