search

NASA-Planetary-Science / HiMAT

23 stars 9 forks source link

acquiring data from Weather Underground API #25

Open aaarendt opened 6 years ago

aaarendt commented 6 years ago
lsetiawan commented 6 years ago

I have signed up for the Anvil Plan which includes:

I will explore what areas are available and see if there are any weather station around our region from them.

lsetiawan commented 6 years ago

This look like a promising source for data. There are daily summaries, not completely visible in API documentation, but I have found a way to see all the stations and then grab data from each station. Some are pretty new, so might not have much data.

The only thing I am concerned is the vocabulary personal weather station. From the wunderground site, It seems like the actual equipment has to be within the list. So I think that the data are pretty reliable. There is also a notion of goldstar station which is basically a very reliable station, here's a sample of one of those station (IKATHMAN7).

WUNDERGROUND STATIONS MAP

IKATHMAN7 in Nepal:

ikathman7

lsetiawan commented 6 years ago

It think for these API, since I have a limit on how much request I can make in a day, we should put the data in a database for the stations that we want.

YiwenMei commented 6 years ago

Dear Himat Teams,

Here is a Matlab program I developed to fetch data from Weather Underground (WU). The program used the "geolookup" and "history" feature available from the WU API. So, to get the codes run you need to register for a WU API.

fetch_WU.zip

If you find any bug, let me know. I will fix it.

Thank you Yiwen

YiwenMei commented 6 years ago

Hi Himat Teams,

Here is a simple skeleton of the codes and some sample airport data downloaded from WU.

Thank you Yiwen

Introduction of the codes: Yiwen_Mei_110217.pptx

Sample data: airportVABB_20071001_20080930.zip

YiwenMei commented 6 years ago

Hi Himat Teams,

Please see the attached files for the ground-based observations for the HMA area from Weather Underground. A QC process was performed on the original records by myself and I briefly described the process in the Readme file. Both the original records (WU_Raw.zip) and the QCed data (WU_Processed.zip) were attached. A shapefiles showing the station location was also provided.

WU_Raw.zip WU_Processed.zip

Good luck Yiwen

aaarendt commented 6 years ago

@YiwenMei thanks for sharing these datasets. How easy is it to build a map of the stations? Then I could use that to advertise to the team what is available.

YiwenMei commented 6 years ago

Hi Himat Teams,

Please see the attached for the locations of the station from Weather Underground (with length of record labeled).

hrz

Yiwen @aaarendt

MelissaLinares commented 6 years ago

@YiwenMei thanks for sharing. I was wondering if you saw some inconsistencies in the WU data that made you perform your own QC because from what I have read on their website they perform QC on all stations and remove station data that has values outside of normal range. (eg, ytemperature is > 150 F or < -150 F), data values changing at too slow or too fast a rate, weather data values do not agree with stations nearby. I am thinking about using their data so any insight will greatly be appreciated. Thanks

YiwenMei commented 6 years ago

@MelissaLinares : sorry for the late reply. Yes, I saw inconsistency in the data downloaded from WU. That is the reason I put quality control on the data. Take for example this record "ObsSum_airportOAKB_20080930_20090930.mat" from "WU_Raw.zip". If you load the .mat file you will see a matlab variable called "ObsSum" listing all the records. Let's plot column 2, 3 or 9 (air temperature, dew point temperature or air pressure), you will see some extreme values; specially for the air pressure. The QC code is basically aiming to remove those outliers and with an option to perform linear interpolation to fill the missing values (can be turn off).