Karlheinzniebuhr / the-weather-scraper

A Lightweight Weather Scraper
105 stars 35 forks source link

Code only extracts header but doesn't extract data #11

Open map06 opened 8 months ago

map06 commented 8 months ago

I need to extract data for weather stations across Asia. I tried installing and running it as said in the README file. However, for some reason, the code is not extracting the data but is only extracting the headers. I don't get any error.

Karlheinzniebuhr commented 8 months ago

What are the links you added to the file? Did you make sure there is data available on those stations in the date range you provided ?

map06 commented 8 months ago

For example one of the link is https://www.wunderground.com/dashboard/pws/IGURUG1

This is my settings,

from datetime import date

# Set Date format like: YYYY, MM, DD
# Note that FIND_FIRST_DATE uses START_DATE as default start date
START_DATE = date(2018, 1, 1)
END_DATE = date(2020, 7, 19)

# set to "metric" or "imperial"
UNIT_SYSTEM = "metric"
# UNIT_SYSTEM = "imperial"

# Automatically find first date where data is logged
FIND_FIRST_DATE = True

When I run: python3 weather_scraper.py: It prints:

** Initializing binary search to find the first date with data **
low is 0 - 2018-01-01
high is 2197 - 2024-01-07
mid is 1098 - 2021-01-03
----//----
low is 1099 - 2021-01-04
high is 2197 - 2024-01-07
mid is 1648 - 2022-07-07
----//----
low is 1649 - 2022-07-08
high is 2197 - 2024-01-07
mid is 1923 - 2023-04-08
----//----
low is 1924 - 2023-04-09
high is 2197 - 2024-01-07
mid is 2060 - 2023-08-23
----//----
low is 1924 - 2023-04-09
high is 2059 - 2023-08-22
mid is 1991 - 2023-06-15
----//----
low is 1992 - 2023-06-16
high is 2059 - 2023-08-22
mid is 2025 - 2023-07-19
----//----
low is 2026 - 2023-07-20
high is 2059 - 2023-08-22
mid is 2042 - 2023-08-05
----//----
low is 2026 - 2023-07-20
high is 2041 - 2023-08-04
mid is 2033 - 2023-07-27
----//----
low is 2026 - 2023-07-20
high is 2032 - 2023-07-26
mid is 2029 - 2023-07-23
----//----
First date found! 2023-07-23
https://www.wunderground.com/dashboard/pws/IGURUG1/table/2023-07-23/2023-07-23/daily

When I check IGURUG1.csv It only has :

Date,Time,Temperature_C,Dew_Point_C,Humidity_%,Wind,Speed_kmh,Gust_kmh,Pressure_hPa,Precip_Rate_mm,Precip_Accum_mm,UV,Solar_w/m2
Date,Time,Temperature_C,Dew_Point_C,Humidity_%,Wind,Speed_kmh,Gust_kmh,Pressure_hPa,Precip_Rate_mm,Precip_Accum_mm,UV,Solar_w/m2
Date,Time,Temperature_C,Dew_Point_C,Humidity_%,Wind,Speed_kmh,Gust_kmh,Pressure_hPa,Precip_Rate_mm,Precip_Accum_mm,UV,Solar_w/m2
Date,Time,Temperature_C,Dew_Point_C,Humidity_%,Wind,Speed_kmh,Gust_kmh,Pressure_hPa,Precip_Rate_mm,Precip_Accum_mm,UV,Solar_w/m2
Date,Time,Temperature_C,Dew_Point_C,Humidity_%,Wind,Speed_kmh,Gust_kmh,Pressure_hPa,Precip_Rate_mm,Precip_Accum_mm,UV,Solar_w/m2

I checked the website. It does have data for specified date.