search

g0v / vaccinate

vaxx.tw is a tool to find COVID-19 Vaccine Appointments near you
http://vaxx.tw
MIT License
33 stars 20 forks source link

Asyncify our methods to improve performance #36

Closed kevinjcliao closed 3 years ago

kevinjcliao commented 3 years ago

It's taking on average 40-50 seconds to scrape each time. This is too slow because the appointments are going very quickly now. We will save significant time by adopting async. Now down to 10-12 seconds.

Test plan: yarn scrape confirm still works.

mohw_nantou is broken and I need to file a fix request.

Test Plan:

(vaccine2)  ~/Documents/Programming/g0v/vaccine2/backend  master●  0m  python local_scraper.py                                                                                                            
[(3, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>}), (11, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>}), (19, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>}), (8, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (27, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (21, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (17, {'self_paid': <AppointmentAvailability.AVAILABLE: 'Available'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (10, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>}), (1, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.AVAILABLE: 'Available'>}), (13, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>}), (14, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.AVAILABLE: 'Available'>}), (28, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.AVAILABLE: 'Available'>}), (29, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.AVAILABLE: 'Available'>}), (12, {'self_paid': <AppointmentAvailability.AVAILABLE: 'Available'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (22, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>}), (2, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (4, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (5, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (6, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (7, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (9, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (15, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (16, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (18, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (20, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (23, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (24, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (25, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (26, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (30, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (31, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>})]
----Time: 50.78437399864197----

(vaccine2)  ~/Documents/Programming/g0v/vaccine2/backend  async  0m  python local_scraper.py                                                                                                              
----scrape_tonyen_hsinchu: 0.1959974765777588-----
----parse_tzuchi_hualien: 0.22075915336608887-----
----parse_ntu_yunlin: 0.4962174892425537-----
----parse_ntu_hsinchu: 0.8664581775665283-----
----parse_ntu_taipei: 1.107123613357544-----
----scrape_pch_nantou: 1.1037871837615967-----
-----parse_mohw_nantou: Unexpected error: <class 'KeyError'>
----scrape_changgung_chiayi: 1.19110107421875-----
----parse_tzuchi_taipei: 1.9510462284088135-----
----parse_mohw_keelung: 1.9770476818084717-----
----scrape_siaogang_kaohsiung: 2.4221842288970947-----
----parse_mohw_taitung: 2.4698691368103027-----
----parse_mohw_miaoli: 2.8588149547576904-----
----parse_mohw_taoyuan: 3.3229405879974365-----
----parse_mohw_taichung: 3.3663933277130127-----
----parse_ncku_tainan: 5.681906461715698-----
----parse_mohw_kinmen: 8.428256511688232-----
[(3, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>}), (11, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>}), (19, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>}), (8, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (27, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (21, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (17, {'self_paid': <AppointmentAvailability.AVAILABLE: 'Available'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (10, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>}), (1, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.AVAILABLE: 'Available'>}), (13, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>}), (14, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.AVAILABLE: 'Available'>}), (28, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.AVAILABLE: 'Available'>}), (29, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.AVAILABLE: 'Available'>}), (12, {'self_paid': <AppointmentAvailability.AVAILABLE: 'Available'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (23, {'self_paid': <AppointmentAvailability.AVAILABLE: 'Available'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (22, {'self_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>, 'government_paid': <AppointmentAvailability.UNAVAILABLE: 'Unavailable'>}), (2, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (4, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (5, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (6, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (7, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (9, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (15, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (16, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (18, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (20, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (24, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (25, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (26, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (30, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>}), (31, {'self_paid': <AppointmentAvailability.NO_DATA: 'No data'>, 'government_paid': <AppointmentAvailability.NO_DATA: 'No data'>})]
--- 11.3687424659729 seconds ---