search

adizafri2000 / billscraper

Sample headless automation project to scrape bills, calculate monthly bill (rent + utilities) and generate a text message which is later sent to desired whatsapp recipients.
1 stars 0 forks source link

Save page source (HTML) during automation errors #26

Closed adizafri2000 closed 6 months ago

adizafri2000 commented 6 months ago

In the case of elements not found etc., webdriver will save the page source via driver.page_source and upload to a bucket on Supabase. This is just to log current HTML state of the visited webpages on error encounters

adizafri2000 commented 6 months ago

Folder created on Supabase as html/, which contains subfolders tnb/ and air/. Created function to generate the page source and write to a .html file, which will later fall under e.g. html/tnb/file_name.html.

Automation script modified for both tnb and air to encapsulate ALL of its steps within try/except block. On WebDriverExceptions, program will generate screenshot at point of error, get page HTML source, save those files to Supabase, and terminate the program.