search

rezamike / home_finder

0 stars 0 forks source link

Web scraping work-around #2

Open rezamike opened 2 months ago

rezamike commented 2 months ago

I have axios and cheerio pulling data from requested url - works for other sites, but it's not working for our target site because of CORS issues. We can use Node on the server to avoid such issues during web scraping, but I have no idea how to do it, set up or implement.

rezamike commented 2 months ago

looks like we can use axios with county provided api key to pull full html, then parse for what we want . running into an issue of making back to back axios calls through node (solveable, but i don't know how). we'll need to make an initial CALL for a search page based on params, then scrape search results for the dynamic param to specific result, then make another CALL to that result page to scrape the pdf file. it's definitely doable, but i have forgotten a lot about this stuf....