Open Arvedek opened 1 year ago
Hi @Arvedek - For this data comparison I used two different pipelines. 1) URA endpoint. https://www.ura.gov.sg/maps/api/ > This allows us to extract the data historically for the rentals. It refreshes the data on the 15th of each month. 2) For property guru data I just built a scraper using Playwright. Given the fact that distributing scraping tools may be against regulations I would prefer not to. But if you want to build your own I can share some guidance.
Thanks for the reply. I just got the data from URA api as well(looks like they only provide the recent 3 years data?)
Scraper wise, would be great to get some help from you to understand how to build/use
May I inquire as to what the purpose of the use of the data is? Is it for academic purposes? Or commercial ones? Do you need to extract the data using a remote machine (like a cloud VM) or just in your local desktop / laptop?
Educational use only. Just trying to understand the real estate market derive data based insight. All the jobs will be done in personal desktop only
So I was testing the package I wrote to crawl the site... i am afraid they changed something on their side and now they have a bot protecting service so It blocks the requests after a few pages are downloaded. If I find hot to solve it Ill let you know.
I'm interested in data extraction, may I ask for more details?