search

cesarsroy / rent_comparison_singapore

This project compares the official rents published by the URA vs the price listings in property guru
1 stars 0 forks source link

Data extraction #1

Open Arvedek opened 1 year ago

Arvedek commented 1 year ago

I'm interested in data extraction, may I ask for more details?

cesarsroy commented 1 year ago

Hi @Arvedek - For this data comparison I used two different pipelines. 1) URA endpoint. https://www.ura.gov.sg/maps/api/ > This allows us to extract the data historically for the rentals. It refreshes the data on the 15th of each month. 2) For property guru data I just built a scraper using Playwright. Given the fact that distributing scraping tools may be against regulations I would prefer not to. But if you want to build your own I can share some guidance.

Arvedek commented 1 year ago

Thanks for the reply. I just got the data from URA api as well(looks like they only provide the recent 3 years data?)

Scraper wise, would be great to get some help from you to understand how to build/use

cesarsroy commented 1 year ago

May I inquire as to what the purpose of the use of the data is? Is it for academic purposes? Or commercial ones? Do you need to extract the data using a remote machine (like a cloud VM) or just in your local desktop / laptop?

Arvedek commented 1 year ago

Educational use only. Just trying to understand the real estate market derive data based insight. All the jobs will be done in personal desktop only

cesarsroy commented 1 year ago

So I was testing the package I wrote to crawl the site... i am afraid they changed something on their side and now they have a bot protecting service so It blocks the requests after a few pages are downloaded. If I find hot to solve it Ill let you know.