Martin-Kennedy
commented
2 years ago @ryancoughlin Hi Ryan, has been a REALLY, REALLY long time since you commented on this. I actually use another git repo and hadn't checked this one in a while so my apologies. I actually revisited this and am essentially manually getting the spots I need via the countries listed on their site map at https://magicseaweed.com/site-map.php. The geocoding is not needed as they manually dump json onto the site via whatever view system they are using to serve each page then they have a separate script that manually populates the data onto the markup loaded into the page. I am assuming this is because the site is rather old and this was an easier solution than a complete rebuild? In the older version I was using the google maps geocoding api, the problem I would run into is rate limiting on magic seaweed and api costs for the 6000+ spots they have listed. Either way the script I have written and updated pulls from this json object using cheerio and then just does a simple fs write or append to a json file in whatever directory you want it to. Not sure if you are still interested and you have probably found a solution to this by now but if not I just threw it up on the repo I have been using recently. https://github.com/Martin-Kennedy/MS-Scraper Cheers!
Would love to chat about how you're scraping this. I was going to do something similar and store: name, lon, lat (would need to reverse geocode location for this) and id in a mongo db and geo query against that to get the station id then hit MS API.