search

derekantrican / MountainProject

A scraper and reddit bot for the website MountainProject.com
57 stars 5 forks source link

[DBBuilder] Only update changed items #28

Closed derekantrican closed 4 years ago

derekantrican commented 5 years ago

Every time we run the DBBuilder, we run through every route and every area on MountainProject. This is bad and might be putting some heavy load on MountainProject. Alternatively, we can just update the recently changed items. MountainProject actually has an RSS feed of recently changed items: https://www.mountainproject.com/rss/new?selectedIds=105708956&routes=on&areas=on

derekantrican commented 5 years ago

This should be the default for the DBBuilder, but there should also be a command line parameter option for "all" (as in - rebuild the entire database). It should also default to this if it doesn't see a "previously-made" xml