DedSecInside / gotor

This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.
GNU General Public License v3.0
159 stars 44 forks source link

Add cache to avoid excessive lookups #23

Closed KingAkeem closed 2 years ago

KingAkeem commented 3 years ago

There are two methods of interacting with a node.

When a caller asks to load a node into memory, it is currently constructed in real-time. If the caller just loaded the node previously then it isn't likely to have changed within a short amount of time. In order to avoid re-constructing the entire node tree everytime it's called, the nodes can be stored in a database and the lastloaded time can be used to check if a certain time constraint has been met before re-building the tree. If the time period has not been met then return the cached tree.

Default value: 2 minutes Database: MongoDB