The current crawler looks like a quick patch in our attempt to shift from BoltDB to Postgresql. The logic is crappy as we tend to fetch PeerInfo one by one from the SQL database (which is awful and a waste of CPU cycles).
This PR aims to solve those problems by refactoring the crawler to use better the power of SQL (as we want to keep it because it is pretty nice to post-process the data).
Tasks
[x] Divide the different aspects of the SQL database into different modules/tables
[ x] PeerInfo
[ x] IpInfo
[ x] ConnectionEvents
[ x] Network-related data (Ethereum, IPFS, etc)
[x] Apply new logic to the crawler
[ ] Optimise the logic of the crawler to get the advantage of the SQL queries
[ ] peering Strategy
[ ] Make new Prometheus metrics exporters based on the SQL database
[ ] Add unitary tests in the individual modules to make maintenance faster
Motivation
The current crawler looks like a quick patch in our attempt to shift from BoltDB to Postgresql. The logic is crappy as we tend to fetch PeerInfo one by one from the SQL database (which is awful and a waste of CPU cycles). This PR aims to solve those problems by refactoring the crawler to use better the power of SQL (as we want to keep it because it is pretty nice to post-process the data).
Tasks