search

nucypher / nucypher-monitor

NuCypher network intelligence crawler and web dashboard
7 stars 15 forks source link

Improve monitor efficiency by caching staker activity partition results for 2 hours at a time #101

Closed derekpierre closed 3 years ago

derekpierre commented 3 years ago

Related to #99 , #100 .

codecov-commenter commented 3 years ago

Codecov Report

Merging #101 (9874b45) into main (a6067ee) will decrease coverage by 0.14%. The diff coverage is 26.31%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #101      +/-   ##
==========================================
- Coverage   50.46%   50.32%   -0.15%     
==========================================
  Files          11       11              
  Lines        1072     1075       +3     
==========================================
  Hits          541      541              
- Misses        531      534       +3
Impacted Files Coverage Δ
monitor/crawler.py 42.38% <26.31%> (-0.33%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update a6067ee...9874b45. Read the comment docs.

derekpierre commented 3 years ago

Perhaps for the moment, caching may be overkill. The scraping rounds on the status monitor that is running code from #100 (i.e. before this change) are a little over 1 minute, which is good enough. Having our own web3 node really helps:

crawler_1   | Scraping round completed (duration 0:01:57).
crawler_1   | Scraping round completed (duration 0:01:08).
crawler_1   | Scraping round completed (duration 0:01:22).
crawler_1   | Scraping round completed (duration 0:01:22).
crawler_1   | Scraping round completed (duration 0:01:02).
crawler_1   | Scraping round completed (duration 0:01:24).
crawler_1   | Scraping round completed (duration 0:01:13).
crawler_1   | Scraping round completed (duration 0:01:22).
crawler_1   | Scraping round completed (duration 0:01:25).
crawler_1   | Scraping round completed (duration 0:01:21).
crawler_1   | Scraping round completed (duration 0:01:22).
crawler_1   | Scraping round completed (duration 0:01:27).
crawler_1   | Scraping round completed (duration 0:01:09).
crawler_1   | Scraping round completed (duration 0:01:20).
crawler_1   | Scraping round completed (duration 0:01:18).
crawler_1   | Scraping round completed (duration 0:00:55).
...

and of that checking staker activity (i.e. running partition):

crawler_1   | ✓ ... Staker Confirmation Status [62.0s]
crawler_1   | ✓ ... Staker Confirmation Status [46.0s]
crawler_1   | ✓ ... Staker Confirmation Status [61.0s]
crawler_1   | ✓ ... Staker Confirmation Status [61.0s]
crawler_1   | ✓ ... Staker Confirmation Status [41.0s]
crawler_1   | ✓ ... Staker Confirmation Status [62.0s]
crawler_1   | ✓ ... Staker Confirmation Status [52.0s]
crawler_1   | ✓ ... Staker Confirmation Status [61.0s]
crawler_1   | ✓ ... Staker Confirmation Status [63.0s]
crawler_1   | ✓ ... Staker Confirmation Status [59.0s]
crawler_1   | ✓ ... Staker Confirmation Status [60.0s]
crawler_1   | ✓ ... Staker Confirmation Status [64.0s]
crawler_1   | ✓ ... Staker Confirmation Status [46.0s]
crawler_1   | ✓ ... Staker Confirmation Status [60.0s]
crawler_1   | ✓ ... Staker Confirmation Status [58.0s]
crawler_1   | ✓ ... Staker Confirmation Status [39.0s]
...

I'm trying to limit changes made to agents when contracts are going to change anyway.

Will close for now, but will keep an eye out and revisit later if there are efficiency problems.