nucypher / nucypher-monitor

NuCypher network intelligence crawler and web dashboard

7 stars 15 forks source link

Updates to work with the new FleetSensor API #94

Closed fjarri closed 3 years ago

fjarri commented 3 years ago

In https://github.com/nucypher/nucypher/pull/2352 FleetSensor and node metadata API changed. This PR updates the calls in the monitor. Needs https://github.com/nucypher/nucypher/pull/2574 to work, so the CI will likely fail...

There is also a change in how crawler storage is implemented. Before it was derived from ForgetfulNodeStorage, which presented several problems:

crawler stores node statuses and fleet states, while ForgetfulNodeStorage stores node metadata, so the hierarchical relation between them is questionable
node status cannot be obtained from a node object itself, so the crawler storage simply did not have access to enough data

In this PR, CrawlerStorage is its own class, and its calls are hooked to FleetSensor updates (see hooked_tracker_class()).

Important: One consequence of this is that we now do not store nodes right away before they are even verified; now the nodes are stored (or removed) when the fleet state changes. This can be modified, but currently I am unclear as to what behavior was really expected from the crawler, so I went with the logical one.

fjarri commented 3 years ago

Presumably, you've run it locally with a nucypher version containing #2574?

I did, but to be on the safe side someone else should too.

derekpierre commented 3 years ago

Ran it locally with relocked dependencies (pointed to nucypher on main), and it seemed to run fine. The scraping round took some time (9-10mins), but I usually find that it takes longer on my local machine, and ends up being much faster on the server (~ 1-2mins).

fjarri commented 3 years ago

Current status: waiting for the new release with merged https://github.com/nucypher/nucypher/pull/2574 so that CI could run

fjarri commented 3 years ago

With nucypher updated to 4.8, tests pass now.

codecov-io commented 3 years ago

Codecov Report

Merging #94 (79ebef3) into main (f8df51a) will decrease coverage by 1.00%. The diff coverage is 75.80%.

@@            Coverage Diff             @@
##             main      #94      +/-   ##
==========================================
- Coverage   51.89%   50.89%   -1.01%     
==========================================
  Files          11       11              
  Lines        1079     1059      -20     
==========================================
- Hits          560      539      -21     
- Misses        519      520       +1

Impacted Files	Coverage Δ
monitor/cli/_utils.py	`40.00% <0.00%> (ø)`
monitor/crawler.py	`43.65% <75.00%> (-3.09%)`	:arrow_down:
monitor/db.py	`95.69% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update f8df51a...79ebef3. Read the comment docs.

fjarri commented 3 years ago

Actually the dependency relock can probably be rolled back to minimize the changes. nucypher 4.8 will be installed anyway.