MatrixAI / Polykey

Polykey Core Library
https://polykey.com
GNU General Public License v3.0
31 stars 4 forks source link

Add priority and TTL to `Discovery` process #462

Open tegefaulkes opened 2 years ago

tegefaulkes commented 2 years ago

Specification

Discovery has recently been updated in PR #451 . Further review of the discovery logic needs to be done and the core logic refactored. The processVertex logic is expansive so if possible, parts of it needs to be extracted out into protected functions to help with readability.

Priority order needs to implemented for discovering gestalts. We can make use of the TaskManager's priority system to do this. We care about gestalts in 3 tiers. Gestalts should be discovered in this order and any other gestalts we don't really care about.

  1. Our own gestalt
  2. Any gestalt we trust
  3. Any gestalt that the user triggers manually

A TTL system needs to be applied to re-visit gestalts to check for any updates. This really just comes down to triggering discovery on the gestalt again after a delay.

We skip vertices that we have visited recently. this is currently done rather simply with verticies stored in an in-memory set. We to implement a more sophisticated method for tracking whether we should skip over a vertex. This needs to be persistent and only last for a certain amount of time. This can be done with a database entry tracking the last time a vertex was visited.

Note that for now the ability to Seek to the last processed claim for a node has been disabled. Given the level of complexity discovery is currently at we need to process a nodes whole sigchain to do re-discovery. To use the seeking feature we need to have re-discovery properly implemented.

Re-discovery will make use of already known links within the gestalt graph to populate the discovery queue. We will need to keep track of the last time a node was discovered and only re-discover nodes that haven't been checked after a given time. To this end we need to track for each node the last time it was discovered and the last claimId of its sigchain we processed.

Additional context

Tasks

CMCDragonkai commented 1 year ago

@tegefaulkes please update this issue with information about https://github.com/MatrixAI/Polykey/pull/446#issuecomment-1334739788

tegefaulkes commented 7 months ago

Points 2, 3, 5, 6, 7, 8 of this have been addressed by generally and by recent changes in #696. So the only thing really left to address here is a priority system for ordering processing. That's pretty low priority at this stage.