Open tegefaulkes opened 2 years ago
@tegefaulkes please update this issue with information about https://github.com/MatrixAI/Polykey/pull/446#issuecomment-1334739788
Points 2, 3, 5, 6, 7, 8 of this have been addressed by generally and by recent changes in #696. So the only thing really left to address here is a priority system for ordering processing. That's pretty low priority at this stage.
Specification
Discovery has recently been updated in PR #451 . Further review of the discovery logic needs to be done and the core logic refactored. The
processVertex
logic is expansive so if possible, parts of it needs to be extracted out into protected functions to help with readability.Priority order needs to implemented for discovering gestalts. We can make use of the
TaskManager
's priority system to do this. We care about gestalts in 3 tiers. Gestalts should be discovered in this order and any other gestalts we don't really care about.A TTL system needs to be applied to re-visit gestalts to check for any updates. This really just comes down to triggering discovery on the gestalt again after a delay.
We skip vertices that we have visited recently. this is currently done rather simply with verticies stored in an in-memory set. We to implement a more sophisticated method for tracking whether we should skip over a vertex. This needs to be persistent and only last for a certain amount of time. This can be done with a database entry tracking the last time a vertex was visited.
Note that for now the ability to Seek to the last processed claim for a node has been disabled. Given the level of complexity discovery is currently at we need to process a nodes whole
sigchain
to do re-discovery. To use the seeking feature we need to have re-discovery properly implemented.Re-discovery will make use of already known links within the gestalt graph to populate the discovery queue. We will need to keep track of the last time a node was discovered and only re-discover nodes that haven't been checked after a given time. To this end we need to track for each node the last time it was discovered and the last
claimId
of itssigchain
we processed.Additional context
Tasks
processVertex
logic, it's branching and messy.ClaimId
.GestaltGraph
for nodes and identities to re-discover.ClaimId
we saw.