MI-DPLA / combine

Combine /kämˌbīn/ - Metadata Aggregator Platform
MIT License
26 stars 11 forks source link

Introduce graph theory to Job data model #327

Closed ghukill closed 5 years ago

ghukill commented 5 years ago

As the data model for Jobs, and their interconnectedness grew, it's becoming clear that it's defying simple sorting and "upstream" and "downstream" thinking.

It's likely that baking in a sense of graph theory would be exteremely helpful: https://www.geeksforgeeks.org/topological-sorting/ https://en.wikipedia.org/wiki/Topological_sorting

Do not foresee this as small or quick addition, but noting for future consideration.

ghukill commented 5 years ago

At minimum, consider use of toposort: https://bitbucket.org/ericvsmith/toposort

Even includes cyclical checks that would prevent linkages like A-->D and D-->A from existing.

ghukill commented 5 years ago

Toposort added. Closing issue here, as this is too large a change / consideration for a single issue. Using topographical sorting has helped tremendously for determining the correct re-run lineage, and can trickle into other areas, but treating the Job networks as graphs, and reaping the benefit of doing so, would require some real planning.