JRFC 37 - Dependencies on Github

[jbenet]

Adding Dependencies to Github Issues

This is an old idea, many people have had it. I'm just laying out here as a point for discussion, so that we can build it.

Desires

• I would like to be able to link github issues to each other with a "depends on" relationship.
• I want graphs like this:
• It wont happen on github core: https://github.com/isaacs/github/issues/210
• I want gantt charts

  Proposal

  1. Parse Dependencies from Comments

We can parse comments in an issue and detect text like:

Depends on https://github.com/ipfs/go-ipfs/issues/100

To make the dependency graph explicit. Then we can create a tool (or a site like waffle.io) that crawls a repo's issues.

2. Put dependencies on ipfs

Take the crawled dependencies and put them on IPFS (dag base)

3. Generate visual graphs

Use tools like graphviz to visualize dependency graphs.

4. Webapp showing deps

We can make a webapp that shows dependency graphs for:

• a single gtihub issue
• a selcted set of github issues (github search)
• a github label
• a github milestone
• a github repo
• a github org

(Note that if we implement support based on running github searches, we get most of these for free.)

Other Considerations

• Tool/site should be able to crawl a set of orgs (all repos), or a set of repos.
• Tool should show what's done (green), ready to be worked on (yellow), what's blocked by a dependency (red), etc. (See how other tools figure out the state / info to show)
• Security: option to only look at the main issue comment/post (first, top). This way people can't troll an issue adding bogus dependencies. Requires admin powers to edit the main issue post and add the dependencies there.

[lidel]

Sounds really useful, especially with a webapp: dependency graph could be embedded via ![](https://webapp/ghuser/project/deps.png) (just like CI badges are).

[jbenet]

Oh yeah, great idea! On Thu, Aug 4, 2016 at 04:15 Marcin Rataj notifications@github.com wrote:

Sounds really useful, especially with public webapp: dependency graph could be embedded via (just like CI badges are).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jbenet/random-ideas/issues/37#issuecomment-237484572, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIcocJNd8ZWowBiq5XV3_rcdAERSkWoks5qcZ-zgaJpZM4JcSnd .

[jbenet]

Relationships:

• Depends on denotes a "dependency", an issue that MUST be finished before this one can finish.
• Contains denotes a "sub-issue", something that's not just a dependency, it's "part of" this issue.

Could use checkboxes to show when they're done. Would prob involve manual overhead though, unless we wanted to do a lot of scripting...

# example with checkboxes:
- [ ] Depends on: <link-to-issue>
- [ ] Contains: <link-to-issue>

[jbenet]

I'm actively looking for someone to make this for me, could pay you for it -- I would really benefit from this and dont have the bw to make it atm.

[hackergrrl]

Hey @jbenet :wave:

I really like the idea, and would also like to see this happen. I spent some time today writing https://github.com/noffle/github-dependency-crawl. It crawls a github repo recursively for issue dependencies, and spits out the dependency graph. Its pretty pluggable too (you don't even need to use github as the data source, so you could e.g. pull issue data from IPFS). I'm actually using it to manage its own dependencies :grinning: https://github.com/noffle/github-dependency-crawl/issues/7

[jbenet]

@noffle :wave: very nice! thanks for making that, it's is a great start. 👍 🎉

you don't even need to use github as the data source, so you could e.g. pull issue data from IPFS

may want to keep the full URL of the issue then? looks like right now it removes github.com/ and issues/ which may not be desired if issues are pulled from elsewhere? (not sure)

[jbenet]

graphviz on the web:

• https://github.com/mdaines/viz.js/ and https://mdaines.github.io/viz.js/ <--- this looks to be the best bet for graphviz
• http://www.webgraphviz.com/ <-- not sure if it's the same. no links to code.
• https://github.com/mstefaniuk/graph-viz-d3-js
• http://graphviz.it/#/gallery/longflat.gv

(3 years ago i really wanted these and they didnt exist. very happy they do now)

other libs:

• http://js.cytoscape.org/
• http://arborjs.org/
• https://github.com/OpenKieler/klayjs
• https://github.com/cpettitt/dagre
  • https://github.com/cpettitt/dagre-d3
  • http://cpettitt.github.io/project/dagre-d3/latest/demo/interactive-demo.html an
• http://marvl.infotech.monash.edu/webcola/
  • http://marvl.infotech.monash.edu/webcola/examples/downwardedges.html
• http://hughsk.io/colony/

not related, but this is pretty cool:

• https://www.websequencediagrams.com/

[jbenet]

This seems to be the trac plugin's code:

• https://trac-hacks.org/browser/tracticketdepgraphplugin/0.11/depgraph#templates
• this produces the graphivz: https://trac-hacks.org/browser/tracticketdepgraphplugin/0.11/depgraph/depgraph.py
• should be able to port that to using @noffle's module

[hackergrrl]

may want to keep the full URL of the issue then? looks like right now it removes github.com/ and issues/ which may not be desired if issues are pulled from elsewhere? (not sure)

Without getting into detail here, the canonical string should be sufficient to let you deduce the source URL from the data source of choice (owner/repo/issue has a well-documented mapping to an API endpoint, and if it's on IPFS that mapping should also be well-documented, e.g. /ipfs/HASH/gh/owner/repo/issue).

[jbenet]

I took a stab at the dot generation. this is a super basic version: https://github.com/jbenet/issue-dep-graph

> issue-dep-dot https://github.com/noffle/github-dependency-crawl
digraph deps {
    noffle_github_dependency_crawl_12 [label="noffle/github-dependency-crawl/12"];
    noffle_github_dependency_crawl_11 [label="noffle/github-dependency-crawl/11"];
    noffle_github_dependency_crawl_10 [label="noffle/github-dependency-crawl/10"];
    noffle_github_dependency_crawl_9 [label="noffle/github-dependency-crawl/9"];
    noffle_github_dependency_crawl_8 [label="noffle/github-dependency-crawl/8"];
    noffle_github_dependency_crawl_7 [label="noffle/github-dependency-crawl/7"];
    noffle_github_dependency_crawl_4 [label="noffle/github-dependency-crawl/4"];
    noffle_github_dependency_crawl_5 [label="noffle/github-dependency-crawl/5"];
    noffle_github_dependency_crawl_6 [label="noffle/github-dependency-crawl/6"];
    noffle_github_dependency_crawl_3 [label="noffle/github-dependency-crawl/3"];
    noffle_ipget_18 [label="noffle/ipget/18"];
    noffle_github_dependency_crawl_2 [label="noffle/github-dependency-crawl/2"];
    noffle_github_dependency_crawl_1 [label="noffle/github-dependency-crawl/1"];
    ipfs_ipget_24 [label="ipfs/ipget/24"];
    ipfs_ipget_26 [label="ipfs/ipget/26"];
    ipfs_ipget_20 [label="ipfs/ipget/20"];
    ipfs_ipget_21 [label="ipfs/ipget/21"];

    noffle_github_dependency_crawl_7 -> noffle_github_dependency_crawl_4;
    noffle_github_dependency_crawl_7 -> noffle_github_dependency_crawl_5;
    noffle_github_dependency_crawl_7 -> noffle_github_dependency_crawl_6;
    noffle_github_dependency_crawl_3 -> noffle_ipget_18;
    noffle_github_dependency_crawl_2 -> noffle_github_dependency_crawl_3;
    noffle_github_dependency_crawl_1 -> noffle_github_dependency_crawl_2;
    noffle_github_dependency_crawl_1 -> noffle_github_dependency_crawl_3;
    noffle_ipget_18 -> ipfs_ipget_24;
    noffle_ipget_18 -> ipfs_ipget_26;
    noffle_ipget_18 -> ipfs_ipget_20;
    noffle_ipget_18 -> ipfs_ipget_21;
}

[zjiekai]

Is gantt charts something like this(http://dhtmlx.com/docs/products/dhtmlxGantt/01_basic.html)?

[jbenet]

@zjiekai yeah! I'd like to generate them from github issues that have some dependency (and "containing") structure.

It's hard to figure out the time it will take, but even charting out the dependenceis, some basic assumptions (like N hrs per issue) may generate a really good perspective, and logging the past accurately would be super helpful.

"containing issues" means one issue "contains" another (more than a simple dependency, it's treating an issue like a bundle of other issues (milestones or epics). GH Milestones don't fit all needs because GH Milestones are per repo and not across repos. (some prefer milestones backed by an issue)

[jbenet]

More on graphviz:

• looks like https://github.com/mstefaniuk/graph-viz-d3-js is better maintained. it's also what backs http://graphviz.it/ which is pretty good.
• either way, we should be generating SVG cause then we can hook onto that to render other things, like links, attaching images (for assignee images, labels, colors, and more).
• one random thing to deal with: normalize size of the nodes, using some max size for the issue identifier

[zjiekai]

@jbenet Hi, I have a tiny demo here(https://zjiekai.github.io/issue-dep-graph/). I'm not sure about how to set the start of an issue. Ideally it may look like this:

Do you prefer using the dot graph?

@noffle I'm not sure how to get the issue data. So I'm using this hack(https://github.com/zjiekai/github-dependency-crawl/commit/e7a7b1783a472606fd9cca8b290d29dd104b5853) for now.

Update: @jbenet I just saw your above comment. Personally I find the graphviz layout to be more pleasing than gantt chart. I'm wondering why not use graphviz directly once but having every web view running the js version in a web worker?

[hackergrrl]

@zjiekai The API doesn't expose raw issue data yet. Coming Soon(tm): https://github.com/noffle/github-dependency-crawl/issues/13

[jbenet]

@zjiekai

• thanks, that's pretty cool! that will come in handy.
• Does it use the open time and close time of an issue?
• My goal with gantt charts is to use dependencies and some "duration estimate" to figure out realistically how far out an issue is. read up on gantt charts and how they're used here: https://en.wikipedia.org/wiki/Gantt_chart

[zjiekai]

Hi, @jbenet In the demo, I'm using the open time as the start and a 7-day duration.

• Should the open time of an issue also be the start time? But an issue can be opened before its dependents, but can only be started after its dependents are finished? If there are no careful restrictions the gantt charts can be rather messy.
• I have zero experience working on an open source project. Maybe it would be better if someone with more experience can take the lead and I can try to contribute.

[serapath]

reminds me of https://en.wikipedia.org/wiki/Precedence_diagram_method

[jbenet]

@zjiekai

• an issue can be opened before its dependents, but can only be started after its dependents are finished?

well not strictly. gantt charts try to forecast project durations based on strict pre-reqs. maybe some would argue an issue should be split in two (the part than can be started before-pre-req, and the part that hard-requires the pre-req). It's a tough call, it depends on what the utility is supposed to be. Maybe the chart could show a faded background color from "open date" to "close date", and a darker color from "all pre-reqs closed date" and "close date". or something.

• If there are no careful restrictions the gantt charts can be rather messy.

yeah not very useful.

[jbenet]

cc @jesseclay

[jbenet]

Made some mocks of a "depviz" webapp to address all this.

depviz.mocks.pdf

preview:

---

---

---

[jbenet]

I have created https://github.com/jbenet/depviz to track work on this webapp.

[burdges]

I'd worry less about the charts per se than simply about allowing users to edit the information sanely.

In particular, if you're discussing github issues, then an arrow might just be someone's opinion that resolving an issue in necessary for some other issue. If however either issue got resolved somewhat differently, then that arrow might not make sense.

Arrow existence and direction still remains more relevant than just an internal github backwards link though because it'll help people understand the discussion faster. And help newbies pick issues to work on.

[jbenet]

@burdges yeah makes sense. i think the flexibility we have with marking depends on X in an issue is enough here-- people will be able to decide on that. Toggling modes so that only markings in the initial issue comment counts (i.e. must have edit perms) can strike the right balance for what a project's thoughts are on what counts as a dependency (whether to be strict, or loose)

[jbenet]

Quick update here: the excellent @wking and I have been making some headway and progress in the implementaiton of depviz (impl all by wking). Follow the progress: https://github.com/jbenet/depviz or try it out: https://jbenet.github.io/depviz

examples:

• https://jbenet.github.io/depviz/#/http/github.com/jbenet/depviz
• https://jbenet.github.io/depviz/#/http/github.com/jbenet/depviz/30

[SparksFyz]

HI, I have a question. When you use viz.js, the node in viz is svg, but deps node here is very complex(like embeded html) and interactive(embeded buttons...), how your guys solve this issue?

[wking]

On Tue, Dec 27, 2016 at 05:31:39AM -0800, Sparks 范宜臻 wrote:

… the node in viz is svg, but deps node here is very complex(like embeded html) and interactive(embeded buttons...), how your guys solve this issue?

The nodes are rectangular, so all depviz uses viz.js for positioning rectangular nodes (and eventually for routing edges between them, jbenet/depviz#24). The graph-positioning engine doesn't have to care about who's drawing the node content. depviz currently uses SVG node content as well (which supports embedded buttons, etc.), but if you wanted you could use foreignObject and HTML inside the nodes 1.

[SparksFyz]

thanks~ I understand what you mean. @wking

[moul]

Something new from GH: https://github.com/blog/2300-visualize-your-project-s-community

[wking]

On Thu, Jan 05, 2017 at 08:36:50AM -0800, Manfred Touron wrote:

Something new from GH: https://github.com/blog/2300-visualize-your-project-s-community

That is for dependencies between projects. This issue and depviz are for dependencies between issues. The latter are likely to be a more-specific version of the former (e.g. “graphql-parallel depends on graphql-ruby” is a project-level dependency and “graphql-parallel#123 depends on graphql-ruby#456” is an issue-level dependency).