Dependency tracking

[jakirkham]

I have been thinking about this for a long time. We need to have a way to visualize and handle dependency relations between feedstocks both at the overview level and at the maintainer level. The problem is still a bit amorphous, at least in my mind, but I hope by putting this out there we can get a good discussion going and figure out how to handle and manage issues like these and come up with good tools to help us.

[jjhelmus]

I think it may be possible to use NetworkX for at least the visualization part.

[pelson]

I think that would be a fantastic way to visualise the packages that conda-forge provides. :heart:

[ocefpaf]

I made a very hackish visualization of conda packages dependencies in the past.

See https://ocefpaf.github.io/python4oceanographers/blog/2014/11/17/networkX/

I am sure you guys can improve upon that :wink:

[pelson]

I'd also like to see the packages upon which our feedstocks depend, particularly those which don't themselves have feedstocks.

[msarahan]

indeed - that would be a great tool to highlight where to work next. Ideally, the visualization/list could also be platform-specific, since we have many packages that omit some platforms.

[jakirkham]

I could see some other nice use cases as well. Take for instance a package is the dependency of lots of things, but has few maintainers. We could identify this (perhaps with some color coding) and try to encourage people to help support it. Also, it could help identify packages that are not actively supported by anyone.

[jakirkham]

Looks like @pelson is coming up with a script to detect missing dependencies in this PR ( https://github.com/conda-forge/conda-forge.github.io/pull/80 ).

[pelson]

Looks like @pelson is coming up with a script to detect missing dependencies in this PR ( #80 ).

It is pretty rudimentary, but gives me some useful information & I wanted to share early.

[jakirkham]

See this comment. There is a strategy for doing this at Continuum. We should look and see what changes are required to accommodate that here. Thus far, it doesn't sound so bad.

[jakirkham]

So there is now feedstockrot thanks to @RSully, which should help.

[jakirkham]

Also saw this for pip, which is kind of nice. Maybe something worth drawing on.

[jakirkham]

As far as using networkx, that seems like a good idea, but we still need a package for it ( https://github.com/conda-forge/staged-recipes/issues/1259 ). 😜

[jakirkham]

Also ran across toposort, which seems pretty handy for planning large scale upgrades. Maybe also helpful in diagnosing where things might have gone wrong during such an upgrade.

[pmlandwehr]

Worth mentioning snakefood here.

[jakirkham]

Think this is to some degree addressed by @CJ-Wright and @justcalamari's work (ref below). Would be good to be able to visualize that graph though. Not sure if either of you have something or ideas along those lines.

ref: https://github.com/regro/cf-scripts

[sodre]

I will take a stab at the visualization using datashader+holoviews+bokeh similar to how Anaconda did here.

edit: changed link to page instead of notebook

[jakirkham]

Sound like a good idea. Might also want to take a look at this tutorial. Admittedly Datashader may be needed to get this to work at our scale.

[CJ-Wright]

The data for this is in https://github.com/regro/cf-graph-countyfair and https://github.com/regro/libcfgraph