Solve dependency hell: link with more than one versions of the same project

[wilzbach]

Description

Let's see that in project X you use an XML parser of v1. This XML parser depends on a collections library with v1. Now, however, the team decides to a add a JSON parser too, but this one depends on a newer version of the collections library with v2. The result is that the team can't use the JSON parser in their project and currently the only viable strategies are:

• copy/paste the collections library to a new namespace and let the JSON parser depend on this new collections namespace
• try to upgrade the XML parser to v2

(1) is very cumbersome, requires a lot of manual work and makes it hard to sync up with upstream. (2) is often not possible.

Hence, the aim of this project is to allow people to use v1 and v2 of collections in parallel in their dub projects. Rust allows to fix this dependency hell problem by appending a unique suffix to each symbol of the conflicting versions. This works as long as each dependency only uses one direct version of a dependency. This is a very reasonable assumption and this approach should work for D too.

What are rough milestones of this project?

0) investigate solutions used in other languages or active research 1) create two DMD PR

• to allow it to append unique suffixes to all generated symbols (e.g. --mangle-suffix=v1.1)
• to map unique suffixes to used symbols (e.g. --mangle-suffix-map=json:v1.1) 2) add the respective CLI flags to all dub invocations (be sure only to do so if frontend >= X)
  • solve a small minimal example 3) only add CLI flags to dub dmd invocations when they are required (e.g. SemVer conflict)
  • create more test cases 4) open DUB PR 5) get DUB PR merged
  • make sure it passes on all CIs
  • address review

Optional improvements:

• allow the user to completely disable this behavior and error when an unsolvable conflict has been found (= revert to current behavior)
• allow to user to declare the exact semver multi-mapping in his/her's project config

How does this project help the D community?

It is a first milestone and a requirement to moving the standard library Phobos to Dub. Additionally, it will have a massive impact on the D community as any bigger D project that has a non-trivial amount of dependencies runs into a dependency hell.

Recommended skills

• medium experience with D
• experience with a package manager
• understanding of the linking process
• no fear of the DMD frontend

What can students expect to get out of doing this project?

• learn about D's name mangling system
• learn how D packages are built
• create and integrate a high-impact project for the D community

Rating

Medium

Project Type

Infrastructure/Automation

Point of Contact/Possible Mentors

@wilzbach @maxhaton

References

• https://stephencoakley.com/2019/04/24/how-rust-solved-dependency-hell

[andre2007]

This topic becomes more and more important, I add the gsoc2021 label.

[RazvanN7]

I'm going to remove this from GSOC 2021 as we don't have a mentor for it.

[ahmetsait]

I got frustrated by this before so I would be interested in solving this :)

[maxhaton]

Added GSOC '21 since I am available to mentor it

[ahmetsait]

Alright so D Language Foundation wasn't able to participate in GSoC 2021 but there is SAoC coming up next.

I would like to ask if this project would be a good fit for SAoC, do you think 4 months time frame of SAoC lines up with the work expected from this project or would it fall short (i.e. too easy) ?

[RazvanN7]

This is a good fit for SAoC.