New implementation for `add-local` / `add-path`

[Geod24]

This has been discussed on Slack with @WebFreak001 . Posting it here if someone wants to pick up the issue.

I was thinking last night about location / add-local / add-path. Something that bothered me is that we had 3 files per location: 1 config file, 1 override, and 1 for add-local / add-path (path are just marked with "name": "*" in the JSON). As you know, I want to change the directory structure so that we don't have to iterate over the directory to list packages, we can just use the path. That would solve it for packages fetched, but not for packages added locally / in the search path. Packages added locally are not the problematic though, it's just a JSON read. However, I think we could do the following:

• Whenever one does add-local, create a file in packages/$NAME/$VERSION/$NAME that points to the path (like git submodule does;
• Move extra search path to the config file;

This way, we reduce the special-casing around local packages (they just become a symlink handled by the application) and reduce locations config to a single file. Thoughts ?

@WebFreak001 answered:

sounds like a solid plan and also easier for the user to understand than random config files

[s-ludwig]

I don't think anything can be done in that regard for add-local/add-path packages. The contained "version" can change at any moment by checking out a different commit, so it needs to be re-checked on every run. There could maybe some kind of optimization where it is assumed that the same package stays in the same folder, but even that needs to be updated at some point.

[Geod24]

The contained "version" can change at any moment by checking out a different commit, so it needs to be re-checked on every run.

But we don't. We determine the version (or get it provided explicitly) when doing add-local and stick with that. We even have a check that we don't add something with a different version. Full code here: https://github.com/dlang/dub/blob/7175eadfa7a7f58522caa524e42e4419918da408/source/dub/packagemanager.d#L713-L737

Regarding add-path, yes, we need to re-scan every time. But moving the entries to the location/repository config removes one extra file and makes things more visible.

[s-ludwig]

But we don't. We determine the version (or get it provided explicitly) when doing add-local and stick with that. We even have a check that we don't add something with a different version.

Okay, that's really weird behavior, no idea why it was decided to be done that way, TBH. So it would indeed work for add-local then in that regard. However, that still doesn't cover the precedence issue.

Regarding add-path, yes, we need to re-scan every time. But moving the entries to the location/repository config removes one extra file and makes things more visible.

I just get a weird feeling when an application starts to modify user provided configuration files. So far, settings.json only contains fields set by users and local-packages.json/local-overrides.json are completely managed by dub. Is configy able to retain the existing formatting when applying changes?

[Geod24]

I just get a weird feeling when an application starts to modify user provided configuration files. So far, settings.json only contains fields set by users and local-packages.json/local-overrides.json are completely managed by dub. Is configy able to retain the existing formatting when applying changes?

Think of it as git config :) And yes, D-YAML is able to retain the style, but Configy probably will need to be extended for this (it was never intended for writing).

[WebFreak001]

we also need such a framework for user-friendly JSON writing for dub.selections.json (dub select)