Vcpkg support - Githubissues

JamieMagee commented 4 years ago

What would you like Renovate to be able to do?

Vcpkg is a dependency manager for C++ projects. It's still in the early stages, but it's backed by Microsoft, and is likely to be widely used.

Currently vcpkg doesn't have support for versioning, but there is currently an RFC open at microsoft/vcpkg#11758. The format of vcpkg.json is likely to be:

{
    "name": "project", 
    "version": "1.0.0",
    "version-scheme": "semver",
    "dependencies": [
        { "name": "zlib", "version": "1.2.11" },
        { "name": "rapidjson", "version": "2020-02-08", "version-scheme": "date" }]
}

There are more complex examples in the PR, including minimum versions, overrides, and exclusions.

Did you already have any implementation ideas?

Are there any workarounds or alternative ideas you've tried to avoid needing this feature?

Also need to wait for registry support to be complete microsoft/vcpkg#13038

Is this a feature you'd be interested in implementing yourself?

Yes, whenever the RFC is complete.

rarkins commented 4 years ago

Is it an alternative to Conan, or somehow complementary?

JamieMagee commented 4 years ago

It's an alternative. Currently vcpkg appears to be much earlier in development than conan is, so if I had to prioritise one I'd prioritise conan right now.

ghost commented 3 years ago

Is there any plan to implement vcpkg support at this point?

JamieMagee commented 3 years ago

I'm not actively working on it right now, and I don't think I'll be working on it in the near-term. I'm busy with #8647.

We can give help and guidance if it's something you're interested in implementing yourself? We have a Slack for contributors if you'd like to join.

HonkingGoose commented 3 years ago

Currently vcpkg appears to be much earlier in development than conan is, so if I had to prioritise one I'd prioritise conan right now.

I'm not actively working on it right now, and I don't think I'll be working on it in the near-term. I'm busy with #8647.

I'll label this priority-4-low then, as we wanted to prioritize the Conan package manager first, and it doesn't seem like we'll get around to working on the vcpkg manager soon.

JamieMagee commented 3 years ago

I agree with the priority-4-low label, but not with Conan over vcpkg. In the year since I made that comment I think vcpkg has overtaken Conan in terms of industry use (based on files checked in on GitHub and stars on each project).

Though if support for either comes via the community, we'll obviously accept it.

ghost commented 3 years ago

@JamieMagee sorry for the late reply. I've done some reconnaissance for possible implementation methods and I think I can try to work on this. Here is my idea of how it could be implemented, so even if I'm not able to implement it for some reason others will have somewhere to start.

To start, we keep an index of package versions. This is updated when needed.

vcpkg does versioning by commit, and so to check for new versions, I see two decent options to monitor the latest commit on the main branch:

We use the GitHub API (via REST/GraphQL/octokit).
We use git ls-remote.
We clone the repo once at the start of every deployment, and run git fetch to check

If there is a new commit, we can now update our package index. (The following step can be ignored if we use strategy 3 above.) In order to get the package versions, we have to clone the repo. However, if we clone it with --single-branch and --depth=1, the total size after downloading (as of this post) goes from 60 MB to 27 MB.

Once we have the repo, we can get the package versions which are hard-coded into the repo. In figuring out how to do this, the script used by the vcpkg website will probably be good place to start.

Once we've updated this package index, it's just a matter of comparing against the last one for differences (which can probably be done as we create the updated package index), and sending out PRs. (I'm guessing there's an internal API/module to handle this.)

Note: Any git commands I've provided would probably be done via simple-git (which Renovate already uses).

We can give help and guidance if it's something you're interested in implementing yourself? We have a Slack for contributors if you'd like to join.

That would be great if you can!

rarkins commented 3 years ago

@applemonkey496 great!

Am I understanding correctly that vpkg keeps a central git repository of packages and versions instead of having a traditional "registry"? And I'm guessing it's https://github.com/microsoft/vcpkg/tree/master/ports ?

This reminds me of Cocoapods, who moved away from this approach as it was non-performant and crushing GitHub as the host. Also similar to homebrew, who now changed the client to not do shallow clones AFAIK. Any similar concerns about vcpkg?

I think that doing a clone the first time that there's a vcpkg datasource lookup would be ok, regardless of shallow or full. Then we could maybe limit our fetching to every x minutes instead of every single time there's another lookup.

ghost commented 3 years ago

Am I understanding correctly that vpkg keeps a central git repository of packages and versions instead of having a traditional "registry"? And I'm guessing it's https://github.com/microsoft/vcpkg/tree/master/ports ?

Yes, I believe so.

Any similar concerns about vcpkg?

If by concerns you mean whether vcpkg may also move away from this approach of a centralised git repository for packages, I wasn't able to find any mention of this in the repository (I tried several search terms). I don't think it's a concern for them, at least not at the moment.

I think that doing a clone the first time that there's a vcpkg datasource lookup would be ok, regardless of shallow or full. Then we could maybe limit our fetching to every x minutes instead of every single time there's another lookup.

To clarify, what do you mean by "vcpkg datasource lookup"? Do you mean like strategy 3, cloning on deployment?

JamieMagee commented 3 years ago

I think that the microsoft/vcpkg repository is the default registry, but vcpkg also supports custom registries as well. Here's a few articles that I think will help:

@rarkins Yes it's going via GitHub, but Microsoft might have a bit more leeway than the Cocoapods team did 😉

We might be able to avoid cloning the whole repository as well. The official website has an API that we might be able to use. There are also community supported package indexes and APIs:

vcpkg.info, API, no GitHub repo
vcpkgx.com, API and GitHub repo
vcpkg.dev, no API, but GitHub repo

IMO some of the unofficial APIs seem more powerful than the official one. Either we'll have to reach out to the people running the community ones, or push for improvements in the official one (I can reach out within MSFT).

Also we'll need to be careful with vcpkg's versioning, which actually encompasses 4 different versioning schemes to cover all the possible versioning schemes in the C++ ecosystem. The relevant docs are:

I know this seems like a lot, but there are some clear pieces that can be broken down (manager, datasource, versioning)

ghost commented 3 years ago

Can't believe I missed that! I was sifting through vcpkg.io's source code to see how they got their data (that's where I found that PowerShell script), and I even saw the committed output.json file. I never considered that it would have to be published on the website somewhere... (In retrospect, it would have also been possible to use the raw.githubusercontent.com link to achieve a similar result.)

rarkins commented 3 years ago

To clarify, what do you mean by "vcpkg datasource lookup"? Do you mean like strategy 3, cloning on deployment?

In Renovate we separate out the concept of "managers" (and their associated package files) from "datasources" (which are registries with lists of packages and versions). Sometimes they are very related, e.g. the npm manager uses npm datasources, but sometimes they may be separate such as poetry manager and pypi datasource. In this case we'll probably call both the manager and datasource to be "vcpkg" but would implement the datasource part first.

rarkins commented 3 years ago

I'd prefer to avoid unofficial APIs if possible, because they are more likely to disappear one day or have downtime. Do we think they are basing their data wholly on the output.json, or on other data from within the registry itself?

Is the output.json committed to the repo? I got that impression from @applemonkey496's last comment but I don't see it being updated in any of the commits.

ghost commented 3 years ago

output.json is committed to the repo, but I see two concerns using it:

It's rarely updated. They do update it but it's only about once a month.
It's also an internal API used by the website, which means that the interface may be unstable.

However, looking at the unofficial APIs, they update much more frequently:

vcpkg.dev updates every night at midnight.
vcpkgx.com updates every hour.
vcpkg.info even has a changelog with history, but although it updates often, it's difficult to tell exactly how often because we can't see the source code. (I found the author's initial announcement of the website and he doesn't mention a repo either.)

I do also see similar "unstable" concerns for the unofficial APIs. None of them appear to be documented and so they may be for internal use. Whichever API we choose (if any), official or unofficial, I think it's important that we reach out to the maintainers to ensure a stable interface. (Or, we could fork one of them and host a stable version on GitHub Pages.)

ghost commented 3 years ago

@rarkins @JamieMagee I am still willing to work on this. However, I am unable to do so unless a consensus is reached on API choice.

rarkins commented 3 years ago

I think it's best if we construct our index from the repo itself, unless there were any good reasons against that which I missed? We can clone it if cold but then fetch it every hour perhaps, and only re-clone if the fetch fails. Ideally this should be long-lived and not cleared per-run or per-repo

ghost commented 3 years ago

Ok, I'll start some work on this. Once I have something, I can open a draft PR for feedback. This may take a bit because I'm quite busy and I'll be working on this in the background.

JamieMagee commented 3 years ago

I think I forgot to mention, but there is a GitHub repository for the official website, and there is an open issue about adding automation to keep the package list up-to-date: https://github.com/vcpkg/vcpkg.github.io/issues/21

JamieMagee commented 2 years ago

I'm working on this upstream https://github.com/vcpkg/vcpkg.github.io/pull/60

renovatebot / renovate

Vcpkg support #7135