search

renovatebot / renovate

Home of the Renovate CLI: Cross-platform Dependency Automation by Mend.io
https://mend.io/renovate
GNU Affero General Public License v3.0
17.59k stars 2.32k forks source link

Add support for nix manager #8000

Closed JamieMagee closed 1 year ago

JamieMagee commented 3 years ago

What would you like Renovate to be able to do?

Add support for the nix package manger. A typical nix expression might contain something like:

src = fetchFromGitHub {
  owner = "Ryujinx";
  repo = "Ryujinx";
  rev = "2dcc6333f8cbb959293832f52857bdaeab1918bf";
  sha256 = "1hfa498fr9mdxas9s02y25ncb982wa1sqhl06jpnkhqsiicbkgcf";
};

It appears to me that this is something that Renovate could help with.

Did you already have any implementation ideas?

Requires renovatebot/docker-buildpack#80

References:

jbalme commented 3 years ago

+1. The difficult thing here is going to be getting the NAR hash and updating it, otherwise Nix projects could just use the regex manager and git datasource.

JamieMagee commented 3 years ago

Related PR renovatebot/docker-buildpack#94

github-actions[bot] commented 2 years ago

Hi there,

You're requesting support for a new package manager. We need to know some basic information about this package manager first. Please copy/paste the new package manager questionnaire, and fill it out in full.

Once the questionnaire is filled out we will evaluate if adding support for this manager is something we want to do.

Good luck,

The Renovate team

JamieMagee commented 2 years ago

Please feel free to suggest corrections if any of this is wrong 😄

New package manager questionnaire

Did you read our documentation on adding a package manager?

Basics

Name of package manager

nix

What language does this support?

nix

How popular is this package manager?

5.4k stars on the package manager repo 8.5k stars on the default registry repo

Does this language have other (competing?) package managers?

Package File Detection

What type of package files and names does it use?

Nix files have the .nix extension Lockfiles are called flake.lock

What fileMatch pattern(s) should be used?

See above

Is it likely that many users would need to extend this pattern for custom file names?

Is the fileMatch pattern likely to get many "false hits" for files that have nothing to do with package management?

No

Parsing and Extraction

Can package files have "local" links to each other that need to be resolved?

Is there a reason why package files need to be parsed together (in serial) instead of independently?

What format/syntax is the package file in?

Nix files are written in the nix language. Lockfiles are written in JSON

How do you suggest parsing the file?

Does the package file structure distinguish between different "types" of dependencies? e.g. production dependencies, dev dependencies, etc?

List all the sources/syntaxes of dependencies that can be extracted

Describe which types of dependencies above are supported and which will be implemented in future

Versioning

What versioning scheme does the package file(s) use?

Does this versioning scheme support range constraints, e.g. ^1.0.0 or 1.x?

Is this package manager used for applications, libraries, or both? If both, is there a way to tell which is which?

If ranges are supported, are there any cases when Renovate should pin ranges to exact versions if rangeStrategy=auto?

Lookup

Is a new datasource required? Provide details

Will users need the capability to specify a custom host/registry to look up? Can it be found within the package files, or within other files inside the repository, or would it require Renovate configuration?

Do the package files contain any "constraints" on the parent language (e.g. supports only v3.x of Python) or platform (Linux, Windows, etc) that should be used in the lookup procedure?

Will users need the ability to configure language or other constraints using Renovate config?

Artifacts

Are lock files or checksum files used? Are they mandatory?

Used, yes. Mandatory, no.

If so, what tool and exact commands should be used if updating one or more package versions in a dependency file?

Not 100% sure for nix files For flake lockfiles, they can be updated with nix flake update

If applicable, describe how the tool maintains a cache and if it can be controlled via CLI or env? Do you recommend the cache be kept or disabled/ignored?

If applicable, what command should be used to generate a lock file from scratch if you already have a package file? This will be used for "lock file maintenance"

nix flake lock to create nix flake update to update

Other

Is there anything else to know about this package manager?

budimanjojo commented 2 years ago

is there anybody that got this working with regexManager?

JamieMagee commented 2 years ago

flake lockfile updates are a WIP: https://github.com/JamieMagee/renovate-flake/pull/1

rbozan commented 2 years ago

awesome! it'll be hard to implement it all though. so much things can be updated within a single nix file, but even having basic functionality would be awesome.

JamieMagee commented 2 years ago

I'd like to start with two scenarios:

l0b0 commented 2 years ago

One scenario which might be easy to support is JSON lockfiles. Example:

{
  "name": "release-22.05-2022-10-27T10-42-29Z",
  "url": "https://github.com/NixOS/nixpkgs/archive/0a773b4ddf75c41c08a46f5c979f435bbfa6f6cb.tar.gz",
  "sha256": "0vf73j9179h4wxyj9yx8mkxnnngzrva49176raw3fnlrnb94s2sg"
}

Use in Nix file:

{ pkgs ? import
    (
      fetchTarball (
        builtins.fromJSON (
          builtins.readFile ./nixpkgs.json)))
    { }
}

Update script:

#!/usr/bin/env bash

set -o errexit -o nounset -o pipefail
shopt -s failglob inherit_errexit

if [[ $# -ne 1 ]]; then
    cat >&2 <<EOF
bump-nixpkgs.bash: Update nixpkgs.json with latest info from given release

Usage:

./bump-nixpkgs.bash RELEASE

Example:

./bump-nixpkgs.bash 22.05
    Bumps nixpkgs within the 22.05 release.
EOF
    exit 2
fi

release="$1"

cleanup() {
    rm --force --recursive "$working_dir"
}
trap cleanup EXIT
working_dir="$(mktemp --directory)"

release_file="${working_dir}/release.json"
curl "https://api.github.com/repos/NixOS/nixpkgs/git/refs/heads/release-${release}" >"$release_file"
commit_id="$(jq --raw-output .object.sha "$release_file")"
commit_date="$(curl "https://api.github.com/repos/NixOS/nixpkgs/commits/$commit_id" | jq --raw-output '.commit.committer.date' | tr ':' '-')"

partial_file="${working_dir}/nixpkgs-partial.json"
jq --arg commit_date "$commit_date" --raw-output '{name: (.ref | split("/")[-1] + "-" + $commit_date), url: ("https://github.com/NixOS/nixpkgs/archive/" + .object.sha + ".tar.gz")}' "$release_file" >"$partial_file"

archive_checksum="$(nix-prefetch-url --unpack "$(jq --raw-output .url "$partial_file")")"
full_file="${working_dir}/nixpkgs.json"
jq '. + {sha256: $hash}' --arg hash "$archive_checksum" "$partial_file" >"$full_file"

target_file='./nixpkgs.json'
if diff "$full_file" "$target_file"; then
    echo "No change; aborting." >&2
else
    mv "$full_file" "$target_file"
fi
renovate-release commented 1 year ago

:tada: This issue has been resolved in version 34.23.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

JamieMagee commented 1 year ago

Nix is in Renovate, but currently disabled by default. The minimal renovate.json configuration to enable it is:

"nix": {
  "enabled": true
},
"lockFileMaintenance": {
  "enabled": true
}

Please give it a go and leave any feedback here.

AnthonyEnr1quez commented 1 year ago

@JamieMagee Gave this a test run today and have some feedback.

Was hoping to use this as a replacement for https://github.com/DeterminateSystems/update-flake-lock, so all my dep managers are using renovate.

JamieMagee commented 1 year ago

Thanks for testing it out. Nix support is definitely in alpha or beta and it's why I've set it as disabled by default.

xanderio commented 1 year ago

One issue I'm currently facing is updating the lockfile for flake that have internal non public flakes in there inputs. 

Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists. 
warning: could not read HEAD ref from repo at 'ssh://**redacted**@gitlab.[company-internal].de/[namespace]/[repository].git'

We normally fix this issue by configuring git to replace the ssh url with an https url. With something like this:

git config --global --add url.https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.[company-internal].de/.insteadOf "git@gitlab.[company-internal].de:"

But as nix is started in a sidecar container this configuration isn't picked up by git.