pbrisbin commented 3 years ago

What would you like Renovate to be able to do?

I would like renovate-bot to support Haskell, specially Stackage-based projects using an LTS resolver and extra dependencies from Hackage:

Resolver

An overall configuration point in a Stackage-based Haskell project is the resolver. For example, resolver: lts-major.minor. I would like renovate-bot to find this and suggest an update to (for example) resolver: lts-major.(minor + 1) when it becomes available. There are more resolvers than these LTS versions, but this is our initial use-case.

Hackage

While choosing a resolver defines a strict set of exact versions available, you can make additional dependencies available via extra-deps:

extra-deps:
  - some-package-x.y.z@sha256:sha,size

I would like renovate-bot to compare this to what's available on Hackage and suggest updates accordingly. There are other kinds of dependencies that can be specified here, and the sha256 verification is optional, but this is our initial use-case.

Did you already have any implementation ideas?

Yaml configuration can be declared directly in a file, named stack.yaml by default:

resolver: lts-16.25
extra-deps:
  - bugsnag-haskell-0.0.4.0@sha256:44a48e25058e45c8cc351355f6a601d9b4bc6c9895e61e84b4dcd3366302a26b,7395

(This path can be changed by configuration, so renovate-bot would have to support that)

Or stack.yaml can point to a different file, with slightly different keys:

# stack.yaml
resolver: ./snapshot.yaml

# snapshot.yaml
resolver: lts-16.25
packages:
  - bugsnag-haskell-0.0.4.0@sha256:44a48e25058e45c8cc351355f6a601d9b4bc6c9895e61e84b4dcd3366302a26b,7395

We would need renovate-bot to:

Find stack.yaml[resolver], probably limit things to only work with lts-.* OR <filepath>
If it's pointing to another file, follow that file and read resolver and packages
Otherwise, read the resolver and exra-deps directly in this file
Limit things further, to only Hackage dependencies in packages/extra-deps
Outcome: resolver = lts-16.25, hackage = [bugsnag-haskell, 0.0.4.1]
Look up latest LTS resolver by following and reading https://www.stackage.org/lts?_accept=application/json
Outcome: lts-16.27
Look up latest for each Hackage item, by following and scraping https://hackage.haskell.org/package/:name
Outcome: bugsnag-haskell-0.0.4.1
Open a PR updating to this resolver and any new Hackage dependencies

  # snapshot.yaml
- resolver: lts-16.25
+ resolver: lts-16.27
  packages:
-   - bugsnag-haskell-0.0.4.0@sha256:44a48e25058e45c8cc351355f6a601d9b4bc6c9895e61e84b4dcd3366302a26b,7395
+   - bugsnag-haskell-0.0.4.1@sha256:...,...

A changelog can also be scraped from https://hackage.haskell.org/package/{name}-{version}/changelog, and the https://www.stackage.org/lts?_accept=application/json response includes any changed dependencies, for which changelogs can also be scraped. It would be amazing to include these changelog details in the renovate PR. We have a project that does some of this work here.

I'm willing to try and contribute this support, but I'm having trouble navigating the project enough to know where to add all the moving parts. Would you be able to offer some guidance, given the information above?

rarkins commented 3 years ago

Thanks @pbrisbin for the proposal.

Logically here's how we'd approach an implementation:

Datasources

Is there a need to access new APIs to retrieve versions of packages? In this case it seems like Hackage is needed. It would go into lib/datasource/hackage/*. There's a lot of existing datasources which could be used as inspiration. It looks like it would need to support digests too.

Versioning

Does Haskell use its own syntax to define versions or constraints/ranges? I'm seeing "pinned" versions with digests in your examples so maybe our "loose" versioning would work fine for things like 0.0.4.1.

Manager

The minimum you need to implement is a manager (e.g. lib/manager/stackage?) with and extractPackageFile() function. With a YAML parser, should hopefully be simple.

Is there any separate "artifacts" files that need updating, e.g. checksum or lock files?

Changelog

Currently our changelog only supports github and gitlab sources, but it sounds like they are published to the registry itself for Hackage? Do any packages also publish to a changelog file in their github source?

pbrisbin commented 3 years ago

Does Haskell use its own syntax to define versions or constraints/ranges?

It does, called the PVP. In short, it defines a 4-part version A.B.C.D where

A.B is MAJOR (yes, together), incremented on breaking changes
C is MINOR, incremented on backwards compatible additions
D is PATCH (and is optional), incremented on bugfixes, etc

In case it's helpful, in my own packages, I set A to something like 0 or 1 and forget about it, then I just treat B.C.D as the typical Major.Minor.Patch of SemVer. This simplifies things for me, and the result still follows the PVP rules.

I'm seeing "pinned" versions with digests

Yes, a stack.yaml is effectively a "freeze file". Even the resolver bit is just an alias to a big set of pinned versions of available packages. So you are meant to put complete and explicit versions in it, when specifying your own in extra-deps. It doesn't support a bounds or ~0.4 or anything like that. The Digests are optional, but good practice.

(Note that this is just one way to specify dependencies in a Haskell project, so building this support in the way I describe would only support Haskell projects that choose this tooling. I'm happy to describe other approaches, if you're interested in understanding the broader space.)

Is there any separate "artifacts" files that need updating, e.g. checksum or lock files?

Ah yes, in fact. There is a stack.yaml.lock. I'm not sure how to update it after changing stack.yaml other than having a Haskell (stack) tool-chain available.

published to the registry itself for Hackage? Do any packages also publish to a changelog file in their github source?

Sort of both:

The project will have a CHANGELOG in its source tree [example]
The project defines "extra source file" in its packaging manifest [example]
The registry, upon receiving the package, will look for files with certain names e.g. (CHANGELOG|ChangeLog)(.md)?
That file is linked to from the main registry listing [example)

(Project README's are handled similarly.)

rarkins commented 3 years ago

I think we'll need a new versioning to handle A,B as major. Otherwise, if you use our loose versioning then it would consider changes to C to only be patch. It makes it easier though if you only need to deal with exact versions and not ranges.

If there's a lock file then it means we'd want to support "artifacts" updating like quite a few other managers.

For changelogs, they might "just work" if the majority of packages (a) define their source repo in metadata, and (b) include a changelog file. It doesn't look like we'll need to use the registry's changelogs.

I'd recommend doing this in this order, in separate PRs:

Versioning
Datasource
Extract + Artifacts

Steps 1 & 2 won't really achieve much on their own, but it's easier to do this incrementally than all in one go.

pbrisbin commented 3 years ago

Awesome, thanks. Granular steps make total sense, even if the first few don't actually get us there.

github-actions[bot] commented 2 years ago

Hi there,

You're requesting support for a new package manager. We need to know some basic information about this package manager first. Please copy/paste the new package manager questionnaire, and fill it out in full.

Once the questionnaire is filled out we will evaluate if adding support for this manager is something we want to do.

Good luck,

The Renovate team

pbrisbin commented 2 years ago

NOTE: I'm answering this questionnaire for the feature of specifically stack-related dependencies. If we want to automatically manage version bounds in a non-Stack project, or library making use of such bounds in its *.cabal file, things are more complicated.

New package manager questionnaire

Did you read our documentation on adding a package manager?

[x] I've read the adding a package manager documentation.

Basics

Name of package manager

Stack.

What language does this support?

Haskell.

How popular is this package manager?

Probably >50% of Haskell projects us it.

Does this language have other (competing?) package managers?

[x] Yes (Cabal, Nix, and to a lesser-extent Bazel)
[ ] No

Package File Detection

What type of package files and names does it use?

stack.yaml, or whatever $STACK_YAML holds
Any other yaml path, if referenced from stack.yaml as a "custom snapshot"

What fileMatch pattern(s) should be used?

Can ${STACK_YAML:-stack.yaml} be used? If not, hard-coded stack.yaml should be sufficient -- let the user configure otherwise.

Is it likely that many users would need to extend this pattern for custom file names?

[ ] Yes
[x] No

Possible, but not likely, IMO.

Is the fileMatch pattern likely to get many "false hits" for files that have nothing to do with package management?

No.

Parsing and Extraction

Can package files have "local" links to each other that need to be resolved?

Yes, stack.yaml can optionally use a "custom snapshot" by resolver: ./<other-yaml>.

Is there a reason why package files need to be parsed together (in serial) instead of independently?

I can't think of one.

What format/syntax is the package file in?

[ ] JSON
[ ] TOML
[x] YAML
[ ] Custom (explain below)

How do you suggest parsing the file?

[x] Off the shelf parser
[ ] Using regex
[ ] Custom-parsed line by line
[ ] Other

Does the package file structure distinguish between different "types" of dependencies? e.g. production dependencies, dev dependencies, etc?

[ ] Yes, production and development dependencies
[x] No, all dependencies are treated the same

List all the sources/syntaxes of dependencies that can be extracted

In stack.yaml, resolver: x is either a named (remote) snapshot or path to a custom snapshot. For the case of a named (remote) snapshot, Renovate would not need to dig into specific dependencies but look at x itself and see if it's out of date and update it accordingly. The snapshot is itself the dependency in a way.

In the case of a path to a custom snapshot, you will find in that file a packages: key of the following syntax:

- {name}[-{version}[(@sha256:{sha}]]

- git: {repo}
  commit: {sha}

- ./<path>

stack.yaml (not the custom snapshot) can also contain a key extra-deps, which is a list of the same syntax to overlay over what came from above. These individual items would need to be inspected to be updated based on what they are.

Describe which types of dependencies above are supported and which will be implemented in future

Versioning

What versioning scheme does the package file(s) use?

Snapshots, use two formats:

lts-{Major}.{Minor}
nightly-{YYYY}-{MM}-{DD}

Non-git packages/extra-deps items follow PVP, which is A.B.C.D where:

A.B is major, it must increment on breaking changes
C is minor, incremented on non-breaking additions
D is optional and patch

Does this versioning scheme support range constraints, e.g. `^1.0.0` or `1.x`?

[ ] Supports range constraints (e.g ^1.0.0 or 1.x)
[x] No

Not in this file. Items are specified fully, even down to checksum!

Is this package manager used for applications, libraries, or both? If both, is there a way to tell which is which?

[ ] Applications
[ ] Libraries
[x] Both (explain how to tell which is which)

A library commonly uses version bounds in its project files, for when it's build not by the developers of the project itself and perhaps across a wide range of dependencies. This feature-request is not concerned with such bounds.

An app commonly uses no version bounds in its project file, relying on the fact that it would only be built by the project maintainers itself, in a Stack-managed context, where the set of available dependencies is fixed by the stack.yaml.

I think either of these scenarios can be handled the same by Renovate.

If ranges are supported, are there any cases when Renovate should pin ranges to exact versions if rangeStrategy=auto?

N/A

Lookup

Is a new datasource required? Provide details

[x] Yes, provide details.
[ ] No.

Available resolvers can be found on https://www.stackage.org. There are some JSON endpoints, but parsing HTML may be required at points.

Non-git packages/extra-deps would come from https://hackage.haskell.org. Again, there are some JSON endpoints, parsing HTML may be required, or you could download "the index" which is a gzipped tarball of information about al available dependencies. I would avoid this, personally, as it's a ton of data to pull and parse when you only need information about a subset of packages.

Will users need the capability to specify a custom host/registry to look up? Can it be found within the package files, or within other files inside the repository, or would it require Renovate configuration?

No.

Do the package files contain any "constraints" on the parent language (e.g. supports only v3.x of Python) or platform (Linux, Windows, etc) that should be used in the lookup procedure?

There is such a feature in Haskell, but I think Renovate could ignore it for now.

Will users need the ability to configure language or other constraints using Renovate config?

Not that I can think of.

Artifacts

Are lock files or checksum files used? Are they mandatory?

Yes. Not mandatory, but highly recommended.

If so, what tool and exact commands should be used if updating one or more package versions in a dependency file?

stack build --dry-run will recreate stack.yaml.lock after a change to stack.yaml.

If applicable, describe how the tool maintains a cache and if it can be controlled via CLI or env? Do you recommend the cache be kept or disabled/ignored?

~/.stack
./.stack-work

stack --work-dir x can control the latter, the former is fixed.

If applicable, what command should be used to generate a lock file from scratch if you already have a package file? This will be used for "lock file maintenance"

Same, stack build --dry-run

Other

Is there anything else to know about this package manager?

More details can be found in previous comments on this Issue.

domenkozar commented 10 months ago

Existing implementation: https://github.com/nomeata/haskell-bounds-bump-action

mihaimaruseac commented 2 months ago

Any update here? I was mistakenly thinking that Haskell support was already implemented :(

renovatebot / renovate

Initial Haskell support #8187

Datasources

Versioning

Manager

Changelog

New package manager questionnaire

Basics

Name of package manager

What language does this support?

How popular is this package manager?

Does this language have other (competing?) package managers?

Package File Detection

What type of package files and names does it use?

What fileMatch pattern(s) should be used?

Is it likely that many users would need to extend this pattern for custom file names?

Is the fileMatch pattern likely to get many "false hits" for files that have nothing to do with package management?

Parsing and Extraction

Can package files have "local" links to each other that need to be resolved?

Is there a reason why package files need to be parsed together (in serial) instead of independently?

What format/syntax is the package file in?

How do you suggest parsing the file?

Does the package file structure distinguish between different "types" of dependencies? e.g. production dependencies, dev dependencies, etc?

List all the sources/syntaxes of dependencies that can be extracted

Describe which types of dependencies above are supported and which will be implemented in future

Versioning

What versioning scheme does the package file(s) use?

Does this versioning scheme support range constraints, e.g. ^1.0.0 or 1.x?

Is this package manager used for applications, libraries, or both? If both, is there a way to tell which is which?

If ranges are supported, are there any cases when Renovate should pin ranges to exact versions if rangeStrategy=auto?

Lookup

Is a new datasource required? Provide details

Will users need the capability to specify a custom host/registry to look up? Can it be found within the package files, or within other files inside the repository, or would it require Renovate configuration?

Do the package files contain any "constraints" on the parent language (e.g. supports only v3.x of Python) or platform (Linux, Windows, etc) that should be used in the lookup procedure?

Will users need the ability to configure language or other constraints using Renovate config?

Artifacts

Are lock files or checksum files used? Are they mandatory?

If so, what tool and exact commands should be used if updating one or more package versions in a dependency file?

If applicable, describe how the tool maintains a cache and if it can be controlled via CLI or env? Do you recommend the cache be kept or disabled/ignored?

If applicable, what command should be used to generate a lock file from scratch if you already have a package file? This will be used for "lock file maintenance"

Other

Is there anything else to know about this package manager?

Does this versioning scheme support range constraints, e.g. `^1.0.0` or `1.x`?