search

renovatebot / renovate

Home of the Renovate CLI: Cross-platform Dependency Automation by Mend.io
https://mend.io/renovate
GNU Affero General Public License v3.0
17.29k stars 2.26k forks source link

Support for Crystal & Crystal Shards #6029

Open alexanderadam opened 4 years ago

alexanderadam commented 4 years ago

New package manager questionnaire

Did you read our documentation on adding a package manager?

Basics

Name of package manager

(Crystal) Shards

What language does this support?

Crystal

How popular is this package manager?

It's the only one for Crystal that I'm aware of. I would claim that every Crystal developer is using it.

Does this language have other (competing?) package managers?

Package File Detection

What type of package files and names does it use?

What fileMatch pattern(s) should be used?

A shard.yml for the user defined definitions and a shard.lock for the locked versions. I'm not quite sure whether renovate also looks at the locked version but if it does, it probably could look like this:

{
  "shardfile": {
    "fileMatch": ["^shard\\.yml$"]
  },
  "shardLockfile": {
    "fileMatch": ["^shard\\.lock$"]
  },
}

Is it likely that many users would need to extend this pattern for custom file names?

Is the fileMatch pattern likely to get many "false hits" for files that have nothing to do with package management?

You never know but I think this is very unlikely.

Parsing and Extraction

Can package files have "local" links to each other that need to be resolved?

yes

Is there a reason why package files need to be parsed together (in serial) instead of independently?

I don't know. 🤷

What format/syntax is the package file in?

How do you suggest parsing the file?

Does the package file structure distinguish between different "types" of dependencies? e.g. production dependencies, dev dependencies, etc?

List all the sources/syntaxes of dependencies that can be extracted

I am not quite sure what this means. Everything is documented here.

A definition could look like this:

dependencies:
  openssl:
    github: datanoise/openssl.cr
    branch: master

development_dependencies:
  minitest:
    git: https://github.com/ysbaddaden/minitest.cr.git
    version: "~> 0.1.0"

Describe which types of dependencies above are supported and which will be implemented in future

Could you please elaborate on that?

Versioning

What versioning scheme does the package file(s) use?

Semver broadly used but not required.

Does this versioning scheme support range constraints, e.g. ^1.0.0 or 1.x?

Is this package manager used for applications, libraries, or both? If both, is there a way to tell which is which?

@bcardiff said, that it might be both:

I think the answer is both and not 100% but apps has target and lib might not.

If ranges are supported, are there any cases when Renovate should pin ranges to exact versions if rangeStrategy=auto?

I don't know.

Lookup

Is a new datasource required? Provide details

Will users need the capability to specify a custom host/registry to look up? Can it be found within the package files, or within other files inside the repository, or would it require Renovate configuration?

Sources can be path, git, github, gitlab, bitbucket. So it's either a local path or a git repository that need to be accessible.

Do the package files contain any "constraints" on the parent language (e.g. supports only v3.x of Python) or platform (Linux, Windows, etc) that should be used in the lookup procedure?

I don't think so.

Will users need the ability to configure language or other constraints using Renovate config?

I don't think so.

Artefacts

Are lock files or checksum files used? Are they mandatory?

Yes, they are often used and a lock file called shard.lock but they are not mandatory. Libs usually don't have a lock file.

If so, what tool and exact commands should be used if updating one or more package versions in a dependency file?

$ shards update [<shards>...]

If applicable, describe how the tool maintains a cache and if it can be controlled via CLI or env? Do you recommend the cache be kept or disabled/ignored?

shards uses a cache path which can be set via ENV variable SHARDS_CACHE_PATH.

If applicable, what command should be used to generate a lock file from scratch if you already have a package file? This will be used for "lock file maintenance"

If there is no shard.lock yet, you can use shards install to generate it.

Other

Is there anything else to know about this package manager?

Yes, you should try it. 😉 But seriously, thank you for your work and considering integration of Crystal Shards.

rarkins commented 4 years ago

Datasource: seems like they are git, github, bitbucket or gitlab references, so we shouldn't need any new datasources at least in the initial drop.

Versioning: Looks like it supports a subset of Ruby versioning as per the description in https://github.com/crystal-lang/shards/blob/master/SPEC.md#version

Manager: Seems like a simple YAML parsing extract should be all it takes. No lock files.

rarkins commented 4 years ago

@alexanderadam could you create a public repo that contains an example shard file with the various use cases you'd expect to work? Ideally each of them should be out of date so that it results in some PRs.

alexanderadam commented 4 years ago

Wow, you answer fast! Awesome! :pray:

Seems like a simple YAML parsing extract should be all it takes.

Well, at least dependencies are just written in YAML. I'm not sure what it needs otherwise.

No lock files.

What do you mean? There are lock files.

They are also documented in the shards reference?

could you create a public repo that contains an example shard file with the various use cases you'd expect to work?

Would some of the examples at github be sufficient as well?

What exactly do you mean by "use cases"? I guess @ysbaddaden, @waj, @straight-shoota, @bcardiff and the other contributors could help out in case you're thinking of particular shards features as "use cases".

rarkins commented 4 years ago

OK, I missed the lock files part. Makes it a little more complex but we've done it plenty of times before.

Examples are useful, but to proceed we'd want a hand-crafted repo that deliberately includes all the use cases we intend to support, has each deliberately out-of-date, etc.

alexanderadam commented 4 years ago

to proceed we'd want a hand-crafted repo that deliberately includes all the use cases we intend to support, has each deliberately out-of-date, etc.

Sorry but I'm still struggling what other use cases except out-of-date are relevant. And for out-of-date dependencies, probably any old repo would do, right?

So for instance:

  1. I searched for old lock files
  2. Found & checked out this old geoip repository: izikaj/geoip
  3. Called shards install to install the outdated dependencies and then called shards update to see this:

Peek 2020-04-23 09-56

Afterwards the lockfile should be updated accordingly:

![Screenshot_20200423_095953](https://user-images.githubusercontent.com/372620/80074301-3d8d5200-8549-11ea-89f9-a8cb684fc738.png)

It would really help if you could elaborate what use cases (other than the standard out-of-date one) you need.

Thank you in advance! :pray:

rarkins commented 4 years ago

What I mean is a shard file that deliberately has all the different types of dependencies defined (git, github, gitlab, using an exact version vs ranges, etc). And then a lock file to match of course. It's hard to develop initially against a messy "real" repository instead of one that's been deliberately created to test all use cases.

rarkins commented 4 years ago
  1. Create a new repo
  2. Create a shard file with no dependencies
  3. Add dependencies until you've covered the different types defined in the spec
  4. Try to make sure each dependency is out of date so that we have something to test
alexanderadam commented 4 years ago

Thank you for the explaination.

I created this repository, which has only outdated dependencies. Some of the old dependencies are currently fixed but renovatebot would increase a dependency in the shard.yml automatically, right? Otherwise I would have to update the versions again.

Just tell me, in case I should change anything or in case I misunderstood something.

rarkins commented 4 years ago

At a quick glance, that looks perfect. Renovate will update the yml and then run the lock command to update it accordingly in the same commit.

bcardiff commented 4 years ago

For a start shards has an outdated command. I guess you should try to consume that data. Maybe there needs to be a data friendly format instead of human friendly output.

If there needs to be more integration that the one offered by outdated command I would have in mind that:

rarkins commented 4 years ago

Thanks for the info. We always prefer doing our own version lookups and resolution anyway for flexibility, so we don't need the outdated command output for now.

  • A difference shards has from bundler is that you can have dependencies along the whole tree against branches and not only released versions via tags. This allows for example to depend on fresh shards that are not yet released.

By this do you mean you can depend on any git ref such as branches? Are those unreleased fresh shards ever versioned? Because if they are not versioned, people probably wouldn't want us interfering and trying to upgrade them anyway.

bcardiff commented 4 years ago

By this do you mean you can depend on any git ref such as branches?

Yes

Are those unreleased fresh shards ever versioned?

Maybe. It depends on the author. If the requirement in shard.yml mentions a branch, maybe you want to know if the lock file is outdated regarding the branch. It's a corner case definitely.

There is also no central repository of shards. There is no ruby-gems equivalent.

Usually library shards do not check in the shard.lock file. There are different opinions why/if this should be done. That's probably the reason why you didn't notice a shard.lock in the first place.

waghanza commented 4 years ago

I'm also interested in crystal. I think, I'll try to contribute to this PR.

I used dependabot once, but I'm not sure it the lock files should be parsed @rarkins (I've no lock in the project I need renovatebot https://github.com/the-benchmarker/web-frameworks)

rarkins commented 4 years ago

It's not essential to parse lock files. It's even ok to not have lock file updating capabilities initially, however in that case we'd make crystal disabled by default so that only those without lock files can opt-in to using it. Adding lock file updating afterwards should not be too hard.

caspiano commented 3 years ago

Any chance you need help on this? Happy to throw in some time to get this going.

rarkins commented 3 years ago

@caspiano that would be great!

According to the above, we didn't notice any immediate need for a new "datasource", so that would mean additions are in lib/manager/crystal/* only. I suggest working on the extract function first, so that direct dependencies can be detected and updated. After that, the lock file update can some next.

We have an active slack channel for people developing and wanting to ask questions, so feel free to email me for an invite to that.

github-actions[bot] commented 2 years ago

Hi there,

You're requesting support for a new package manager. We need to know some basic information about this package manager first. Please copy/paste the new package manager questionnaire, and fill it out in full.

Once the questionnaire is filled out we will evaluate if adding support for this manager is something we want to do.

Good luck,

The Renovate team

alexanderadam commented 2 years ago

@rarkins I updated the first post with the questionnaire.