krzema12 commented 2 years ago

What would you like Renovate to be able to do?

Kotlin supports running stand-alone scripts under JVM with a shebang, e.g.:

#!/usr/bin/env kotlin
@file:DependsOn("it.krzeminski:github-actions-kotlin-dsl:0.22.0")
@file:DependsOn("org.eclipse.jgit:org.eclipse.jgit:4.6.0.201612231935-r")
@file:DependsOn("org.jetbrains.lets-plot:lets-plot-common:2.1.0")
@file:DependsOn("org.jetbrains.lets-plot:lets-plot-kotlin-jvm:3.0.2")

// ...

println("Hello world")

They can be found in .main.kts or .kts files. An extensionless script is also valid, but we could ignore this case for the beginning.

I would love to see support for updating dependencies for it in Renovate.

If you have any ideas on how this should be implemented, please tell us here.

Conceptually, it's similar to Gradle support. We would have to extract the dependencies from @file:DependsOn(...), look up Maven Central by default (one can also define custom repos with @file:Repository(...)).

Implementation-wise, I'm wondering if it should be added to the Gradle module, or rather a separate module should be created, specific to Kotlin Script. Please advise. The dependencies are Maven-compatible, so for sure there's some common part with Gradle and Maven modules.

Is this a feature you are interested in implementing yourself?

Yes

rarkins commented 2 years ago

Is it a documented standard?

github-actions[bot] commented 2 years ago

Hi there,

You're requesting support for a new package manager. We need to know some basic information about this package manager first. Please copy/paste the new package manager questionnaire, and fill it out in full.

Once the questionnaire is filled out we will evaluate if adding support for this manager is something we want to do.

Good luck,

The Renovate team

martinbonnin commented 2 years ago

Is it a documented standard?

Yes, the @DependsOn annotation and Kotlin scripting is documented here The maven coordinates are the same as any other maven project and documented here

PhilipAbed commented 2 years ago

@krzema12 i noticed you are interesting in implementing this on your own, if you need help don't hesitate to ask, also is it ok if i assign the issue to you?

krzema12 commented 2 years ago

@PhilipAbed sure, feel free to assign it to me. Me or someone from the Kotlin community will fill the "new package manager" questionnaire within several days, and once you guys give us green light to start implementing, I'll get back to you to discuss the implementation details.

krzema12 commented 2 years ago

New package manager questionnaire

Did you read our documentation on adding a package manager?

[x] I've read the adding a package manager documentation.

Basics

Name of package manager

Kotlin Script

What language does this support?

Kotlin - stand-alone scripts.

How popular is this package manager?

Hard to get real data, however on Kotlin Slack there are ~650 users on #slack channel (https://kotlinlang.slack.com/archives/C0BT46EL8/). It's increasingly popular, as a way to use Kotlin also for quick, self-contained scripts. I've been using it myself for 2-3 years already. I'm also the co-creator of https://github.com/krzema12/github-actions-kotlin-dsl/ which heavily relies on Kotlin scripts.

Does this language have other (competing?) package managers?

[ ] Yes (give names)
[x] No

The @file:DependsOn mechanism is the only viable way to define dependencies.

Package File Detection

What type of package files and names does it use?

The dependencies sit in the same file as the scripts. The scripts can have .main.kts, .kts or no extension. The first two are the most popular one. Whatever is before the extension depends on the user.

What fileMatch pattern(s) should be used?

Something like:

['^((?!gradle).)*kts$']

Is it likely that many users would need to extend this pattern for custom file names?

[ ] Yes
[x] No

Is the fileMatch pattern likely to get many "false hits" for files that have nothing to do with package management?

No - .kts is very popular in case of Gradle Kotlin DSL (build.gradle.kts, settings.gradle.kts), but the above fileMatch accounts for that and excludes matches that match Gradle files.

Other than that, I don't recall cases where we'd have kts files which we don't want to analyze with Renovate.

Parsing and Extraction

Can package files have "local" links to each other that need to be resolved?

I don't know, I don't fully understand the question. One script can include another one, but we can disregard this case for now.

Is there a reason why package files need to be parsed together (in serial) instead of independently?

I don't know, I don't fully understand the question. I think we can disregard this case for now, too, whatever it means.

What format/syntax is the package file in?

[ ] JSON
[ ] TOML
[ ] YAML
[x] Custom (explain below)

Specialized Kotlin file-wide annotations:

@file:Repository("...")
@file:DependsOn("it.krzeminski:github-actions-kotlin-dsl:0.22.0")

How do you suggest parsing the file?

[ ] Off the shelf parser
[x] Using regex
[ ] Custom-parsed line by line
[ ] Other

Does the package file structure distinguish between different "types" of dependencies? e.g. production dependencies, development dependencies, etc?

[ ] Yes, production and development dependencies
[x] No, all dependencies are treated the same

I'm not aware of any distinction, and the annotation's API doesn't suggest it.

List all the sources/syntaxes of dependencies that can be extracted

@file:DependsOn("maven:artifact:1.0")

Describe which types of dependencies above are supported and which will be implemented in future

Just regular, production dependencies.

Versioning

What versioning scheme does the package file(s) use?

I assume the same scheme that's used for Gradle, or a subset of it.

Does this versioning scheme support range constraints, e.g. `^1.0.0` or `1.x`?

[ ] Supports range constraints (e.g ^1.0.0 or 1.x)
[x] No

For now I'd keep it simple and not support ranges, even if they are somehow supported.

Is this package manager used for applications, libraries, or both? If both, is there a way to tell which is which?

[x] Applications
[ ] Libraries
[ ] Both (explain how to tell which is which)

If ranges are supported, are there any cases when Renovate should pin ranges to exact versions if rangeStrategy=auto?

No.

Lookup

Is a new datasource required? Provide details

[ ] Yes, provide details.
[x] No.

I assume that Maven datasource can be reused.

Will users need the capability to specify a custom host/registry to look up? Can it be found within the package files, or within other files inside the repository, or would it require Renovate configuration?

Yes, @file:Repository("...") annotation allows it. It's placed right next to the list of dependencies.

Do the package files have any "constraints" on the parent language (e.g. supports only v3.x of Python) or platform (Linux, Windows, etc) that should be used in the lookup procedure?

Yes, only Kotlin, and OS-agnostic.

Will users need the ability to configure language or other constraints using Renovate config?

No, I don't see such need.

Artifacts

Are lock files or checksum files used? Are they mandatory?

No.

If so, what tool and exact commands should be used if updating one or more package versions in a dependency file?

N/A.

If applicable, describe how the tool maintains a cache and if it can be controlled via CLI or environment variables? Do you recommend the cache be kept or disabled/ignored?

N/A

If applicable, what command should be used to generate a lock file from scratch if you already have a package file? This will be used for "lock file maintenance"

N/A

Other

Is there anything else to know about this package manager?

Kotlin Script is built to closely mimic Gradle, so during the implementation we'll be able to heavily inspire and reuse stuff from Gradle support.

krzema12 commented 2 years ago

@PhilipAbed please review :arrow_up: :pray:

krzema12 commented 2 years ago

Work in progress: https://github.com/krzema12/renovate/commits/add-kotlin-script-support

krzema12 commented 2 years ago

@PhilipAbed I need some help with running Renovate from source. I'm trying with just yarn start, and getting the error You must configure a GitHub token as expected. However I'm struggling to set this token. I tried setting this flag: https://docs.renovatebot.com/self-hosted-configuration/#githubtokenwarn either by env var (RENOVATE_GITHUB_TOKEN_WARN=false yarn start) or CLI option (yarn start --github-token-warn=false) but it doesn't work.

My goal is to run Renovate in some kind of dry-run mode, so that I can iterate on the code conveniently. I've just added some draft-like support for Kotlin Script and now want to check if Renovate detects the dependencies in .kts files and what PRs it would create.

PhilipAbed commented 2 years ago

@krzema12 create a config.js file with the token in your renovate repo or put it in your environment variable see https://docs.renovatebot.com/getting-started/running/#global-config

you can set your github Personal Access Token in there. ( in case you dont know how to create it, ask google )

module.exports = {
  token: 'github personal access token',
  platform: 'github',
  includeForks: true,
  repositories: [
    'orgName, repoName',
  ],
  dryRun: 'full',
  gitAuthor: 'yourName and email ',
};

change the repositories to your repository that u are scanning change the token to your token

then you can run yarn start and it will work ( or yarn debug if you are in an IDE debugging)

krzema12 commented 2 years ago

@PhilipAbed thanks a lot, it works! Just one fix for your hint: it should be config.js - .ts doesn't work.

PhilipAbed commented 2 years ago

@krzema12 ops, typo :D

PhilipAbed commented 2 years ago

Note that dryRun: 'full', option will not create any PRs in your repo, but will add to logs only, if you want to see pull requests getting updated remove it from global config

krzema12 commented 2 years ago

I have something working here: https://github.com/krzema12/renovate/commit/dc55bb4ac03eb2d0984bb032062f97102c2a8754 (example PR: https://github.com/krzema12/test-repo/pull/6). Before I send a PR, I'll add some documentation.

@PhilipAbed I'm struggling with crafting a proper fileMatch regex. For now I set it to fileMatch: ['^.*\\.main\\.kts$'] and it will work fine for Kotlin Script files that indeed end with .main.kts. However, one can also have files with just the extension .kts. The problem is that there are also Gradle config files named build.gradle.kts and settings.gradle.kts and if we changed the regex to be like ^.*\\.kts, we would have a false match to these Gradle files.

I have some ideas how to approach it:

Short-term: assume that Renovate supports only .main.kts files, with a chance to support .kts in the future.
Have a fileMatch that does match the Gradle files, but ignore them inside extraction logic (return null, from what I see that's the API, but I don't know how to get the file name yet).

Try to craft a proper regex so that it ignores Gradle configs. I tried to do so, but "re2" library used for regex doesn't like the negative lookahead sequence that works in e.g. pure JavaScript: (?!. If you have an idea how to create a proper regex that would fulfill such requirements, it would be cool

build.gradle.kts                # NO match
settings.gradle.kts             # NO match
some-dir/build.gradle.kts       # NO match
some-dir/settings.gradle.kts    # NO match
some-script.kts                 # match
some-script.main.kts            # match
some-dir/some-script.kts        # match
some-dir/some-script.main.kts   # match

Please advise :smile:

I created https://github.com/renovatebot/renovate/pull/16684 with approach No. 1 in mind. we can always expand support iteratively if there's high demand.

PhilipAbed commented 2 years ago

something like this? https://regex101.com/r/r64xfq/1

^(?!.*gradle).*\\.kts$

if you have a longer list of things that could get in your way, you probable should exclude them afterwards, make an exclusion list i guess in the extraction functions xD

krzema12 commented 2 years ago

@PhilipAbed I had exactly this regex previously, but it fails in runtime:

$ yarn start
yarn run v1.22.17
$ run-s generate:* 
$ node tools/generate-imports.mjs
generating imports
> data/debian-distro-info.json
> data/node-js-schedule.json
> data/ubuntu-distro-info.json
> node_modules/emojibase-data/en/shortcodes/github.json
generating hashes
$ node -r ts-node/register/transpile-only -- lib/renovate.ts
 INFO: Repository started (repository=krzema12/test-repo)
       "renovateVersion": "0.0.0-semantic-release"
 INFO: Repository has invalid config (repository=krzema12/test-repo)
       "error": {
         "validationSource": "^(?!.*gradle).*\\.kts$",
         "validationError": "Invalid regular expression: ^(?!.*gradle).*\\.kts$",
         "message": "config-validation",
         "stack": "Error: config-validation\n    at regEx (/home/piotr/repos/priv/renovate/lib/util/regex.ts:41:19)\n    at getMatchingFiles (/home/piotr/repos/priv/renovate/lib/workers/repository/extrac
t/file-match.ts:60:21)\n    at tryConfig (/home/piotr/repos/priv/renovate/lib/workers/repository/extract/index.ts:27:46)\n    at extractAllDependencies (/home/piotr/repos/priv/renovate/lib/workers/reposi
tory/extract/index.ts:41:7)\n    at async extract (/home/piotr/repos/priv/renovate/lib/workers/repository/process/extract-update.ts:95:20)\n    at async extractDependencies (/home/piotr/repos/priv/renova
te/lib/workers/repository/process/index.ts:106:26)\n    at async Object.renovateRepository (/home/piotr/repos/priv/renovate/lib/workers/repository/index.ts:42:52)\n    at async Object.start (/home/piotr/
repos/priv/renovate/lib/workers/global/index.ts:154:7)\n    at async /home/piotr/repos/priv/renovate/lib/renovate.ts:16:22"
       }
 INFO: Issue created (repository=krzema12/test-repo)
 WARN: Config Warning (repository=krzema12/test-repo)
       "configError": {
         "validationSource": "^(?!.*gradle).*\\.kts$",
         "validationError": "Invalid regular expression: ^(?!.*gradle).*\\.kts$",
         "message": "config-validation",
         "stack": "Error: config-validation\n    at regEx (/home/piotr/repos/priv/renovate/lib/util/regex.ts:41:19)\n    at getMatchingFiles (/home/piotr/repos/priv/renovate/lib/workers/repository/extrac
t/file-match.ts:60:21)\n    at tryConfig (/home/piotr/repos/priv/renovate/lib/workers/repository/extract/index.ts:27:46)\n    at extractAllDependencies (/home/piotr/repos/priv/renovate/lib/workers/reposi
tory/extract/index.ts:41:7)\n    at async extract (/home/piotr/repos/priv/renovate/lib/workers/repository/process/extract-update.ts:95:20)\n    at async extractDependencies (/home/piotr/repos/priv/renova
te/lib/workers/repository/process/index.ts:106:26)\n    at async Object.renovateRepository (/home/piotr/repos/priv/renovate/lib/workers/repository/index.ts:42:52)\n    at async Object.start (/home/piotr/
repos/priv/renovate/lib/workers/global/index.ts:154:7)\n    at async /home/piotr/repos/priv/renovate/lib/renovate.ts:16:22"
       },
       "res": "created"
 INFO: Repository finished (repository=krzema12/test-repo)
       "durationMs": 4820
Done in 12.14s.

In unit tests, I could see some more details:

SyntaxError: invalid perl operator: (?!
    at regEx (/home/piotr/repos/priv/renovate/lib/util/regex.ts:35:22)
    at /home/piotr/repos/priv/renovate/lib/modules/manager/kotlin-script/index.spec.ts:7:12
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/home/piotr/repos/priv/renovate/lib/modules/manager/kotlin-script/index.spec.ts:6:29)
    at Promise.then.completed (/home/piotr/repos/priv/renovate/node_modules/jest-circus/build/utils.js:333:28)
    at new Promise (<anonymous>)
    at callAsyncCircusFn (/home/piotr/repos/priv/renovate/node_modules/jest-circus/build/utils.js:259:10)
    at _callCircusTest (/home/piotr/repos/priv/renovate/node_modules/jest-circus/build/run.js:277:40)
    at async _runTest (/home/piotr/repos/priv/renovate/node_modules/jest-circus/build/run.js:209:3)
    at async _runTestsForDescribeBlock (/home/piotr/repos/priv/renovate/node_modules/jest-circus/build/run.js:97:9)

See https://github.com/google/re2/issues/288

PhilipAbed commented 2 years ago

if you look at the documentation you might get some help see: https://github.com/renovatebot/renovate/blob/main/docs/development/adding-a-package-manager.md

read about this :extractAllPackageFiles

inside it you can exclude any file name you want

krzema12 commented 2 years ago

Oh, now I noticed that extractPackageFile optionally accepts a path to file. I'm going to use this fact.

krzema12 commented 2 years ago

Ready to review: https://github.com/renovatebot/renovate/pull/16684

renovate-release commented 2 years ago

:tada: This issue has been resolved in version 32.131.0 :tada:

The release is available on:

GitHub release
32.131.0

Your semantic-release bot :package::rocket:

renovatebot / renovate

Add support for Kotlin Script #16662

What would you like Renovate to be able to do?

If you have any ideas on how this should be implemented, please tell us here.

Is this a feature you are interested in implementing yourself?

New package manager questionnaire

Basics

Name of package manager

What language does this support?

How popular is this package manager?

Does this language have other (competing?) package managers?

Package File Detection

What type of package files and names does it use?

What fileMatch pattern(s) should be used?

Is it likely that many users would need to extend this pattern for custom file names?

Is the fileMatch pattern likely to get many "false hits" for files that have nothing to do with package management?

Parsing and Extraction

Can package files have "local" links to each other that need to be resolved?

Is there a reason why package files need to be parsed together (in serial) instead of independently?

What format/syntax is the package file in?

How do you suggest parsing the file?

Does the package file structure distinguish between different "types" of dependencies? e.g. production dependencies, development dependencies, etc?

List all the sources/syntaxes of dependencies that can be extracted

Describe which types of dependencies above are supported and which will be implemented in future

Versioning

What versioning scheme does the package file(s) use?

Does this versioning scheme support range constraints, e.g. ^1.0.0 or 1.x?

Is this package manager used for applications, libraries, or both? If both, is there a way to tell which is which?

If ranges are supported, are there any cases when Renovate should pin ranges to exact versions if rangeStrategy=auto?

Lookup

Is a new datasource required? Provide details

Will users need the capability to specify a custom host/registry to look up? Can it be found within the package files, or within other files inside the repository, or would it require Renovate configuration?

Do the package files have any "constraints" on the parent language (e.g. supports only v3.x of Python) or platform (Linux, Windows, etc) that should be used in the lookup procedure?

Will users need the ability to configure language or other constraints using Renovate config?

Artifacts

Are lock files or checksum files used? Are they mandatory?

If so, what tool and exact commands should be used if updating one or more package versions in a dependency file?

If applicable, describe how the tool maintains a cache and if it can be controlled via CLI or environment variables? Do you recommend the cache be kept or disabled/ignored?

If applicable, what command should be used to generate a lock file from scratch if you already have a package file? This will be used for "lock file maintenance"

Other

Is there anything else to know about this package manager?

Does this versioning scheme support range constraints, e.g. `^1.0.0` or `1.x`?