kscript environment variable substitution for repositories no longer works

[rocketraman]

In kscript 4.2.2, environment variable substitution for repositories no longer works.

The first problem is that the syntax is no longer {{FOO}} -- since the backend is now kotlin scripting, the syntax is now $FOO. See MavenDependenciesResolver.addRepository.

Once that is fixed, the next problem is that the gradle script that kscript writes and executes for --package contains the literal values $FOO, as createGradleRepositoryCredentials uses the raw values set for user and password rather than the resolved values.

[aartiPl]

Please explain what you mean by "variable substitution for repositories no longer works"? Is it a user code problem or in kscript itself?

[rocketraman]

Please explain what you mean by "variable substitution for repositories no longer works"? Is it a user code problem or in kscript itself?

@aartiPl Substitution of environment variables for the username and password components of a repository is a documented feature of kscript. See https://github.com/kscripting/kscript/blob/1c97c2197ddd08382834c014096ffe8b26727a4f/README.adoc?plain=1#L419-L422.

I've submitted PR https://github.com/kscripting/kscript/pull/403 to illustrate the two issues I have identified. You will note that both test 4 and test 5 currently fail.

Test 4 is at the moment a documentation problem -- the initial implementation of this in kscript used {{FOO}} syntax. But when the backend was changed to kotlin script, the machinery inside kotlin script uses the $FOO syntax. So simply documenting the change is sufficient I think.

Test 5 illustrates an issue with packaging scripts that use custom auth with env vars. I'm not sure this ever worked.

For test 4, the call to the relevant code is https://github.com/kscripting/kscript/blob/1c97c2197ddd08382834c014096ffe8b26727a4f/src/main/kotlin/io/github/kscripting/kscript/resolver/DependencyResolver.kt#L32.

For test 5, the relevant code is https://github.com/kscripting/kscript/blob/1c97c2197ddd08382834c014096ffe8b26727a4f/src/main/kotlin/io/github/kscripting/kscript/code/GradleTemplates.kt#L109-L114.

[rocketraman]

I submitted 3 PRs to resolve this. Please review.

[rocketraman]

The approach I took was to move kscript in a way compatible with Kotlin scripting. An alternate approach would be to keep the syntax as {{FOO}}, resolve these inside kscript, and pass the Kotlin script backend (and Gradle) the resolved vars. This would actually simplify the code a bit, but personally I think kscript moving towards syntactic compatibility with Kotlin Script is a better option, as hopefully the kscript wrapper will get smaller and smaller over time. However, that does make it a breaking change for existing scripts.

[aartiPl]

Ok, I see. Your approach to changing templates from {{}} to '$' makes sense to me. The only drawback I see now is that IntelliJ will try to resolve a local variable when it finds '$'. So in the code, it will be necessary to escape '$' with a backslash, or IntelliJ will complain about the non-existing variable. Have you tried it? How does it work? Should we escape '$' or not?

Regarding the implementation. There is already a code resolving username and password. It is in SectionResolver:

The resolution of the URL, username, and password should happen there. I think that the best algorithm would be (for all three parameters): 1) Check if it starts with the dollar sign, and in such a case, try to replace it with the env variable value 2) If the env variable is unresolved, and the name is one of the KSCRIPTREPOSITORY* names, then we should use those names as they are now (details can still be written in the configuration file)
3) If variables are still not resolved, we should probably throw an exception

Please merge all 3 PRs into only one; it's much easier for me to test and review. Also, please write at least a simple Unit Test.

And thanks for your great work! :+1:

[rocketraman]

Ok, I see. Your approach to changing templates from {{}} to '$' makes sense to me. The only drawback I see now is that IntelliJ will try to resolve a local variable when it finds '$'. So in the code, it will be necessary to escape '$' with a backslash, or IntelliJ will complain about the non-existing variable. Have you tried it? How does it work? Should we escape '$' or not?

Kotlin script appears to require it be unescaped i.e. it reads it as a literal value. Adding the escape bypasses Kotlin script's var resolution. While we will now do the parsing for this before it ever gets to Kotlin Script and could change this behavior, I suggest we stay compatible to the syntax used in Kotlin Script. IDEA will likely be updated at some point to correctly interpret the $ in scripts.

Regarding the implementation. There is already a code resolving username and password. It is in SectionResolver:

Ok.

2. If the env variable is unresolved, and the name is one of the KSCRIPTREPOSITORY* names, then we should use those names as they are now (details can still be written in the configuration file)

One tweak to this which will be more powerful as it allows environment variables to compose from properties, and also easier to implement:

2. After env variable resolution, if the name is one of the KSCRIPTREPOSITORY* names, then we should use those names as they are now

Also, please write at least a simple Unit Test.

Done.

PR submitted for all these changes https://github.com/kscripting/kscript/pull/406.