search

adorsys / keycloak-config-cli

Import YAML/JSON-formatted configuration files into Keycloak - Configuration as Code for Keycloak.
Apache License 2.0
714 stars 134 forks source link

import.path starting with ../ fail to load files on unix #671

Closed ianwallen closed 2 years ago

ianwallen commented 2 years ago

Current Behavior

When I run the program with --import.path=../realms

It will not load the files in the realms folder as it identifies ".." as being a hidden folder.

This was working in previous release (v4.7.0).

Expected Behavior

I expect it to load the files within folder specified.

Steps To Reproduce

My env has the following structure
./bin/   script to run keycloak-config-cli.jar
./lib/   contains jar file
./realm  contains 1 test.yaml to be loaded

The command in my script that fails is the following. It is executed from the bin folder so it references ../realm to get the import files.

java -jar ../lib/keycloak-config-cli.jar \
    --spring.config.import=configtree:/mnt/secrets-store/realms/ \
    --logging.level.keycloak-config-cli=debug \
    --keycloak.url=http://localhost:8080/auth  \
    --keycloak.ssl-verify=true \
    --keycloak.user=admin \
    --keycloak.password=admin \
    --import.path=../realm \
    --import.validate=true \
    --import.var-substitution=true \
    --import.var-substitution-undefined-throws-exceptions=true \
    --keycloak.availability-check.enabled=true \
    --keycloak.availability-check.timeout=120s

The output logs contains the following

2022-03-15 18:31:45.527  INFO 525 --- [           main] d.a.k.config.KeycloakConfigApplication   : No active profile set, falling back to 1 default profile: "default"
2022-03-15 18:31:47.862  INFO 525 --- [           main] d.a.k.config.KeycloakConfigApplication   : Started KeycloakConfigApplication in 4.66 seconds (JVM running for 6.455)
2022-03-15 18:31:50.329  INFO 525 --- [           main] d.a.k.c.provider.KeycloakImportProvider  : 0 configuration files found.
2022-03-15 18:31:50.335  INFO 525 --- [           main] d.a.k.config.KeycloakConfigRunner        : keycloak-config-cli running in 00:00.009.

"0 configuration files found." is not expected.

Environment

Anything else?

This seem to be a UNIX specific issue. I'm not able to reproduce on Windows.

I tried to create a unit test starting with "../" but I was not able so I tried with ".." in the folder however that did not seem to fail. I had tried with the following PR and then abandoned the request. https://github.com/adorsys/keycloak-config-cli/pull/670

On my end the following commit fixed the issue. https://github.com/ianwallen/keycloak-config-cli/commit/5b5648844c4b204c46b589cb640c01d9f6a6a1c7 But I don't find this fix to be very pretty so hopefully you can perform a better fix.

jkroepke commented 2 years ago

Hi,

This seem to be a UNIX specific issue. I'm not able to reproduce on Windows.

this is true. if a file is starting with a dot, in unix its a hidden file. On windows, a separate dos attribute is resposable to mark a file as hidden.

jkroepke commented 2 years ago

It looks like a Java 8 specific issue here: https://github.com/adorsys/keycloak-config-cli/pull/674

ianwallen commented 2 years ago

It is odd that this is being reported as a Java 8 issue. Maybe some java 11 versions are also affected? This is the java version that I'm using

$ java --version
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
openjdk 11.0.13 2021-10-19
OpenJDK Runtime Environment 18.9 (build 11.0.13+8)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8, mixed mode, sharing)

We are using the following image in docker FROM openjdk:11.0.13-bullseye

jkroepke commented 2 years ago

Hi @ianwallen,

I was able to reproduce the issue local. It appends, if the working directory is different that the jar and/or the current workdir is not a base path of the import.path.

A workaround for your case could the glob pattern: --import.path=../realm/* Using this way, I was not able to reproduce the issue anymore.

Since we currently have 2 different implementation to scrape a directory, I'm going the deprecate DirectoryImporter, since the glob based importer have the same functionality.

ianwallen commented 2 years ago

Using the v.4.8.1 release I tried the following 2 options --import.path=../realms/* --import.path=${PWD}/../realms And got message like the following 2022-03-17 21:53:57.430 INFO 1120 --- [ main] d.a.k.c.provider.KeycloakImportProvider : 0 configuration files found.

Tried --import.path=../realms/*.yaml and got the following error

2022-03-17T22:11:37.296904800Z 2022-03-17 22:11:37.295 ERROR 1112 --- [           main] o.s.boot.SpringApplication               : Application run failed
2022-03-17T22:11:37.296952800Z 
2022-03-17T22:11:37.296961700Z java.lang.IllegalStateException: Failed to execute CommandLineRunner
2022-03-17T22:11:37.296966500Z  at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:780)
2022-03-17T22:11:37.296970400Z  at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:761)
2022-03-17T22:11:37.296974700Z  at org.springframework.boot.SpringApplication.run(SpringApplication.java:310)
2022-03-17T22:11:37.296978400Z  at org.springframework.boot.SpringApplication.run(SpringApplication.java:1312)
2022-03-17T22:11:37.296982000Z  at org.springframework.boot.SpringApplication.run(SpringApplication.java:1301)
2022-03-17T22:11:37.296987600Z  at de.adorsys.keycloak.config.KeycloakConfigApplication.main(KeycloakConfigApplication.java:35)
2022-03-17T22:11:37.296992700Z  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2022-03-17T22:11:37.296996300Z  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2022-03-17T22:11:37.296999900Z  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2022-03-17T22:11:37.297017100Z  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
2022-03-17T22:11:37.297021600Z  at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
2022-03-17T22:11:37.297025700Z  at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
2022-03-17T22:11:37.297029700Z  at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
2022-03-17T22:11:37.297033100Z  at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467)
2022-03-17T22:11:37.297036900Z Caused by: de.adorsys.keycloak.config.exception.InvalidImportException: No resource extractor found to handle config property import.path=file:../realms/*.yaml! Check your settings.
2022-03-17T22:11:37.297041000Z  at de.adorsys.keycloak.config.provider.KeycloakImportProvider.readFromPaths(KeycloakImportProvider.java:153)
2022-03-17T22:11:37.297045300Z  at de.adorsys.keycloak.config.provider.KeycloakImportProvider.get(KeycloakImportProvider.java:102)
2022-03-17T22:11:37.297049500Z  at de.adorsys.keycloak.config.KeycloakConfigRunner.run(KeycloakConfigRunner.java:66)
2022-03-17T22:11:37.297053300Z  at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:777)
2022-03-17T22:11:37.297057200Z  ... 13 common frames omitted

Tried the following and it worked since it is supplying the full path without ".." in the path. - This is my current workaround --import.path=$(realpath ../realms)

jkroepke commented 2 years ago

Hi @ianwallen

are you interest to look into #677? The change contains some breaking change (all import related properties has been changed/docmented in the CHANGELOG.md). I also rewrite the import pipeline, i hope it also fixes this issue for now.

ianwallen commented 2 years ago

With your v5 branch it still did not work. But if I changed it to use $(realpath ../realms/*) then it found the file.

$ java -jar ./keycloak-config-cli.jar \
>     --logging.level.keycloak-config-cli=debug \
>     --keycloak.url=http://localhost:8080/auth  \
>     --keycloak.user=admin `# Future change to --keycloak.client-id` \
>     --keycloak.password=admin `# Future change to --keycloak.client-secret` \
>     --import.files.locations=../realms/* \
>
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
2022-03-20 12:05:42.734  INFO 1184 --- [           main] d.a.k.config.KeycloakConfigApplication   : Starting KeycloakConfigApplication v4.8.2-SNAPSHOT using Java 11.0.13 on 6f82390e414d with PID 1184 (/opt/jboss/keycloak-config-cli/bin/keycloak-config-cli.jar started by appuser in /opt/jboss/keycloak-config-cli/bin)
2022-03-20 12:05:42.737  INFO 1184 --- [           main] d.a.k.config.KeycloakConfigApplication   : No active profile set, falling back to 1 default profile: "default"
2022-03-20 12:05:43.299  INFO 1184 --- [           main] d.a.k.config.KeycloakConfigApplication   : Started KeycloakConfigApplication in 1.035 seconds (JVM running for 1.414)
2022-03-20 12:05:44.048 ERROR 1184 --- [           main] d.a.k.config.KeycloakConfigRunner        : No files matching '../realms/*'!
2022-03-20 12:05:44.048  INFO 1184 --- [           main] d.a.k.config.KeycloakConfigRunner        : keycloak-config-cli running in 00:00.010.
2022-03-20 12:05:44.055  INFO 1184 --- [           main] ConditionEvaluationReportLoggingListener :

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2022-03-20 12:05:44.081 ERROR 1184 --- [           main] o.s.boot.SpringApplication               : Application run failed

java.lang.IllegalStateException: Failed to execute CommandLineRunner
        at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:780)
        at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:761)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:310)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1312)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1301)
        at de.adorsys.keycloak.config.KeycloakConfigApplication.main(KeycloakConfigApplication.java:35)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467)
Caused by: de.adorsys.keycloak.config.exception.InvalidImportException: No files matching '../realms/*'!
        at de.adorsys.keycloak.config.provider.KeycloakImportProvider.readFromLocations(KeycloakImportProvider.java:116)
        at de.adorsys.keycloak.config.KeycloakConfigRunner.run(KeycloakConfigRunner.java:71)
        at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:777)
        ... 13 common frames omitted
ianwallen commented 2 years ago

@jkroepke, Using your V5 branch I was able to fix the issue by using the getCanonicalFile() function as shown in this commit.

https://github.com/ianwallen/keycloak-config-cli/commit/42c95083e39bf9e340354b5b4248eedeef023a90

I did not check to see if this will cause other issues.

jkroepke commented 2 years ago

I guess getCanonicalFile has some unexpected behaviors like resolve symlinks, e.g.

I'm going to exclude . and .. from the hidden check.

ianwallen commented 2 years ago

Not sure if the following would help?

https://stackoverflow.com/questions/45938856/how-can-i-determine-the-canonical-path-of-a-file-without-following-symbolic-link

jkroepke commented 2 years ago

Could you test .toAbsolutePath().normalize() ?

My problem is.. each time I said it works for me, you says.. it wont work ...

ianwallen commented 2 years ago

The following commit worked for me

https://github.com/ianwallen/keycloak-config-cli/commit/b22e106492b5aa8d75ff215d51605762e7ecbb03

jkroepke commented 2 years ago

It throws some NullPointerExceptions on some tests, I need to surrend it with a try/catch bloch

ianwallen commented 2 years ago

I tested with the latest v5 branch and it seems to be working as expected. So you can probably close this issue once PR #677 is merged

Thank you