search

cryostatio / cryostat

Secure JDK Flight Recorder management for containerized JVMs
https://cryostat.io
Other
8 stars 8 forks source link

test(tls): set up TLS in oauth-proxy config #426

Closed mwangggg closed 4 days ago

mwangggg commented 2 months ago

Welcome to Cryostat3! 👋

Before contributing, make sure you have:

To recreate commits with GPG signature git fetch upstream && git rebase --force --gpg-sign upstream/main

Related to: https://github.com/cryostatio/cryostat-agent/issues/141 depends on: https://github.com/cryostatio/cryostat-agent/pull/257

github-actions[bot] commented 2 months ago

Build Error! No Linked Issue found. Please link an issue or mention it in the body using #

github-actions[bot] commented 2 months ago

Build Error! No Linked Issue found. Please link an issue or mention it in the body using #

github-actions[bot] commented 2 months ago

Build Error! No Linked Issue found. Please link an issue or mention it in the body using #

github-actions[bot] commented 2 months ago

Build Error! No Linked Issue found. Please link an issue or mention it in the body using #

github-actions[bot] commented 2 months ago

Build Error! No Linked Issue found. Please link an issue or mention it in the body using #

github-actions[bot] commented 2 months ago

Build Error! No Linked Issue found. Please link an issue or mention it in the body using #

github-actions[bot] commented 2 months ago

Build Error! No Linked Issue found. Please link an issue or mention it in the body using #

github-actions[bot] commented 2 months ago

Build Error! No Linked Issue found. Please link an issue or mention it in the body using #

github-actions[bot] commented 2 months ago

Build Error! No Linked Issue found. Please link an issue or mention it in the body using #

github-actions[bot] commented 2 months ago

Build Error! No Linked Issue found. Please link an issue or mention it in the body using #

github-actions[bot] commented 2 months ago

Build Error! No Linked Issue found. Please link an issue or mention it in the body using #

andrewazores commented 2 months ago

Change looks good, just needs a rebase. It depends on https://github.com/cryostatio/cryostat-agent/pull/257 as well, right?

mwangggg commented 2 months ago

yes - I have too many tabs open and accidentally linked the wrong PR

mwangggg commented 2 months ago

/build_test

github-actions[bot] commented 2 months ago

Workflow started at 5/3/2024, 11:37:04 AM. View Actions Run.

github-actions[bot] commented 2 months ago

No OpenAPI schema changes detected.

github-actions[bot] commented 2 months ago

No GraphQL schema changes detected.

github-actions[bot] commented 2 months ago

CI build and push: All tests pass ✅ (JDK17) https://github.com/cryostatio/cryostat3/actions/runs/8941414851

andrewazores commented 2 months ago

I'm not sure if I am missing something or if the PR is missing something. I tried the following steps:

  1. Check out related Agent PR, mvn install
  2. Rebuild quarkus-test to use the Agent from above
  3. Check out and build this PR
  4. sh certs/generate-dev-certs.sh generate, sh compose/agent_certs/generate.sh, and sh compose/auth_certs/generate.sh
  5. ./smoktest.bash -Ot

Everything initially comes up, but after a few moments it exits. Relevant looking logs:

$ podman logs -f compose_quarkus-test-agent_1 
Starting the Java application using /opt/jboss/container/java/run/run-java.sh ...
INFO exec -a "java" java -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager -javaagent:/deployments/app/cryostat-agent.jar -cp "." -jar /deployments/quarkus-run.jar 
INFO running in /deployments
2024-05-03 17:51:33:579 +0000 [cryostat-agent-main] INFO io.cryostat.agent.Agent - Cryostat Agent starting...
2024-05-03 17:51:33:580 +0000 [cryostat-agent-main] INFO io.cryostat.agent.Agent - Cryostat Agent starting...
2024-05-03 17:51:33:653 +0000 [cryostat-agent-main] ERROR io.cryostat.agent.ConfigModule - Failed to resolve host
java.net.UnknownHostException: auth: Name or service not known
    at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934)
    at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543)
    at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
    at java.base/java.net.InetAddress.getByName(InetAddress.java:1256)
    at io.cryostat.agent.ConfigModule$URIRange.check(ConfigModule.java:493)
    at io.cryostat.agent.ConfigModule$URIRange.lambda$static$1(ConfigModule.java:464)
    at io.cryostat.agent.ConfigModule$URIRange.test(ConfigModule.java:501)
    at io.cryostat.agent.ConfigModule$URIRange.validate(ConfigModule.java:511)
    at io.cryostat.agent.Agent.accept(Agent.java:218)
    at io.cryostat.agent.Agent.lambda$agentmain$0(Agent.java:159)
    at java.base/java.lang.Thread.run(Thread.java:840)
2024-05-03 17:51:33:654 +0000 [cryostat-agent-main] ERROR io.cryostat.agent.ConfigModule - Failed to resolve host
java.net.UnknownHostException: auth: Name or service not known
    at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934)
    at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543)
    at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
    at java.base/java.net.InetAddress.getByName(InetAddress.java:1256)
    at io.cryostat.agent.ConfigModule$URIRange.check(ConfigModule.java:493)
    at io.cryostat.agent.ConfigModule$URIRange.lambda$static$3(ConfigModule.java:467)
    at io.cryostat.agent.ConfigModule$URIRange.test(ConfigModule.java:501)
    at io.cryostat.agent.ConfigModule$URIRange.validate(ConfigModule.java:511)
    at io.cryostat.agent.Agent.accept(Agent.java:218)
    at io.cryostat.agent.Agent.lambda$agentmain$0(Agent.java:159)
    at java.base/java.lang.Thread.run(Thread.java:840)
2024-05-03 17:51:33:655 +0000 [cryostat-agent-main] ERROR io.cryostat.agent.ConfigModule - Failed to resolve host
java.net.UnknownHostException: auth: Name or service not known
    at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934)
    at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543)
    at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
    at java.base/java.net.InetAddress.getByName(InetAddress.java:1256)
    at io.cryostat.agent.ConfigModule$URIRange.check(ConfigModule.java:493)
    at io.cryostat.agent.ConfigModule$URIRange.lambda$static$5(ConfigModule.java:473)
    at io.cryostat.agent.ConfigModule$URIRange.test(ConfigModule.java:501)
    at io.cryostat.agent.ConfigModule$URIRange.validate(ConfigModule.java:511)
    at io.cryostat.agent.Agent.accept(Agent.java:218)
    at io.cryostat.agent.Agent.lambda$agentmain$0(Agent.java:159)
    at java.base/java.lang.Thread.run(Thread.java:840)
2024-05-03 17:51:33:784 +0000 [cryostat-agent-main] INFO io.cryostat.agent.CryostatClient - Using Cryostat baseuri https://auth:8443/
2024-05-03 17:51:33:787 +0000 [cryostat-agent-main] ERROR io.cryostat.agent.Agent - Agent startup failure
java.lang.RuntimeException: java.io.FileNotFoundException: /certs/keystore.pass (No such file or directory)
    at io.cryostat.agent.MainModule.provideServerSslContext(MainModule.java:248)
    at io.cryostat.agent.MainModule_ProvideServerSslContextFactory.provideServerSslContext(MainModule_ProvideServerSslContextFactory.java:76)
    at io.cryostat.agent.MainModule_ProvideServerSslContextFactory.get(MainModule_ProvideServerSslContextFactory.java:61)
    at io.cryostat.agent.MainModule_ProvideServerSslContextFactory.get(MainModule_ProvideServerSslContextFactory.java:13)
    at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
    at io.cryostat.agent.MainModule_ProvideHttpServerFactory.get(MainModule_ProvideHttpServerFactory.java:49)
    at io.cryostat.agent.MainModule_ProvideHttpServerFactory.get(MainModule_ProvideHttpServerFactory.java:15)
    at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
    at io.cryostat.agent.MainModule_ProvideWebServerFactory.get(MainModule_ProvideWebServerFactory.java:70)
    at io.cryostat.agent.MainModule_ProvideWebServerFactory.get(MainModule_ProvideWebServerFactory.java:19)
    at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
    at dagger.internal.DelegateFactory.get(DelegateFactory.java:35)
    at io.cryostat.agent.MainModule_ProvideRegistrationFactory.get(MainModule_ProvideRegistrationFactory.java:75)
    at io.cryostat.agent.MainModule_ProvideRegistrationFactory.get(MainModule_ProvideRegistrationFactory.java:13)
    at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
    at io.cryostat.agent.DaggerAgent_Client$ClientImpl.registration(DaggerAgent_Client.java:288)
    at io.cryostat.agent.Agent.accept(Agent.java:226)
    at io.cryostat.agent.Agent.lambda$agentmain$0(Agent.java:159)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.FileNotFoundException: /certs/keystore.pass (No such file or directory)
    at java.base/java.io.FileInputStream.open0(Native Method)
    at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:111)
    at io.cryostat.agent.MainModule.provideServerSslContext(MainModule.java:205)
    ... 18 more
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2024-05-03 17:51:34,024 INFO  [io.quarkus] (main) quarkus-test 1.0.0-SNAPSHOT on JVM (powered by Quarkus 2.7.2.Final) started in 0.444s. Listening on: http://0.0.0.0:10010
2024-05-03 17:51:34,025 INFO  [io.quarkus] (main) Profile prod activated. 
2024-05-03 17:51:34,025 INFO  [io.quarkus] (main) Installed features: [cdi, rest-client, rest-client-jackson, resteasy, smallrye-context-propagation, vertx]
2024-05-03 17:52:18,748 INFO  [io.qua.ver.htt.run.fil.GracefulShutdownFilter] (Shutdown thread) Waiting for HTTP requests to complete
2024-05-03 17:52:23,748 ERROR [io.qua.run.shu.ShutdownRecorder] (Shutdown thread) Timed out waiting for graceful shutdown, shutting down anyway.
2024-05-03 17:52:23,761 INFO  [io.quarkus] (Shutdown thread) quarkus-test stopped in 5.013s

and

$ podman logs -f compose_auth_1 
[2024/05/03 17:52:17] [main.go:71] WARNING: You are using alpha configuration. The structure in this configuration file may change without notice. You MUST remove conflicting options from your existing configuration.
[2024/05/03 17:52:17] [oauthproxy.go:127] using htpasswd file: /tmp/auth_proxy_htpasswd
[2024/05/03 17:52:17] [watcher.go:40] watching '/tmp/auth_proxy_htpasswd' for updates
[2024/05/03 17:52:17] [proxy.go:89] mapping path "^/storage/(.*)$" => upstream "http://s3:8333"
[2024/05/03 17:52:17] [proxy.go:89] mapping path "/grafana/" => upstream "http://grafana:3000"
[2024/05/03 17:52:17] [proxy.go:89] mapping path "/" => upstream "http://cryostat:8181"
[2024/05/03 17:52:17] [oauthproxy.go:171] OAuthProxy configured for Google Client ID: CLIENT_ID
[2024/05/03 17:52:17] [oauthproxy.go:177] Cookie settings: name:_oauth2_proxy secure(https):true httponly:true expiry:168h0m0s domains: path:/ samesite: refresh:disabled
[2024/05/03 17:52:17] [main.go:58] ERROR: Failed to initialise OAuth2 Proxy: error setting up server: could not build app server: error setting up TLS listener: could not load certificate: could not load key data: open /certs/private.key: permission denied
[2024/05/03 17:52:17] [main.go:71] WARNING: You are using alpha configuration. The structure in this configuration file may change without notice. You MUST remove conflicting options from your existing configuration.
[2024/05/03 17:52:17] [oauthproxy.go:127] using htpasswd file: /tmp/auth_proxy_htpasswd
[2024/05/03 17:52:17] [watcher.go:40] watching '/tmp/auth_proxy_htpasswd' for updates
[2024/05/03 17:52:17] [proxy.go:89] mapping path "^/storage/(.*)$" => upstream "http://s3:8333"
[2024/05/03 17:52:17] [proxy.go:89] mapping path "/grafana/" => upstream "http://grafana:3000"
[2024/05/03 17:52:17] [proxy.go:89] mapping path "/" => upstream "http://cryostat:8181"
[2024/05/03 17:52:17] [oauthproxy.go:171] OAuthProxy configured for Google Client ID: CLIENT_ID
[2024/05/03 17:52:17] [oauthproxy.go:177] Cookie settings: name:_oauth2_proxy secure(https):true httponly:true expiry:168h0m0s domains: path:/ samesite: refresh:disabled
[2024/05/03 17:52:17] [main.go:58] ERROR: Failed to initialise OAuth2 Proxy: error setting up server: could not build app server: error setting up TLS listener: could not load certificate: could not load key data: open /certs/private.key: permission denied

so it seems that the generated certs are somehow not getting into the volumes, or the volumes are not getting attached to the containers. Any ideas?

mwangggg commented 2 months ago

hmm even though the logs show that the quarkus-test-agent is registered, it can't be discovered and no event templates etc. can be found... I'll look into it

mwangggg commented 2 months ago

/build_test

github-actions[bot] commented 2 months ago

Workflow started at 5/10/2024, 11:27:35 AM. View Actions Run.

github-actions[bot] commented 2 months ago

CI build and push: All tests pass ✅ (JDK17) https://github.com/cryostatio/cryostat3/actions/runs/9034486126

mwangggg commented 3 weeks ago

/build_test

github-actions[bot] commented 3 weeks ago

Workflow started at 6/25/2024, 2:22:07 PM. View Actions Run.

github-actions[bot] commented 3 weeks ago

No GraphQL schema changes detected.

github-actions[bot] commented 3 weeks ago

No OpenAPI schema changes detected.

github-actions[bot] commented 3 weeks ago

CI build and push: All tests pass ✅ (JDK17) https://github.com/cryostatio/cryostat/actions/runs/9667643097

mwangggg commented 3 weeks ago

/build_test

github-actions[bot] commented 3 weeks ago

Workflow started at 6/27/2024, 2:14:34 PM. View Actions Run.

github-actions[bot] commented 3 weeks ago

No GraphQL schema changes detected.

github-actions[bot] commented 3 weeks ago

No OpenAPI schema changes detected.

github-actions[bot] commented 3 weeks ago

CI build and push: At least one test failed ❌ (JDK17) https://github.com/cryostatio/cryostat/actions/runs/9701567436

mwangggg commented 3 weeks ago

/build_test

github-actions[bot] commented 3 weeks ago

Workflow started at 6/27/2024, 3:16:15 PM. View Actions Run.

github-actions[bot] commented 3 weeks ago

No GraphQL schema changes detected.

github-actions[bot] commented 3 weeks ago

No OpenAPI schema changes detected.

github-actions[bot] commented 3 weeks ago

CI build and push: All tests pass ✅ (JDK17) https://github.com/cryostatio/cryostat/actions/runs/9702336600

andrewazores commented 2 weeks ago

Trying to run without having yet set up certs:

$ ./smoketest.bash -Okt
...
+ docker-compose -f /home/work/workspace/cryostat/compose/cryostat.yml -f /home/work/workspace/cryostat/compose/db.yml -f /home/work/workspace/cryostat/compose/sample-apps.yml -f /home/work/workspace/cryostat/compose/cryostat-grafana.yml -f /home/work/workspace/cryostat/compose/jfr-datasource.yml -f /home/work/workspace/cryostat/compose/auth_proxy.yml -f /home/work/workspace/cryostat/compose/s3-seaweed.yml up --renew-anon-volumes --remove-orphans --abort-on-container-exit
Creating network "compose_default" with the default driver
Creating volume "compose_postgresql" with local driver
ERROR: Volume auth_proxy_certs declared as external, but could not be found. Please create the volume manually using `docker volume create --name=auth_proxy_certs` and try again.
+ cleanup
+ set +xe
Removing network compose_default
Volume jmxtls_cfg is external, skipping
Volume templates is external, skipping
Removing volume compose_postgresql
Volume auth_proxy_cfg is external, skipping
Volume auth_proxy_certs is external, skipping
Removing volume compose_seaweed_data
WARNING: Volume compose_seaweed_data not found.
proxy_cfg_helper
Error: no container with ID or name "proxy_certs_helper" found: no such container
auth_proxy_cfg
Error: no volume with name "auth_proxy_certs" found: no such volume
jmxtls_cfg_helper
jmxtls_cfg
templates_helper
templates
mwangggg commented 2 weeks ago

ah yes that's because the certs volume is included in auth_proxy.yml, but I just added the USE_HTTPS check when creating the auth_proxy_certs volume...

andrewazores commented 2 weeks ago

After running the two cert generation scripts, ./smoketest.bash -Ot everything looks good. I can open the UI from https://localhost:8443, log in to the auth proxy with user:pass, and generally interact with the UI as expected. I also commented out the vertx-fib-demo sample apps for now, and adjusted the quarkus-test-agent to use my quay.io/andrewazores image since that's what I built it as using the corresponding Agent PR.

Removing the generated certs and running ./smoktest.bash -Okt, everything also looks good.

It might be nice to have a script that can both run the other two scripts to generate the certs required for a TLS-enabled setup, and can delete them to clean up for running a TLS-disabled setup.

Maybe another switch can be added for disabling TLS on the sample applications (or their Agents)? -k can continue for disabled TLS on the auth proxy, and some other letter for disabling TLS on the Agents when used in conjunction with -t? This probably also means splitting up a sample-apps.yml and sample-apps-https.yml.

andrewazores commented 2 weeks ago

Not sure if I am just doing something wrong, but whether I provide the -A flag or not, the quarkus-test-agent always seems to have an https:// URL?

mwangggg commented 2 weeks ago

Not sure if I am just doing something wrong, but whether I provide the -A flag or not, the quarkus-test-agent always seems to have an https:// URL?

hmm I'm not seeing the same thing... what is the exact command you're using? Screenshot from 2024-07-03 10-31-33

andrewazores commented 2 weeks ago

I have tried -OAt, -OAkt, and -Okt. -k is working as expected.

Looking at the implementation, maybe it's an ordering bug - since -A works by removing the file from the array of files to be loaded, I guess the flags are processed in order and it needs to be done as -OtA?

Maybe -A should just set a variable, and after the flag handling is all done then that variable is checked to see if the file should be removed from the list.

andrewazores commented 1 week ago

Rebase please