search

jenkinsci / azure-ad-plugin

Authentication and Authorization with Azure AD
https://plugins.jenkins.io/azure-ad/
MIT License
29 stars 57 forks source link

Proxy not resolving all properties correctly #161

Closed meiswjn closed 3 years ago

meiswjn commented 3 years ago

Hey,

First of all, thanks for maintaining this plugin!

I found an issue with the proxy.

Version report

Jenkins: 2.289.3
OS: Linux - 5.4.0-1046-azure
---
chucknorris:1.4
pipeline-model-extensions:1.9.1
pipeline-stage-step:2.5
matlab:2.6.0
bouncycastle-api:2.22
jsch:0.1.55.2
pubsub-light:1.16
uno-choice:2.5.6
list-git-branches-parameter:0.0.9
workflow-cps:2.93
msbuild:1.30
cloudbees-folder:6.16
dark-theme:0.0.12
blueocean-commons:1.24.8
jacoco:3.3.0
powershell:1.5
workflow-basic-steps:2.24
popper-api:1.16.1-2
nuget:1.1
handlebars:3.0.8
jira:3.5
authentication-tokens:1.4
antisamy-markup-formatter:2.1
pipeline-model-definition:1.9.1
next-executions:1.0.15
workflow-support:3.8
matrix-project:1.19
ace-editor:1.1
code-coverage-api:1.4.0
git-server:1.10
copyartifact:1.46.1
bootstrap5-api:5.1.0-1
matrix-auth:2.6.8
jjwt-api:0.11.2-9.c8b45b8bb173
caffeine-api:2.9.2-29.v717aac953ff3
workflow-multibranch:2.26
jquery3-api:3.6.0-2
cloudbees-disk-usage-simple:0.10
okhttp-api:3.14.9
windows-slaves:1.8
docker-commons:1.17
gitlab-plugin:1.5.20
workflow-api:2.46
script-security:1.78
parameterized-trigger:2.41
theme-manager:0.6
build-user-vars-plugin:1.7
plugin-util-api:2.4.0
pipeline-build-step:2.15
pipeline-milestone-step:1.3.2
favorite:2.3.3
git:4.8.1
blueocean:1.24.8
analysis-model-api:10.3.0
warnings-ng:9.5.0
blueocean-github-pipeline:1.24.8
blueocean-events:1.24.8
jaxb:2.3.0.1
token-macro:266.v44a80cf277fd
lockable-resources:2.11
cloudbees-bitbucket-branch-source:2.9.10
jquery-detached:1.2.1
branch-api:2.6.5
extra-columns:1.24
artifactory:3.12.5
snakeyaml-api:1.29.1
github:1.33.1
jackson2-api:2.12.4
jobConfigHistory:2.28.1
ansicolor:1.0.0
checks-api:1.7.2
ldap:2.7
blueocean-i18n:1.24.8
workflow-scm-step:2.13
locale:1.4
azure-sdk:30.vf3165534d6e8
dependency-check-jenkins-plugin:5.1.1
ssh-agent:1.23
build-blocker-plugin:1.7.7
jquery:1.12.4-1
azure-credentials:182.v3ccd4a755864
javadoc:1.6
extended-read-permission:3.2
blueocean-core-js:1.24.8
parameterized-scheduler:1.0
ws-cleanup:0.39
robot:3.0.1
blueocean-rest-impl:1.24.8
pam-auth:1.6
docker-plugin:1.2.2
blueocean-pipeline-scm-api:1.24.8
xunit:3.0.2
ivy:2.1
blueocean-pipeline-api-impl:1.24.8
workflow-aggregator:2.6
cobertura:1.16
workflow-cps-global-lib:2.21
plain-credentials:1.7
bootstrap4-api:4.6.0-3
credentials-binding:1.27
pipeline-model-api:1.9.1
blueocean-dashboard:1.24.8
blueocean-personalization:1.24.8
pipeline-utility-steps:2.8.0
dtkit-api:3.0.0
blueocean-git-pipeline:1.24.8
mailer:1.34
blueocean-web:1.24.8
external-monitor-job:1.7
file-operations:1.11
sonar:2.13.1
resource-disposer:0.16
azure-ad:179.vf6841393099e
github-branch-source:2.11.2
scm-api:2.6.5
docker-workflow:1.26
durable-task:1.39
apache-httpcomponents-client-4-api:4.5.13-1.0
sse-gateway:1.24
echarts-api:5.1.2-9
thinBackup:1.10
build-monitor-plugin:1.12+build.201809061734
ssh-steps:2.0.0
h2-api:1.4.199
envinject-api:1.7
ssh-credentials:1.19
ant:1.11
conditional-buildstep:1.4.1
gradle:1.37.1
workflow-step-api:2.24
forensics-api:1.3.0
data-tables-api:1.10.25-3
jenkins-design-language:1.24.8
variant:1.4
saml:2.0.7
blueocean-jwt:1.24.8
maven-plugin:3.12
pipeline-maven:3.10.0
sshd:3.1.0
pipeline-graph-analysis:1.11
extended-choice-parameter:0.82
junit:1.52
run-condition:1.5
popper2-api:2.9.3-1
pipeline-rest-api:2.19
workflow-job:2.41
docker-java-api:3.1.5.2
ssh-slaves:1.32.0
blueocean-display-url:2.4.1
pipeline-stage-tags-metadata:1.9.1
confluence-publisher:2.0.6
display-url-api:2.3.5
github-api:1.123
command-launcher:1.6
monitoring:1.88.0
workflow-durable-task-step:2.39
Office-365-Connector:4.15.0
build-with-parameters:1.5.1
blueocean-pipeline-editor:1.24.8
blueocean-rest:1.24.8
dashboard-view:2.17
nodejs:1.4.0
plugin-usage-plugin:1.2
config-file-provider:3.8.1
blueocean-autofavorite:1.2.4
metrics:4.0.2.8
pipeline-stage-view:2.19
git-client:3.9.0
blueocean-bitbucket-pipeline:1.24.8
htmlpublisher:1.25
structs:1.23
groovy:2.4
http_request:1.10
trilead-api:1.0.13
momentjs:1.1.1
credentials:2.5
blueocean-config:1.24.8
pipeline-input-step:2.12
envinject:2.4.0
jdk-tool:1.5
email-ext:2.83
integrity-plugin:2.4
simple-theme-plugin:0.7
handy-uri-templates-2-api:2.1.8-1.0
timestamper:1.13
font-awesome-api:5.15.3-4
Ubuntu 18.04.5 LTS

Reproduction steps

MIN_RAM=1G MAX_RAM=6G JAVA_ARGS=" [...] -Djavax.net.ssl.trustStore=/var/lib/jenkins/extra_truststore.jks -Djavax.net.ssl.trustStorePassword='???' -Djetty.ssl.sniHostCheck=false -Djavax.net.debug=ssl:handshake" [...]

- Try to access the "People" API without a proxy. It works.
- Now, configure the proxy in the "Advanced"-section of the plugin manager. For the NO_PROXY, add the following as a minimum (since they are also part of the Azure network, they have 10.* IPs and are internal):

.microsoft.com .microsoftonline.com .office.com .azure.com

- Try to access login.microsoftonline.com via "Validate". It should work. Try to access an URL that is not part of the NO_PROXY. It should also work. Save the configuration.
- Now, try to access the "People" API again. It will work. (**BUG1- Proxy is only applied on startup**)
- Restart Jenkins. Try to access the People API again - it does no longer work as the Azure AD Plugin has applied the new proxy settings.
- Check the system log. An error is thrown:

sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) Caused: sun.security.validator.ValidatorException: PKIX path building failed at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) at java.base/sun.security.validator.Validator.validate(Validator.java:264) at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276) at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141) at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:234) at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:723) at io.netty.internal.tcnative.CertificateVerifierTask.runTask(CertificateVerifierTask.java:36) at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:38) at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$3.run(ReferenceCountedOpenSslEngine.java:1447) at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1512) at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1526) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1390) Caused: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.handshakeException(ReferenceCountedOpenSslEngine.java:1845) at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:812) at java.base/javax.net.ssl.SSLEngine.wrap(SSLEngine.java:522) at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:1039) at io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:925) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1403) at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1245) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1282) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.proxy.ProxyHandler.channelRead(ProxyHandler.java:253) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) at io.netty.handler.proxy.HttpProxyHandler$HttpClientCodecWrapper.channelRead(HttpProxyHandler.java:272) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829)


The certificate error happens when a transparent proxy decrypts the HTTPS traffic and re-encrypts it with its own root certificate. The root certificate must be part of the Jenkins truststore. The truststore has been configured via Java Args.
This is quite the nice error, because it lets us see two issues:
1) This kind of traffic should not be routed through the proxy because it is part of our NO_PROXY. Obviously, it is routed through the proxy because else it would not cause the certificate issue.
2) We should not get a certificate error, because the truststore has the root certificate. We tested the validity of this configuration earlier when validating the proxy settings - if the certificate was wrong, the certificate would also throw an error in that step.

The following assumptions can be made:
- The plugin does not respect the "NO_PROXY" settings of Jenkins **OR** the configured NO_PROXY targets were incomplete.
- The plugin does not respect the truststore settings of Jenkins **OR** the configured NO_PROXY targets were incomplete.

- To find the valid assumption, install a Squid Proxy on the machine and connect it to the parent proxy. Now, we route all Jenkins traffic to Squid and let Squid decide if it is forwarded to the parent proxy or if it is internal and thus part of the "NO_PROXY".
- Set up Squid with the following configuration:

http_port 3128 http_access allow all

cache_peer parent 8080 0 no-query default

acl all src 0.0.0.0/0.0.0.0 acl microsoft dstdomain .office.com .microsoft.com .microsoftonline.com .azure.com never_direct deny microsoft never_direct allow all

debug_options ALL,1 33,2 28,9


- Configure the Jenkins proxy under "Plugins" -> "Advanced" to use "localhost:3128" without any no_proxy. Validate the settings as previously - it should work.
- Restart Jenkins.
- Try to use the People API. It should work, even though we use the same no_proxy settings. 

### Results

Thus, the following assumptions are valid:
- The plugin does not respect the "NO_PROXY" settings of Jenkins.
- The plugin does not respect the truststore settings of Jenkins.
- Furthermore, the plugin only reloads the proxy settings after a restart. Other parts of Jenkins appear to always pull the proxy settings upon establishing the connection or upon changing the proxy settings.
timja commented 3 years ago

The graph proxy is done with custom code and not the standard azure sdk integration, looks like proxy was just missed: https://github.com/jenkinsci/azure-ad-plugin/blob/f6841393099e2c75cab09b0cf0f2a857ec91861b/src/main/java/com/microsoft/jenkins/azuread/GraphProxy.java#L129

should be quite straightforward.

otherwise I can do a PR when I get a chance

all proxy support is best effort for me, I do not have an environment to test it in, and I always prefer someone with an environment to either create the PR or at least validate it

meiswjn commented 3 years ago

Apologies for the late reply. I will have a look at it.

meiswjn commented 3 years ago

I'm sorry, but I don't get the referred code yet. As of now, I don't see any usage of the class "GraphProxy" in the whole project - am I missing something? The class "AzureSecurityRealm" uses the hudson-native proxy configuration, it appears.

timja commented 3 years ago

This method https://github.com/jenkinsci/azure-ad-plugin/blob/f6841393099e2c75cab09b0cf0f2a857ec91861b/src/main/java/com/microsoft/jenkins/azuread/GraphProxy.java#L125

Is called by javascript

meiswjn commented 3 years ago

Not as easy as I thought. Using the OkHttpClient Builder to use the proxy in the proxy() method did not bring any success. Let me know if you have any ideas!

timja commented 3 years ago

can you create a draft PR with what you've got?

It should be roughly: Jenkins.get().getProxy() and then getting the values off of that

meiswjn commented 3 years ago

Done :)