search

spring-cloud / spring-cloud-config

External configuration (server and client) for Spring Cloud
Apache License 2.0
1.97k stars 1.29k forks source link

Issue with multiple git repositories with different ssh keys #2010

Open tomplummercarfax opened 2 years ago

tomplummercarfax commented 2 years ago

Describe the bug We are using both the "Pattern Matching and Multiple Repositories" and "Composite Environment Repositories" features and we have different ssh keys for different git repos. When the config server is started/restarted all of our configuration is served appropriately. However if configuration is updated in one of our Composite Environment Repositories (and the config server is not restarted) the changes are never served (even after the refresh-rate has passed). We turned on debug logging and found the following exception when trying to request configuration from the config server:

2021-12-06 09:02:04,079 DEBUG [o.s.c.c.s.e.MultipleJGitEnvironmentRepository] - Stacktrace for: Could not fetch remote for alpha remote: ssh://git@ourcoderepo:7999/our/config-project-4.git
org.eclipse.jgit.api.errors.InvalidRemoteException: Invalid remote: origin
    at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:251)
    at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.fetch(JGitEnvironmentRepository.java:505)
    at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.refresh(JGitEnvironmentRepository.java:269)
    at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.getLocations(JGitEnvironmentRepository.java:245)
    at org.springframework.cloud.config.server.environment.MultipleJGitEnvironmentRepository.getLocations(MultipleJGitEnvironmentRepository.java:139)
    at org.springframework.cloud.config.server.environment.AbstractScmEnvironmentRepository.findOne(AbstractScmEnvironmentRepository.java:55)
    at org.springframework.cloud.config.server.environment.MultipleJGitEnvironmentRepository.findOne(MultipleJGitEnvironmentRepository.java:173)
    at org.springframework.cloud.config.server.environment.CompositeEnvironmentRepository.findOne(CompositeEnvironmentRepository.java:73)
    at org.springframework.cloud.config.server.environment.EnvironmentEncryptorEnvironmentRepository.findOne(EnvironmentEncryptorEnvironmentRepository.java:61)
    at org.springframework.cloud.config.server.environment.EnvironmentController.getEnvironment(EnvironmentController.java:131)
    at org.springframework.cloud.config.server.environment.EnvironmentController.labelled(EnvironmentController.java:118)
    at org.springframework.cloud.config.server.environment.EnvironmentController.labelledProperties(EnvironmentController.java:162)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
    at org.springframework.cloud.context.scope.GenericScope$LockedScopedProxyFactoryBean.invoke(GenericScope.java:485)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692)
    at org.springframework.cloud.config.server.environment.EnvironmentController$$EnhancerBySpringCGLIB$$7dc99a6a.labelledProperties(<generated>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1063)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:626)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:121)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126)
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103)
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.eclipse.jgit.errors.NoRemoteRepositoryException: ssh://git@ourcoderepo:7999/our/config-project-4.git: Repository not found
The requested repository does not exist, or you do not have permission to access it.

    at org.eclipse.jgit.transport.TransportGitSsh.cleanNotFound(TransportGitSsh.java:218)
    at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:306)
    at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:170)
    at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:137)
    at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:123)
    at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1271)
    at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:243)
    ... 110 common frames omitted

It appears that Spring Cloud Config is attempting to use the key from one of the non-Composite git repos when trying to pull new config from the Composite git repo, because when we added the private key we were using for the non-Composite repo to the Composite repo Spring Cloud Config was able to pull the changes.

We are using org.springframework.cloud:spring-cloud-config-server:3.0.4.

Sample Here is a copy of our application.yml for our spring cloud config server (I changed some project names and excluded the actual keys):

  security:
    basic:
      enabled: false
    ignored: /**
  management:
    security:
      enabled: false
    endpoint:
      health:
        show-details: ALWAYS
      prometheus:
        enabled: true
    endpoints:
      web:
        exposure:
          include: "*"
  server:
    port: 8888
  spring:
    profiles:
      active: composite, git
    cloud:
      config:
        server:
          health:
            enabled: false
          git:
            refreshRate: 10
            uri: ssh://git@ourcoderepo:7999/our/config-project-1.git
            searchPaths: '{application},path1/{application},path2/{application},path3/{application}'
            ignoreLocalSshSettings: true
            cloneOnStart: true
            timeout: 30
            privateKey: |
                         -----BEGIN RSA PRIVATE KEY-----
                         (put key 1 here)
                         -----END RSA PRIVATE KEY-----
            repos:
              database:
                pattern: database*
                uri: ssh://git@ourcoderepo:7999/our/config-project-2.git
                searchPaths: '{application},mongodb/{application},mysql/{application},oracle/{application},postgres/{application}'
                ignoreLocalSshSettings: true
                cloneOnStart: true
                privateKey: |
                            -----BEGIN RSA PRIVATE KEY-----
                            (put key 1 here)
                            -----END RSA PRIVATE KEY-----
              datamajique:
                pattern: antherpattern*
                uri: ssh://git@ourcoderepo:7999/our/config-project-3.git
                searchPaths: '{application}'
                ignoreLocalSshSettings: true
                cloneOnStart: true
                privateKey: |
                            -----BEGIN RSA PRIVATE KEY-----
                            (put key 1 here)
                            -----END RSA PRIVATE KEY-----
          composite:
            - type: git
              uri: ssh://git@ourcoderepo:7999/our/config-project-4.git
              pattern: database*
              searchPaths: '{application},mongodb/{application},mysql/{application},oracle/{application},postgres/{application}'
              refresh-rate: 10
              ignoreLocalSshSettings: true
              cloneOnStart: true
              strict-host-key-checking: false
              privateKey: |
                -----BEGIN RSA PRIVATE KEY-----
                (put key 2 here)
                -----END RSA PRIVATE KEY-----
  logging.level:
    org.springframework.cloud.config.server.environment.MultipleJGitEnvironmentRepository: ERROR
ryanjbaxter commented 2 years ago

I'm not sure i understand this configuration...

You have 3 git repos, why use a composite at all?

tomplummercarfax commented 2 years ago

We are using composite because our "database" configuration comes from both config-project-2 and config-project-4. For example we make the following request to the config server: http://the.url.to.our.config.server/alpha/database(_)appname-alpha.properties and we need configuration that is stored in both config-project-2 and config-project-4.

ryanjbaxter commented 2 years ago

But why can't you choose either a git configuration or a composite repo? If you took what in the composite and put it under the git configuration wouldn't it pull configuration from config-project-2 and config-project-4 but not config-project-2. because the pattern wouldn't match?

tomplummercarfax commented 2 years ago

I just experimented with what you suggested, but there appear to be problems with using only a git configuration or only using a composite configuration. If we put the current composite repo (config-project-4) as another git configuration, our database configuration does not pull from both config-project-2 and config-project-4, it only pulls from the one that matches first. If we make all of the repos composite repos, when we request database configuration we also get configuration from config-project-1 and config-project-3 in addition to config-project-2 and config-project-4. It appears patterns do not work for composite repos.

I understand that we may be handling our config server in an odd way, but why does it work fine the way we have it configured immediately after the config server is started, but not later when a repo needs to be refreshed? I'm open to finding a better way to configure our config server, but I would also like to know why we can't look at the actual problem I'm reporting.

ryanjbaxter commented 2 years ago

Thanks for giving this a try. Before I went through the trouble to trying to see why the refreshed SSH key does not work, I wanted to see if there was another way of configuring the server that avoided the problem all together. Since this is not going to be strait forward to reproduce I wanted to see if there was another way.