search

OpenConext / OpenConext-deploy

Ansible-based deployment automation for the OpenConext platform
Apache License 2.0
12 stars 21 forks source link

Installation failed #290

Closed andreasmerkel closed 4 years ago

andreasmerkel commented 4 years ago

Hi,

we tried to install openconext with this deploy package using a freshly installed centos7 machine. We cloned the repository this morning (2020-09-18) using the master branch as documented in the wiki. We have also followed the step by step instructions described there.

During the installation the following error occurred while executing the role TASK [../../roles/vm_only_provision_manage_eb : push metadata eb]:

fatal: [conext]: FAILED! => {"cache_control": "no-cache, no-store, max-age=0, must-revalidate", "changed": false, "connection": "close", "content": "{\"timestamp\":\"2020-09-18T06:49:18.517+00:00\",\"status\":\"403 FORBIDDEN\",\"error\":\"org.springframework.web.client.HttpClientErrorException\",\"message\":\"<html><body><h1>403 Forbidden</h1>\\nRequest forbidden by administrative rules.\\n</body></html>\\n\",\"path\":\"/internal/push\"}", "content_type": "application/json", "date": "Fri, 18 Sep 2020 06:49:18 GMT", "elapsed": 0, "expires": "0", "json": {"error": "org.springframework.web.client.HttpClientErrorException", "message": "<html><body><h1>403 Forbidden</h1>\nRequest forbidden by administrative rules.\n</body></html>\n", "path": "/internal/push", "status": "403 FORBIDDEN", "timestamp": "2020-09-18T06:49:18.517+00:00"}, "msg": "Status code was 400 and not [200]: HTTP Error 400: ", "pragma": "no-cache", "redirected": false, "status": 400, "transfer_encoding": "chunked", "url": "http://127.0.0.1:9393/internal/push", "x_content_type_options": "nosniff", "x_frame_options": "DENY", "x_xss_protection": "1; mode=block"}

It seems that the user "sysadmin" has not the permission to do that operation, although the scopes are correct. Here is our manage-api-users.yml:

# Valid scopes are manage.api.Scope.values(); =>  ["READ", "WRITE", "PUSH"]
apiUsers:
    - {
     name: "dashboard",
      password: "XXX",
      scopes: [ READ ]
    }
    - {
      name: "pdp",
      password: "YYY",
      scopes: [ READ ]
    }
    - {
      name: "sysadmin",
      password: "ZZZ",
      scopes: [ READ, WRITE, PUSH, SYSTEM ]
    }
    - {
      name: "attribute-aggregation",
      password: "AAA",
      scopes: [ READ ]
    }

Did we missed something?

Thank you in advance.

Best regards, Andreas

quartje commented 4 years ago

You've almost made it to the end. What happens when you restart Manage, and try to do this manually?

systemctl restart manage
curl http://127.0.0.1:9393/internal/push -u sysadmin:put_your_secret_here

If it still fails, you might find more information in the Manage logs, which are located in /var/log/manage/manage.log

andreasmerkel commented 4 years ago

Hi, unfortunately, I got the same result:

{"timestamp":"2020-09-18T08:38:06.798+00:00","status":"403 FORBIDDEN","error":"org.springframework.web.client.HttpClientErrorException","message":"<html><body><h1>403 Forbidden</h1>\nRequest forbidden by administrative rules.\n</body></html>\n","path":"/internal/push"}

And here the corresponding error message from /var/log/manage/manage.log

2020-09-18 10:38:06,378  INFO [http-nio-9393-exec-7] manage.control.SystemController:69 - Push initiated by sysadmin
2020-09-18 10:38:06,787 ERROR [http-nio-9393-exec-7] o.a.c.c.C.[.[.[/].[dispatcherServlet]:175 - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: [<html><body><h1>403 Forbidden</h1>
Request forbidden by administrative rules.
</body></html>
]] with root cause
org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: [<html><body><h1>403 Forbidden</h1>
Request forbidden by administrative rules.
</body></html>
]
    at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:109)
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:184)
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:125)
    at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:782)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:740)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:674)
    at org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:449)
    at manage.control.DatabaseController.doPush(DatabaseController.java:88)
    at manage.control.SystemController.pushInternal(SystemController.java:70)
    at manage.control.SystemController$$FastClassBySpringCGLIB$$c5473025.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
    at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
    at manage.control.SystemController$$EnhancerBySpringCGLIB$$efa3f259.pushInternal(<generated>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:626)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:158)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:204)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:92)
    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:77)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:141)
    at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
quartje commented 4 years ago

The problem is that Manage is trying to push it's configuration to Engineblock which generates the 403. That can be either generated by the Haproxy loadbalancer, or by engineblock itself.

Is the engineblock health endpoint accessible from the machine where you have installed manage? It should be reachable here:

https://engine.YOUR_CONFIGURED_BASEDOMAIN/health

andreasmerkel commented 4 years ago

The health endpoint is accessible from the machine. Using curl, I get {"status":"UP"}

I also restartet the haproxy. It is active (running), with the following warnings:

Sep 18 12:15:09 conext haproxy[14722]: [WARNING] 261/121509 (14722) : Can't open server state file '/var/lib/haproxy/state': No such file or directory
Sep 18 12:15:09 conext haproxy[14722]: [NOTICE] 261/121509 (14722) : New worker #1 (14724) forked
Sep 18 12:15:13 conext haproxy[14722]: [WARNING] 261/121513 (14724) : Server oidc_be/java is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Sep 18 12:15:13 conext haproxy[14722]: [ALERT] 261/121513 (14724) : backend 'oidc_be' has no server available!
Sep 18 12:15:16 conext haproxy[14722]: [WARNING] 261/121516 (14724) : Server welcome_be/php is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Sep 18 12:15:16 conext haproxy[14722]: [ALERT] 261/121516 (14724) : backend 'welcome_be' has no server available!
quartje commented 4 years ago

You can ignore those warnings.

Do the PUSH connection settings in /opt/manage/application.yml under push: match those in the Engineblock configuration file (/app/config/parameters.yml under api.users.metadataPush.username: and api.users.metadataPush.password: ?

andreasmerkel commented 4 years ago

Yes, they are identical: username: serviceregistry password is also the same

quartje commented 4 years ago

Does Engineblock log anything? It's identifier is EBLOG in /var/log/messages. And is Engineblock Apache logging the 403? Apache-EBAPI is the identifier for engine-api.
Is the API itself also healthy?: https://engine-api.YOUR_CONFIGURED_BASEDOMAIN/health

andreasmerkel commented 4 years ago

Ok,

thijskh commented 4 years ago

What is listed in /etc/haproxy/haproxy_frontend.cfg for engine-api? Is it the correct hostname there?

andreasmerkel commented 4 years ago

The haproxy hostname for engine-api is also correct. Perhaps it is the content of /etc/hosts. I used 127.0.0.1 as when following the wiki instructions... Maybe I should use the IP address of the network interface?!

thijskh commented 4 years ago

I think you must use the IP that the haproxy restricted frontend is bound to, in order to have your request arrive at the correct stanza of haproxy_frontend.cfg.

So I think you need the hostname in /etc/hosts to resolve to what you configured as haproxy_sni_ip_restricted.

andreasmerkel commented 4 years ago

That's it. When using 127.0.0.2 as ip-address in /etc/hosts for all hosts (as configured in haproxy_sni_ip_restricted in the group_vars directory), the ansible run completes! Thank you very much for your help! Best, Andreas

thijskh commented 4 years ago

Great. Is there some part of the docs that can be improved there?

andreasmerkel commented 4 years ago

Yes, I think it can be mentioned where to find the <ip-address> when using the /etc/hosts way of configuration (-> environments-external/<environment>/group_vars/<environment>.yml) [see here] resp. completely replace <ip-address> with 127.0.0.2 as this value seems to be hardwired in the environment template...