Closed estemendoza closed 2 years ago
Hi @estemendoza,
Thanks for contacting us. Can you please confirm ECS health for the Paladin cluster? Also, can you please check and let us know if there are any errors in the cloud watch log for auth API?
Hi @santhosh-challa
Cluster seems to be working fine:
And these are the last lines on cloud watch logs for auth api:
Let me know if you need me to send you more info for debug.
Thanks
Thanks for the info. It seems the config service is not running. We can see the running task count is zero and auth is unable to reach the config API. All the APIs need to connect to the config service.
Can you please check the cloudwatch logs for config service and see if there are any errors?
I found this for latest messages on the config service logs:
This seems to be some dependency issue for the config service. We are not seeing this error internally.
Can you please try to redeploy once and see if we can get past this?
Hi
I run the redeploy script but I am still getting error, the paladin cloud-config keeps failing with same error
By the way, also having this issue on install or redeploy:
Step 5: Deleting resources for redeployment
Exception in thread Thread-4:
Traceback (most recent call last):
File "/usr/lib/python3.9/threading.py", line 980, in _bootstrap_inner
self.run()
File "/usr/lib/python3.9/threading.py", line 917, in run
self._target(*self._args, **self._kwargs)
File "/home/ubuntu/CE-1.0.1/installer/core/providers/aws/reinstall.py", line 111, in show_progress_status_all
self.render_terraform_destroy_progress() # Show destroy progress
File "/home/ubuntu/CE-1.0.1/installer/core/providers/aws/reinstall.py", line 118, in render_terraform_destroy_progress
while self.destroy is False and self.terraform_thread.isAlive():
AttributeError: 'Thread' object has no attribute 'isAlive'
Can you please zip and upload the logs directory under install?
That might give us some clue.
Here are the logs: installer_logs.zip
Hi @estemendoza ,
Thank you for the logs. We could not see any errors that are related to the config API issue.
We saw that you are using python 3.9, we tried to redeploy the app on the same version. We are using Python 3.7.x internally. Though we could see the same error on the console as you posted before (thread error related to isAlive), the app is installed successfully and we could not reproduce the config API issue.
This thread issue is happening in Python 3.9.x as isAlive is changed to is_Alive. We will either fix the thread issue or fix the documentation. Thanks for helping us to fix these issues!
To understand the issue with the config API, can you please provide us with the following:
Thank you.
Ok, you can download the file here: https://dl.meecles.net/config.jar
I do not see any issues with the jar.
I can run your jar in terminal and I do not see the error. Can you please try running the jar as below in a terminal:
java -jar config.jar
And let us know if you can see the same error or not?
Thank you.
Hi. Here is the log for that command:
$ java -jar config.jar
2022-07-28 15:00:37.748 INFO 8969 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@35a50a4c: startup date [Thu Jul 28 15:00:37 UTC 2022]; root of context hierarchy
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (jar:file:/home/ubuntu/config.jar!/BOOT-INF/lib/spring-core-5.0.8.RELEASE.jar!/) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022-07-28 15:00:38.942 INFO 8969 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'configurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$acd60c5d] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.0.4.RELEASE)
2022-07-28 15:00:39.665 INFO 8969 --- [ main] c.t.pacman.config.ConfigApplication : The following profiles are active: jdbc
2022-07-28 15:00:39.704 INFO 8969 --- [ main] ConfigServletWebServerApplicationContext : Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@5fa07e12: startup date [Thu Jul 28 15:00:39 UTC 2022]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@35a50a4c
2022-07-28 15:00:41.557 INFO 8969 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Overriding bean definition for bean 'dataSource' with a different definition: replacing [Root bean: class [null]; scope=refresh; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=false; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.jdbc.DataSourceConfiguration$Hikari; factoryMethodName=dataSource; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/jdbc/DataSourceConfiguration$Hikari.class]] with [Root bean: class [org.springframework.aop.scope.ScopedProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in BeanDefinition defined in class path resource [org/springframework/boot/autoconfigure/jdbc/DataSourceConfiguration$Hikari.class]]
2022-07-28 15:00:42.174 INFO 8969 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=2967e5b1-2a2e-3aa2-aecb-c2bf46b0d32c
2022-07-28 15:00:42.440 INFO 8969 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$90bc0960] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2022-07-28 15:00:42.512 INFO 8969 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$acd60c5d] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2022-07-28 15:00:43.422 INFO 8969 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8888 (http)
2022-07-28 15:00:43.492 INFO 8969 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2022-07-28 15:00:43.494 INFO 8969 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet Engine: Apache Tomcat/8.5.32
2022-07-28 15:00:43.514 INFO 8969 --- [ost-startStop-1] o.a.catalina.core.AprLifecycleListener : The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib]
2022-07-28 15:00:43.726 INFO 8969 --- [ost-startStop-1] o.a.c.c.C.[.[localhost].[/api/config] : Initializing Spring embedded WebApplicationContext
2022-07-28 15:00:43.731 INFO 8969 --- [ost-startStop-1] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 4027 ms
2022-07-28 15:00:45.762 INFO 8969 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'characterEncodingFilter' to: [/*]
2022-07-28 15:00:45.762 INFO 8969 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'webMvcMetricsFilter' to: [/*]
2022-07-28 15:00:45.763 INFO 8969 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2022-07-28 15:00:45.763 INFO 8969 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'httpPutFormContentFilter' to: [/*]
2022-07-28 15:00:45.763 INFO 8969 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'requestContextFilter' to: [/*]
2022-07-28 15:00:45.764 INFO 8969 --- [ost-startStop-1] .s.DelegatingFilterProxyRegistrationBean : Mapping filter: 'springSecurityFilterChain' to: [/*]
2022-07-28 15:00:45.765 INFO 8969 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'httpTraceFilter' to: [/*]
2022-07-28 15:00:45.765 INFO 8969 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean : Servlet dispatcherServlet mapped to [/]
2022-07-28 15:00:46.036 INFO 8969 --- [ main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2022-07-28 15:00:46.462 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@5fa07e12: startup date [Thu Jul 28 15:00:39 UTC 2022]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@35a50a4c
2022-07-28 15:00:46.628 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2022-07-28 15:00:46.636 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2022-07-28 15:00:46.660 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/encrypt/status],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> org.springframework.cloud.config.server.encryption.EncryptionController.status()
2022-07-28 15:00:46.661 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/key/{name}/{profiles}],methods=[GET]}" onto public java.lang.String org.springframework.cloud.config.server.encryption.EncryptionController.getPublicKey(java.lang.String,java.lang.String)
2022-07-28 15:00:46.661 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/key],methods=[GET]}" onto public java.lang.String org.springframework.cloud.config.server.encryption.EncryptionController.getPublicKey()
2022-07-28 15:00:46.662 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/encrypt/{name}/{profiles}],methods=[POST]}" onto public java.lang.String org.springframework.cloud.config.server.encryption.EncryptionController.encrypt(java.lang.String,java.lang.String,java.lang.String,org.springframework.http.MediaType)
2022-07-28 15:00:46.662 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/encrypt],methods=[POST]}" onto public java.lang.String org.springframework.cloud.config.server.encryption.EncryptionController.encrypt(java.lang.String,org.springframework.http.MediaType)
2022-07-28 15:00:46.663 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/decrypt],methods=[POST]}" onto public java.lang.String org.springframework.cloud.config.server.encryption.EncryptionController.decrypt(java.lang.String,org.springframework.http.MediaType)
2022-07-28 15:00:46.663 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/decrypt/{name}/{profiles}],methods=[POST]}" onto public java.lang.String org.springframework.cloud.config.server.encryption.EncryptionController.decrypt(java.lang.String,java.lang.String,java.lang.String,org.springframework.http.MediaType)
2022-07-28 15:00:46.675 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{name}-{profiles}.properties],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.lang.String> org.springframework.cloud.config.server.environment.EnvironmentController.properties(java.lang.String,java.lang.String,boolean) throws java.io.IOException
2022-07-28 15:00:46.680 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{name}/{profiles}/{label:.*}],methods=[GET]}" onto public org.springframework.cloud.config.environment.Environment org.springframework.cloud.config.server.environment.EnvironmentController.labelled(java.lang.String,java.lang.String,java.lang.String)
2022-07-28 15:00:46.681 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{name}-{profiles}.yml || /{name}-{profiles}.yaml],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.lang.String> org.springframework.cloud.config.server.environment.EnvironmentController.yaml(java.lang.String,java.lang.String,boolean) throws java.lang.Exception
2022-07-28 15:00:46.682 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{label}/{name}-{profiles}.properties],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.lang.String> org.springframework.cloud.config.server.environment.EnvironmentController.labelledProperties(java.lang.String,java.lang.String,java.lang.String,boolean) throws java.io.IOException
2022-07-28 15:00:46.684 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{name}-{profiles}.json],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.lang.String> org.springframework.cloud.config.server.environment.EnvironmentController.jsonProperties(java.lang.String,java.lang.String,boolean) throws java.lang.Exception
2022-07-28 15:00:46.684 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{label}/{name}-{profiles}.json],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.lang.String> org.springframework.cloud.config.server.environment.EnvironmentController.labelledJsonProperties(java.lang.String,java.lang.String,java.lang.String,boolean) throws java.lang.Exception
2022-07-28 15:00:46.684 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{label}/{name}-{profiles}.yml || /{label}/{name}-{profiles}.yaml],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.lang.String> org.springframework.cloud.config.server.environment.EnvironmentController.labelledYaml(java.lang.String,java.lang.String,java.lang.String,boolean) throws java.lang.Exception
2022-07-28 15:00:46.685 INFO 8969 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{name}/{profiles:.*[^-].*}],methods=[GET]}" onto public org.springframework.cloud.config.environment.Environment org.springframework.cloud.config.server.environment.EnvironmentController.defaultLabel(java.lang.String,java.lang.String)
2022-07-28 15:00:46.749 INFO 8969 --- [ main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2022-07-28 15:00:46.749 INFO 8969 --- [ main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2022-07-28 15:00:48.159 INFO 8969 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Creating filter chain: org.springframework.security.web.util.matcher.AnyRequestMatcher@1, [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@6e106680, org.springframework.security.web.context.SecurityContextPersistenceFilter@746cd757, org.springframework.security.web.header.HeaderWriterFilter@27c04377, org.springframework.security.web.csrf.CsrfFilter@70242f38, org.springframework.security.web.authentication.logout.LogoutFilter@4567e53d, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@f339eae, org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter@3b0ee03a, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@16c8b7bd, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@77b9d0c7, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@2532b351, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@576c5536, org.springframework.security.web.session.SessionManagementFilter@67403656, org.springframework.security.web.access.ExceptionTranslationFilter@6ed06f69, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@62e7dffa]
2022-07-28 15:00:48.979 INFO 8969 --- [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 2 endpoint(s) beneath base path '/actuator'
2022-07-28 15:00:49.002 INFO 8969 --- [ main] s.b.a.e.w.s.WebMvcEndpointHandlerMapping : Mapped "{[/actuator/health],methods=[GET],produces=[application/vnd.spring-boot.actuator.v2+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(javax.servlet.http.HttpServletRequest,java.util.Map<java.lang.String, java.lang.String>)
2022-07-28 15:00:49.008 INFO 8969 --- [ main] s.b.a.e.w.s.WebMvcEndpointHandlerMapping : Mapped "{[/actuator/info],methods=[GET],produces=[application/vnd.spring-boot.actuator.v2+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(javax.servlet.http.HttpServletRequest,java.util.Map<java.lang.String, java.lang.String>)
2022-07-28 15:00:49.009 INFO 8969 --- [ main] s.b.a.e.w.s.WebMvcEndpointHandlerMapping : Mapped "{[/actuator],methods=[GET],produces=[application/vnd.spring-boot.actuator.v2+json || application/json]}" onto protected java.util.Map<java.lang.String, java.util.Map<java.lang.String, org.springframework.boot.actuate.endpoint.web.Link>> org.springframework.boot.actuate.endpoint.web.servlet.WebMvcEndpointHandlerMapping.links(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2022-07-28 15:00:49.121 INFO 8969 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
2022-07-28 15:00:49.129 INFO 8969 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Bean with name 'dataSource' has been autodetected for JMX exposure
2022-07-28 15:00:49.144 INFO 8969 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Bean with name 'environmentManager' has been autodetected for JMX exposure
2022-07-28 15:00:49.153 INFO 8969 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Bean with name 'refreshScope' has been autodetected for JMX exposure
2022-07-28 15:00:49.154 INFO 8969 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Bean with name 'configurationPropertiesRebinder' has been autodetected for JMX exposure
2022-07-28 15:00:49.163 INFO 8969 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Located managed bean 'environmentManager': registering with JMX server as MBean [org.springframework.cloud.context.environment:name=environmentManager,type=EnvironmentManager]
2022-07-28 15:00:49.181 INFO 8969 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Located managed bean 'refreshScope': registering with JMX server as MBean [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope]
2022-07-28 15:00:49.211 INFO 8969 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=5fa07e12,type=ConfigurationPropertiesRebinder]
2022-07-28 15:00:49.227 INFO 8969 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Located MBean 'dataSource': registering with JMX server as MBean [com.zaxxer.hikari:name=dataSource,type=HikariDataSource]
2022-07-28 15:00:49.351 INFO 8969 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8888 (http) with context path '/api/config'
2022-07-28 15:00:49.361 INFO 8969 --- [ main] c.t.pacman.config.ConfigApplication : Started ConfigApplication in 13.421 seconds (JVM running for 14.921)
Thanks for the log.
As we can see:
022-07-28 15:00:49.361 INFO 8969 --- [ main] c.t.pacman.config.ConfigApplication : Started ConfigApplication in 13.421 seconds (JVM running for 14.921)
Config API started without any error.
Can you please restart the ECS task for config service and see if it is able to start? Please share the latest log if you see any errors.
Thanks.
No problem, what's the proper way of restarting that task? I haven't work with ECS for a while
You can follow the below steps:
As this is a FARGATE, ECS automatically starts the task again.
Also Please share the cluster screenshot once, just want to check the health.
Thanks
This is the status after restarting the task, I waited for a bit, but the task keeps failing
Can you also share the latest log for it, from the starting?
it's basically the same error as before
can you get the full log, from the start?
There is not much of it
I will try to compare it with the internal log and see if there is anything different. Not sure why the same jar is not able to start inside the container.
Hi @estemendoza,
Sorry for the delay. I compared your log with the internal log and couldn't find anything interesting.
The error seems coming due to missing of hybetnate validator jar file and the same is in the config jar. We couldn't reproduce the same.
Can you please destroy and install the Paladin Cloud once and see if we can get past this issue?
Thanks
Hi @santhosh-challa,
I can try it again, no problem. Question, I am running the installer from an ubuntu instance, I think I didn't specify it before, but I mention it just in case, could this affect the installer somehow?
Not sure, we will try that once. Can you try with Amazon Linux this time and let us know the results.
Hi @santhosh-challa
Why is it recommended to use a t2.large instance type for the installer?
Hi @estemendoza,
Yes, t2.large is recommended. Also, we had a release happened yesterday. Please check and start the installation from the latest release.
Please let us know in case of any further information is needed.
Thanks, Santhosh
I have executed the installer again using Amazon Linux image and using the same auth creds than before getting this error:
Step 1: Checking if required tools are availables
Tool: Maven, checking ................................................................ [Found]
Tool: Git, checking .................................................................. [Found]
Tool: MySQL client, checking ......................................................... [Found]
Tool: Terraform, checking ............................................................ [Found]
Tool: Nodejs, checking ............................................................... [Found]
Tool: npm, checking .................................................................. [Found]
Tool: Angular, checking .............................................................. [Found]
Tool: Yarn, checking ................................................................. [Found]
Tool: Docker, checking ............................................................... [Found]
Required tools are available!!!
Step 2: Checking if required python packages are available
Package: docker, Module: Client, checking ............................................ [Found]
Module: boto3, checking .............................................................. [Found]
Required python packages are available!!!
Step 3: Reading required inputs from user
Select AWS authentication mechanism:
1. Using access key and secret key
2. Assuming an IAM role
3. Using IAM role attached to the this instance
Type 1 or 2 or 3 to continue to create services in AWS: 1
Required inputs are available!!!
Step 4: Checking settings and inputs
Checking VPC and CIDR Blocks ......................................................... [Valid]
Checking VPC subnets ................................................................. [Valid]
Checking user-attached policies ...................................................... [Not Present]
Checking group-attached policies ..................................................... [Not Present]
If you have added custom policies with all permissions, please type Yes or No: yes
Step 5: Checking resource existence
Resource existence check completed!!!
Step 6: Terraform file generation started
Error: Either docker service is not available or user do not have permission to run
Error occured, please check error log for more details
Can you start the docker service and retry? Maybe it is not running.
sudo systemctl docker start
Yeah, the docker service was dead, sudo systemctl start docker
fixed the issue, now the installer is running fine, will reply with results.
In the meantime, you should double check how the installer checks that docker is installed and running because even though the docker service was not running, the installer says that docker was found correctly.
Thanks for the feedback. I am thinking it is checking the availability of Docker in step#1. It is not checking the running state may be. We will check that.
Please post the results.
My ssh connection died in the middle of the installer and it failed. When I try to destroy it to install it again, I get this error:
Step 4: Terraform destroy started
Time elapsed: 0m 7s [..Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib64/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/ec2-user/CE-1.0.2/installer/core/providers/aws/destroy.py", line 142, in destroy_resources
PyTerraform.save_terraform_output()
File "/home/ec2-user/CE-1.0.2/installer/core/terraform/__init__.py", line 239, in save_terraform_output
output_dict = cls.load_terraform_output()
File "/home/ec2-user/CE-1.0.2/installer/core/terraform/__init__.py", line 264, in load_terraform_output
working_dir=Settings.TERRAFORM_DIR,
File "/home/ec2-user/CE-1.0.2/installer/core/lib/python_terraform/__init__.py", line 109, in __init__
self.read_state_file(self.state)
File "/home/ec2-user/CE-1.0.2/installer/core/lib/python_terraform/__init__.py", line 419, in read_state_file
self.tfstate = Tfstate.load_file(file_path)
File "/home/ec2-user/CE-1.0.2/installer/core/lib/python_terraform/__init__.py", line 54, in load_file
json_data = json.load(f)
File "/usr/lib64/python3.7/json/__init__.py", line 296, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib64/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Error occured, please check error log for more details
If I try to install again, I get this:
Step 5: Checking resource existence
Resource: Aws Iam Role, name: `paladincloud_ro` ...................................... [Exists]
Resource: Aws Iam Policy, name: `paladincloud_ecs_task_exec` ......................... [Exists]
Resource: Aws Iam Role, name: `paladincloud_run_batch` ............................... [Exists]
Resource: Aws Iam Role, name: `paladincloud_ecs_role` ................................ [Exists]
Resource: Aws Iam Instance Profile, name: `paladincloud_ecs_role_profile` ............ [Exists]
Resource: Aws Iam Role, name: `paladincloud_lambda_basic_execution` .................. [Exists]
Resource: Aws Iam Policy, name: `paladincloud_ro` .................................... [Exists]
Resource: Aws Iam Role, name: `paladincloud` ......................................... [Exists]
Resource: Aws Iam Policy, name: `paladincloud_pacbot-autofix` ........................ [Exists]
Resource: Aws Iam Policy, name: `paladincloud` ....................................... [Exists]
Resource: Aws Security Group, name: `paladincloud` ................................... [Exists]
Resource: Aws Db Option Group, name: `paladincloud-mysql` ............................ [Exists]
Resource: Aws Db Parameter Group, name: `paladincloud-mysql` ......................... [Exists]
Resource: Aws Db Subnet Group, name: `paladincloud-mysql` ............................ [Exists]
Resource: Aws Db Instance, identifier: `paladincloud-data` ........................... [Exists]
Resource: Aws Cloudwatch Log Group, name: `paladincloud/elasticsearch` ............... [Exists]
Resource: Aws Elasticsearch Domain, domain_name: `paladincloud-data` ................. [Exists]
Resource: Aws Lb, name: `paladincloud` ............................................... [Exists]
Resource: Aws Alb Target Group, name: `paladincloud-admin` ........................... [Exists]
Resource: Aws Alb Target Group, name: `paladincloud-asset` ........................... [Exists]
Resource: Aws Alb Target Group, name: `paladincloud-auth` ............................ [Exists]
Resource: Aws Alb Target Group, name: `paladincloud-compliance` ...................... [Exists]
Resource: Aws Alb Target Group, name: `paladincloud-config` .......................... [Exists]
Resource: Aws Alb Target Group, name: `paladincloud-ngnix` ........................... [Exists]
Resource: Aws Alb Target Group, name: `paladincloud-notifications` ................... [Exists]
Resource: Aws Alb Target Group, name: `paladincloud-statistics` ...................... [Exists]
Resource: Aws Ecr Repository, name: `paladincloud-microservices` ..................... [Exists]
Resource: Aws Ecr Repository, name: `paladincloud-webapp` ............................ [Exists]
Resource: Aws Cloudwatch Log Group, name: `paladincloud/apis` ........................ [Exists]
Resource: Aws Cloudwatch Log Group, name: `paladincloud/ui` .......................... [Exists]
Resource: Aws Iam Policy, name: `paladincloud_s3` .................................... [Exists]
Resource: Aws Batch Compute Environment, name: `paladincloud` ........................ [Exists]
Resource: Aws Ecr Repository, name: `paladincloud-rule-engine` ....................... [Exists]
Resource: Aws Batch Job Queue, name: `paladincloud-data` ............................. [Exists]
Resource: Aws Batch Job Queue, name: `paladincloud-rule-engine` ...................... [Exists]
Resource existence check failed
Please delete all the existing resources or change their names
What can I do to clean up the resources and install again?
We have pointed out the faqs in the readme as a troubleshooting step. FYI, https://github.com/PaladinCloud/CE/wiki/Installation-FAQs
Can you please run the redeploy and see if it is successful? You can run the commands in the background to avoid connection timeouts.
Please let us know in case you need any further help.
Thanks.
Getting this for redeploy:
$ sudo python3 manager.py redeploy
###########################################################################################################################
####################################### ____ ____ ######################################
####################################### |/||\| PacBot |/||\| ######################################
####################################### . || . Infra setup and deployment . || . ######################################
####################################### || || ######################################
###########################################################################################################################
Step 1: Checking if required tools are availables
Tool: Maven, checking ................................................................ [Found]
Tool: Git, checking .................................................................. [Found]
Tool: MySQL client, checking ......................................................... [Found]
Tool: Terraform, checking ............................................................ [Found]
Tool: Nodejs, checking ............................................................... [Found]
Tool: npm, checking .................................................................. [Found]
Tool: Angular, checking .............................................................. [Found]
Tool: Yarn, checking ................................................................. [Found]
Tool: Docker, checking ............................................................... [Found]
Required tools are available!!!
Step 2: Checking if required python packages are available
Package: docker, Module: Client, checking ............................................ [Found]
Module: boto3, checking .............................................................. [Found]
Required python packages are available!!!
Step 3: Reading required inputs from user
Select AWS authentication mechanism:
1. Using access key and secret key
2. Assuming an IAM role
3. Using IAM role attached to the this instance
Type 1 or 2 or 3 to continue to create services in AWS: 1
Required inputs are available!!!
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib64/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/ec2-user/CE-1.0.2/installer/custom/commands/redeploy.py", line 165, in run_reinstallation
self.generate_terraform_files_and_upgrade_state(input_instance)
File "/home/ec2-user/CE-1.0.2/installer/custom/commands/redeploy.py", line 153, in generate_terraform_files_and_upgrade_state
PyTerraform.terrafomr12_upgrade() # This is required only when terraform version 12 is used
File "/home/ec2-user/CE-1.0.2/installer/core/terraform/__init__.py", line 352, in terrafomr12_upgrade
working_dir=Settings.TERRAFORM_DIR,
File "/home/ec2-user/CE-1.0.2/installer/core/lib/python_terraform/__init__.py", line 109, in __init__
self.read_state_file(self.state)
File "/home/ec2-user/CE-1.0.2/installer/core/lib/python_terraform/__init__.py", line 419, in read_state_file
self.tfstate = Tfstate.load_file(file_path)
File "/home/ec2-user/CE-1.0.2/installer/core/lib/python_terraform/__init__.py", line 54, in load_file
json_data = json.load(f)
File "/usr/lib64/python3.7/json/__init__.py", line 296, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib64/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Can you please upload the logs from the installer/log directory?
Sure, here they are: logs.zip
Hi @estemendoza ,
Found below error on maven build log:
Failed to execute goal on project gcp-discovery: Could not resolve dependencies for project com.paladincloud:gcp-discovery:jar:0.0.1-SNAPSHOT: The following artifacts could not be resolved: io.grpc:grpc-netty-shaded:jar:1.44.1, io.grpc:grpc-xds:jar:1.47.0, io.opencensus:opencensus-proto:jar:0.2.0, com.google.api:gax:jar:2.12.2: Could not transfer artifact io.grpc:grpc-netty-shaded:jar:1.44.1 from/to central (https://repo.maven.apache.org/maven2): GET request of: io/grpc/grpc-netty-shaded/1.44.1/grpc-netty-shaded-1.44.1.jar from central failed: No space left on device -> [Help 1]
It seems, there is no space left on the installer machine. Can you please check and retry the process?
Thanks, Santhosh
Yes, I saw that and fixed the issue before, but I tried again the redeploy command and still getting that same error as before
Can you please confirm the free space on the instance once? What is the size you are using for installation machine?
Yep, I have this
$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 576K 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/xvda1 20G 11G 10G 51% /
@estemendoza can you try to install the latest release of Paladin Cloud release 1.1.0 and check if the issue persists?
@lakspp sure, no problem, do I need to run a "destroy" command or "install" directly? Because there are still a lot of resources created from the previous installation attempt
yes, do destroy and install again.
@lakspp I am getting this error now with destroy:
****************************************************************************************************
*** Terraform Destroy Started ***
DateTime: 2022-08-12 13:33:06
****************************************************************************************************Plugin reinitialization required. Please run "terraform init".
Reason: Could not satisfy plugin requirements.
Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.
1 error occurred:
* provider.aws: no suitable version installed
version requirements: "(any version)"
versions installed: none
Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints from each module, run "terraform providers".
Do I need to run "terraform init"?
@estemendoza,
Can you please confirm if you are you running the destroy command from the same machine used for installation?
Thanks.
Yes, I can confirm that, but what can I try to solve this? maybe running provision.sh script again to make sure?
Hi,
Can you share the screenshot of terrafirm directory?
your mean installer/data/terraform
? it's empty
$ ll data/terraform/
total 0
but for installer/core/terraform
is the following:
$ ll core/terraform/
total 20
-rw-rw-r-- 1 ec2-user ec2-user 12188 Aug 3 12:47 __init__.py
drwxr-xr-x 2 root root 65 Aug 12 13:32 __pycache__
drwxrwxr-x 4 ec2-user ec2-user 89 Aug 12 13:32 resources
-rw-rw-r-- 1 ec2-user ec2-user 5461 Aug 3 12:47 utils.py
data/terraform should have the data incase of installation is done on the same machine.
If there is PaladinCloud installation running, you can get the terraform backup from Paladin s3 bucket. Can you please copy the terraform files from the s3 bucket to installer/data directory and try destroy once?
I tried to get the backup from S3 but I couldn't find it, here is the data inside paladin root folder on s3:
and inside paladin cloud folder:
Could this be because the installer was interrupted before finishing correctly and the files weren't uploaded to S3?
On the other hand, I have two folders, one for v1.0.2 and another for v1.1.0. I initially installed Paladin using v1.0.1 but I was checking the terraform folder of v1.1.0. So I copied the files from v1.0.2 to v1.1.0 and tried to run the destroy command again, but I am getting this:
Step 4: Terraform destroy started
Terraform destroy encountered an error
Error occured, please check error log for more details
and in terraform_destroy.log
:
****************************************************************************************************
*** Terraform Destroy Started ***
DateTime: 2022-08-16 16:58:27
****************************************************************************************************Plugin reinitialization required. Please run "terraform init".
Reason: Could not satisfy plugin requirements.
Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.
2 errors occurred:
* provider.null: no suitable version installed
version requirements: "(any version)"
versions installed: none
* provider.aws: no suitable version installed
version requirements: "(any version)"
versions installed: none
Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints from each module, run "terraform providers".
On the terraform directory, I have the following then:
terraform]$ ll
total 772
-rw-r--r-- 1 ec2-user ec2-user 1794 Aug 16 16:26 batch_ecr_RuleEngineDockerImageBuild.tf.json
-rw-r--r-- 1 ec2-user ec2-user 846 Aug 16 16:26 batch_ecr_RuleEngineEcrRepository.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1916 Aug 16 16:26 batch_env_RuleEngineBatchJobEnv.tf.json
-rw-r--r-- 1 ec2-user ec2-user 839 Aug 16 16:26 batch_job_BatchJobsQueue.tf.json
-rw-r--r-- 1 ec2-user ec2-user 862 Aug 16 16:26 batch_job_RuleEngineJobQueue.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1955 Aug 16 16:26 batch_job_SubmitAndRuleEngineJobDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user 109 Aug 16 16:26 data_aws_info_AwsAccount.tf.json
-rw-r--r-- 1 ec2-user ec2-user 99 Aug 16 16:26 data_aws_info_AwsRegion.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1026 Aug 16 16:26 datastore_db_DBOptionGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user 982 Aug 16 16:26 datastore_db_DBParameterGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1054 Aug 16 16:26 datastore_db_DBSubnetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1976 Aug 16 16:26 datastore_db_MySQLDatabase.tf.json
-rw-r--r-- 1 ec2-user ec2-user 900 Aug 16 16:26 datastore_es_ESCloudWatchLogGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1279 Aug 16 16:26 datastore_es_ESCloudWatchLogResourcePolicy.tf.json
-rw-r--r-- 1 ec2-user ec2-user 2254 Aug 16 16:26 datastore_es_ESDomain.tf.json
-rw-r--r-- 1 ec2-user ec2-user 907 Aug 16 16:26 datastore_es_ESDomainPolicy.tf.json
-rw-r--r-- 1 ec2-user ec2-user 0 Aug 16 16:26 errored.tfstate
-rw-r--r-- 1 ec2-user ec2-user 852 Aug 16 16:26 iam_all_read_role_AllReadConfigRolePolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 850 Aug 16 16:26 iam_all_read_role_AllReadIAMFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 869 Aug 16 16:26 iam_all_read_role_AllReadLambdaFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 839 Aug 16 16:26 iam_all_read_role_AllReadOnlyAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1084 Aug 16 16:26 iam_all_read_role_AllReadRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user 950 Aug 16 16:26 iam_all_read_role_AllReadRoleAutoFixPolicy.tf.json
-rw-r--r-- 1 ec2-user ec2-user 869 Aug 16 16:26 iam_all_read_role_AllReadRoleAutoFixPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 4112 Aug 16 16:26 iam_all_read_role_AllReadRoleAutoFixPolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user 849 Aug 16 16:26 iam_all_read_role_AllReadRoleLambdaPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 900 Aug 16 16:26 iam_all_read_role_AllReadRolePolicy.tf.json
-rw-r--r-- 1 ec2-user ec2-user 492 Aug 16 16:26 iam_all_read_role_AllReadRolePolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user 853 Aug 16 16:26 iam_all_read_role_AllReadSupportAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 601 Aug 16 16:26 iam_all_read_role_LambdaPolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user 806 Aug 16 16:26 iam_base_role_BaseAWSSupportPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 832 Aug 16 16:26 iam_base_role_BaseECSTaskExecPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 815 Aug 16 16:26 iam_base_role_BaseGuardDutyPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 820 Aug 16 16:26 iam_base_role_BaseReadOnlyAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1060 Aug 16 16:26 iam_base_role_BaseRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user 939 Aug 16 16:26 iam_base_role_ECSTaskExecutionRolePolicy.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1131 Aug 16 16:26 iam_base_role_ECSTaskExecutionRolePolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1057 Aug 16 16:26 iam_base_role_PolicyDocumentForBaseRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user 903 Aug 16 16:26 iam_base_role_policy_BaseRolePolicy.tf.json
-rw-r--r-- 1 ec2-user ec2-user 485 Aug 16 16:26 iam_base_role_policy_BaseRolePolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user 849 Aug 16 16:26 iam_base_role_policy_BatchBaseRolePolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 837 Aug 16 16:26 iam_base_role_policy_EcsBaseRolePolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 816 Aug 16 16:26 iam_batch_role_BatchAWSSupportPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 820 Aug 16 16:26 iam_batch_role_BatchIAMRolePolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 830 Aug 16 16:26 iam_batch_role_BatchReadOnlyAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1074 Aug 16 16:26 iam_batch_role_BatchRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user 660 Aug 16 16:26 iam_batch_role_BatchRolePolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user 836 Aug 16 16:26 iam_ecs_role_ECSAWSLambdaFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 796 Aug 16 16:26 iam_ecs_role_ECSAWSSupportPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 870 Aug 16 16:26 iam_ecs_role_ECSCloudWatchEventsFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 860 Aug 16 16:26 iam_ecs_role_ECSCloudWatchLogsFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 876 Aug 16 16:26 iam_ecs_role_ECSContainerServiceForEC2PolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 810 Aug 16 16:26 iam_ecs_role_ECSReadOnlyAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1053 Aug 16 16:26 iam_ecs_role_ECSRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user 755 Aug 16 16:26 iam_ecs_role_ECSRoleInstanceProfile.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1434 Aug 16 16:26 iam_ecs_role_ECSRolePolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user 848 Aug 16 16:26 iam_lambda_role_LambdaBatchFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 870 Aug 16 16:26 iam_lambda_role_LambdaCloudWatchFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 830 Aug 16 16:26 iam_lambda_role_LambdaFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1097 Aug 16 16:26 iam_lambda_role_LambdaRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user 706 Aug 16 16:26 iam_lambda_role_LambdaRolePolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1147 Aug 16 16:26 lambda_rule_engine_function_EventRulesLambdaPermissions.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1254 Aug 16 16:26 lambda_rule_engine_function_RuleEngineCloudWatchEventTargets.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1346 Aug 16 16:26 lambda_rule_engine_function_RuleEngineEventRules.tf.json
-rw-r--r-- 1 ec2-user ec2-user 2141 Aug 16 16:26 lambda_rule_engine_function_RuleEngineLambdaFunction.tf.json
-rw-r--r-- 1 ec2-user ec2-user 250004 Aug 16 16:26 lambda_rule_engine_function_RulesListVariable.auto.tfvars
-rw-r--r-- 1 ec2-user ec2-user 1178 Aug 16 16:26 lambda_rule_engine_s3_upload_UploadLambdaRuleEngineZipFile.tf.json
-rw-r--r-- 1 ec2-user ec2-user 2331 Aug 16 16:26 lambda_submit_function_CloudNotificationCollectorCloudWatchEventTarget.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1118 Aug 16 16:26 lambda_submit_function_CloudNotificationCollectorEventRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1213 Aug 16 16:26 lambda_submit_function_CloudNotificationCollectorEventRuleLambdaPermission.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1869 Aug 16 16:26 lambda_submit_function_DataCollectorCloudWatchEventTarget.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1053 Aug 16 16:26 lambda_submit_function_DataCollectorEventRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1135 Aug 16 16:26 lambda_submit_function_DataCollectorEventRuleLambdaPermission.tf.json
-rw-r--r-- 1 ec2-user ec2-user 2368 Aug 16 16:26 lambda_submit_function_DataShipperCloudWatchEventTarget.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1138 Aug 16 16:26 lambda_submit_function_DataShipperEventRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1118 Aug 16 16:26 lambda_submit_function_DataShipperEventRuleLambdaPermission.tf.json
-rw-r--r-- 1 ec2-user ec2-user 2395 Aug 16 16:26 lambda_submit_function_RecommendationsCollectorCloudWatchEventTarget.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1108 Aug 16 16:26 lambda_submit_function_RecommendationsCollectorEventRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1201 Aug 16 16:26 lambda_submit_function_RecommendationsCollectorEventRuleLambdaPermission.tf.json
-rw-r--r-- 1 ec2-user ec2-user 2239 Aug 16 16:26 lambda_submit_function_SubmitJobLambdaFunction.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1138 Aug 16 16:26 lambda_submit_s3_upload_UploadLambdaSubmitJobZipFile.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1347 Aug 16 16:26 pacbot_app_alb_ApplicationLoadBalancer.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1194 Aug 16 16:26 pacbot_app_alb_listener_rules_AdminALBListenerRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1087 Aug 16 16:26 pacbot_app_alb_listener_rules_ApplicationLoadBalancerListener.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1194 Aug 16 16:26 pacbot_app_alb_listener_rules_AssetALBListenerRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1188 Aug 16 16:26 pacbot_app_alb_listener_rules_AuthALBListenerRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1224 Aug 16 16:26 pacbot_app_alb_listener_rules_ComplianceALBListenerRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1200 Aug 16 16:26 pacbot_app_alb_listener_rules_ConfigALBListenerRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1242 Aug 16 16:26 pacbot_app_alb_listener_rules_NotificationsALBListenerRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1224 Aug 16 16:26 pacbot_app_alb_listener_rules_StatisticsALBListenerRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1370 Aug 16 16:26 pacbot_app_alb_target_groups_AdminALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1370 Aug 16 16:26 pacbot_app_alb_target_groups_AssetALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1364 Aug 16 16:26 pacbot_app_alb_target_groups_AuthALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1400 Aug 16 16:26 pacbot_app_alb_target_groups_ComplianceALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1367 Aug 16 16:26 pacbot_app_alb_target_groups_ConfigALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1349 Aug 16 16:26 pacbot_app_alb_target_groups_NginxALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1418 Aug 16 16:26 pacbot_app_alb_target_groups_NotificationsALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1400 Aug 16 16:26 pacbot_app_alb_target_groups_StatisticsALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user 2014 Aug 16 16:26 pacbot_app_build_ui_and_api_BuildUiAndApis.tf.json
-rw-r--r-- 1 ec2-user ec2-user 975 Aug 16 16:26 pacbot_app_cloudwatch_log_groups_ApiCloudWatchLogGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user 969 Aug 16 16:26 pacbot_app_cloudwatch_log_groups_UiCloudWatchLogGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1780 Aug 16 16:26 pacbot_app_ecr_APIDockerImageBuild.tf.json
-rw-r--r-- 1 ec2-user ec2-user 840 Aug 16 16:26 pacbot_app_ecr_APIEcrRepository.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1784 Aug 16 16:26 pacbot_app_ecr_UIDockerImageBuild.tf.json
-rw-r--r-- 1 ec2-user ec2-user 829 Aug 16 16:26 pacbot_app_ecr_UIEcrRepository.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1873 Aug 16 16:26 pacbot_app_ecs_services_AdminEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user 876 Aug 16 16:26 pacbot_app_ecs_services_ApplicationECSCluster.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1873 Aug 16 16:26 pacbot_app_ecs_services_AssetEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1864 Aug 16 16:26 pacbot_app_ecs_services_AuthEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1918 Aug 16 16:26 pacbot_app_ecs_services_ComplianceEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1949 Aug 16 16:26 pacbot_app_ecs_services_ConfigEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1878 Aug 16 16:26 pacbot_app_ecs_services_NginxEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1945 Aug 16 16:26 pacbot_app_ecs_services_NotificationsEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1918 Aug 16 16:26 pacbot_app_ecs_services_StatisticsEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1118 Aug 16 16:26 pacbot_app_ecs_services_WaitConfigServiceToUp.tf.json
-rw-r--r-- 1 ec2-user ec2-user 2576 Aug 16 16:26 pacbot_app_ecs_task_defintions_AdminEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user 2576 Aug 16 16:26 pacbot_app_ecs_task_defintions_AssetEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user 2567 Aug 16 16:26 pacbot_app_ecs_task_defintions_AuthEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user 2621 Aug 16 16:26 pacbot_app_ecs_task_defintions_ComplianceEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user 2592 Aug 16 16:26 pacbot_app_ecs_task_defintions_ConfigEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user 2074 Aug 16 16:26 pacbot_app_ecs_task_defintions_NginxEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user 2647 Aug 16 16:26 pacbot_app_ecs_task_defintions_NotificationsEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user 2621 Aug 16 16:26 pacbot_app_ecs_task_defintions_StatisticsEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user 1215 Aug 16 16:26 pacbot_app_import_db_ImportDbSql.tf.json
-rw-r--r-- 1 ec2-user ec2-user 6777 Aug 16 16:26 pacbot_app_import_db_ReplaceSQLPlaceHolder.tf.json
-rw-r--r-- 1 ec2-user ec2-user 889 Aug 16 16:26 s3_bucket_BucketStorage.tf.json
-rw-r--r-- 1 ec2-user ec2-user 858 Aug 16 16:26 s3_bucket_S3ResourcePolicy.tf.json
-rw-r--r-- 1 ec2-user ec2-user 822 Aug 16 16:26 s3_bucket_S3ResourcePolicyAttachToBaseRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user 856 Aug 16 16:26 s3_bucket_S3ResourcePolicyAttachToBaseRoleToECSRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user 589 Aug 16 16:26 s3_bucket_S3ResourcePolicyDocument.tf.json
drwxr-xr-x 6 ec2-user ec2-user 176 Aug 16 16:26 scripts_and_files
drwxr-xr-x 2 ec2-user ec2-user 6 Aug 16 16:26 upload_to_s3
-rw-r--r-- 1 ec2-user ec2-user 2125 Aug 16 16:26 vpc_security_group_InfraSecurityGroupResource.tf.json
Microservices
Summary
Can't login in the app
Reproduce steps
Expected Results
Actual Results
Getting a "503 Service Temporary Unavailable" error when sending auth data to
/api/auth/user/login