PaladinCloud / CE

Paladin Cloud is an extensible, cloud security platform for developers. Join the community at https://discord.gg/xvCFD29Jj4
https://paladincloud.io/
Apache License 2.0
567 stars 53 forks source link

[BUG] Can't login in the app with default credentials #317

Closed estemendoza closed 2 years ago

estemendoza commented 2 years ago

Microservices

Summary

Can't login in the app

Reproduce steps

Expected Results

Actual Results

Getting a "503 Service Temporary Unavailable" error when sending auth data to /api/auth/user/login

santhosh-challa commented 2 years ago

Hi @estemendoza,

Thanks for contacting us. Can you please confirm ECS health for the Paladin cluster? Also, can you please check and let us know if there are any errors in the cloud watch log for auth API?

estemendoza commented 2 years ago

Hi @santhosh-challa

Cluster seems to be working fine:

Screenshot 2022-07-21 at 17 40 34

And these are the last lines on cloud watch logs for auth api:

Screenshot 2022-07-21 at 17 44 27 palading_bug

Let me know if you need me to send you more info for debug.

Thanks

santhosh-challa commented 2 years ago

Thanks for the info. It seems the config service is not running. We can see the running task count is zero and auth is unable to reach the config API. All the APIs need to connect to the config service.

Can you please check the cloudwatch logs for config service and see if there are any errors?

estemendoza commented 2 years ago

I found this for latest messages on the config service logs:

Screenshot 2022-07-21 at 18 58 55
santhosh-challa commented 2 years ago

This seems to be some dependency issue for the config service. We are not seeing this error internally.

Can you please try to redeploy once and see if we can get past this?

estemendoza commented 2 years ago

Hi

I run the redeploy script but I am still getting error, the paladin cloud-config keeps failing with same error

estemendoza commented 2 years ago

By the way, also having this issue on install or redeploy:

Step 5: Deleting resources for redeployment
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/CE-1.0.1/installer/core/providers/aws/reinstall.py", line 111, in show_progress_status_all
    self.render_terraform_destroy_progress()  # Show destroy progress
  File "/home/ubuntu/CE-1.0.1/installer/core/providers/aws/reinstall.py", line 118, in render_terraform_destroy_progress
    while self.destroy is False and self.terraform_thread.isAlive():
AttributeError: 'Thread' object has no attribute 'isAlive'
santhosh-challa commented 2 years ago

Can you please zip and upload the logs directory under install?

That might give us some clue.

estemendoza commented 2 years ago

Here are the logs: installer_logs.zip

santhosh-challa commented 2 years ago

Hi @estemendoza ,

Thank you for the logs. We could not see any errors that are related to the config API issue.

We saw that you are using python 3.9, we tried to redeploy the app on the same version. We are using Python 3.7.x internally. Though we could see the same error on the console as you posted before (thread error related to isAlive), the app is installed successfully and we could not reproduce the config API issue.

This thread issue is happening in Python 3.9.x as isAlive is changed to is_Alive. We will either fix the thread issue or fix the documentation. Thanks for helping us to fix these issues!

To understand the issue with the config API, can you please provide us with the following:

  1. Navigate to S3
  2. click on paladin data bucket, it will be like paladincloud-data-XXXX-XXXXX
  3. Navigate to paladincloud/ directory
  4. Kindly upload the config.jar

Thank you.

estemendoza commented 2 years ago

Ok, you can download the file here: https://dl.meecles.net/config.jar

santhosh-challa commented 2 years ago

I do not see any issues with the jar.

I can run your jar in terminal and I do not see the error. Can you please try running the jar as below in a terminal:

java -jar config.jar

And let us know if you can see the same error or not?

Thank you.

estemendoza commented 2 years ago

Hi. Here is the log for that command:

$ java -jar config.jar
2022-07-28 15:00:37.748  INFO 8969 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@35a50a4c: startup date [Thu Jul 28 15:00:37 UTC 2022]; root of context hierarchy
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (jar:file:/home/ubuntu/config.jar!/BOOT-INF/lib/spring-core-5.0.8.RELEASE.jar!/) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022-07-28 15:00:38.942  INFO 8969 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'configurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$acd60c5d] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.0.4.RELEASE)

2022-07-28 15:00:39.665  INFO 8969 --- [           main] c.t.pacman.config.ConfigApplication      : The following profiles are active: jdbc
2022-07-28 15:00:39.704  INFO 8969 --- [           main] ConfigServletWebServerApplicationContext : Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@5fa07e12: startup date [Thu Jul 28 15:00:39 UTC 2022]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@35a50a4c
2022-07-28 15:00:41.557  INFO 8969 --- [           main] o.s.b.f.s.DefaultListableBeanFactory     : Overriding bean definition for bean 'dataSource' with a different definition: replacing [Root bean: class [null]; scope=refresh; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=false; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.jdbc.DataSourceConfiguration$Hikari; factoryMethodName=dataSource; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/jdbc/DataSourceConfiguration$Hikari.class]] with [Root bean: class [org.springframework.aop.scope.ScopedProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in BeanDefinition defined in class path resource [org/springframework/boot/autoconfigure/jdbc/DataSourceConfiguration$Hikari.class]]
2022-07-28 15:00:42.174  INFO 8969 --- [           main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=2967e5b1-2a2e-3aa2-aecb-c2bf46b0d32c
2022-07-28 15:00:42.440  INFO 8969 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$90bc0960] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2022-07-28 15:00:42.512  INFO 8969 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$acd60c5d] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2022-07-28 15:00:43.422  INFO 8969 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8888 (http)
2022-07-28 15:00:43.492  INFO 8969 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2022-07-28 15:00:43.494  INFO 8969 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/8.5.32
2022-07-28 15:00:43.514  INFO 8969 --- [ost-startStop-1] o.a.catalina.core.AprLifecycleListener   : The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib]
2022-07-28 15:00:43.726  INFO 8969 --- [ost-startStop-1] o.a.c.c.C.[.[localhost].[/api/config]    : Initializing Spring embedded WebApplicationContext
2022-07-28 15:00:43.731  INFO 8969 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 4027 ms
2022-07-28 15:00:45.762  INFO 8969 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'characterEncodingFilter' to: [/*]
2022-07-28 15:00:45.762  INFO 8969 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'webMvcMetricsFilter' to: [/*]
2022-07-28 15:00:45.763  INFO 8969 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2022-07-28 15:00:45.763  INFO 8969 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'httpPutFormContentFilter' to: [/*]
2022-07-28 15:00:45.763  INFO 8969 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'requestContextFilter' to: [/*]
2022-07-28 15:00:45.764  INFO 8969 --- [ost-startStop-1] .s.DelegatingFilterProxyRegistrationBean : Mapping filter: 'springSecurityFilterChain' to: [/*]
2022-07-28 15:00:45.765  INFO 8969 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'httpTraceFilter' to: [/*]
2022-07-28 15:00:45.765  INFO 8969 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean  : Servlet dispatcherServlet mapped to [/]
2022-07-28 15:00:46.036  INFO 8969 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2022-07-28 15:00:46.462  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@5fa07e12: startup date [Thu Jul 28 15:00:39 UTC 2022]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@35a50a4c
2022-07-28 15:00:46.628  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2022-07-28 15:00:46.636  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2022-07-28 15:00:46.660  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/encrypt/status],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> org.springframework.cloud.config.server.encryption.EncryptionController.status()
2022-07-28 15:00:46.661  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/key/{name}/{profiles}],methods=[GET]}" onto public java.lang.String org.springframework.cloud.config.server.encryption.EncryptionController.getPublicKey(java.lang.String,java.lang.String)
2022-07-28 15:00:46.661  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/key],methods=[GET]}" onto public java.lang.String org.springframework.cloud.config.server.encryption.EncryptionController.getPublicKey()
2022-07-28 15:00:46.662  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/encrypt/{name}/{profiles}],methods=[POST]}" onto public java.lang.String org.springframework.cloud.config.server.encryption.EncryptionController.encrypt(java.lang.String,java.lang.String,java.lang.String,org.springframework.http.MediaType)
2022-07-28 15:00:46.662  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/encrypt],methods=[POST]}" onto public java.lang.String org.springframework.cloud.config.server.encryption.EncryptionController.encrypt(java.lang.String,org.springframework.http.MediaType)
2022-07-28 15:00:46.663  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/decrypt],methods=[POST]}" onto public java.lang.String org.springframework.cloud.config.server.encryption.EncryptionController.decrypt(java.lang.String,org.springframework.http.MediaType)
2022-07-28 15:00:46.663  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/decrypt/{name}/{profiles}],methods=[POST]}" onto public java.lang.String org.springframework.cloud.config.server.encryption.EncryptionController.decrypt(java.lang.String,java.lang.String,java.lang.String,org.springframework.http.MediaType)
2022-07-28 15:00:46.675  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{name}-{profiles}.properties],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.lang.String> org.springframework.cloud.config.server.environment.EnvironmentController.properties(java.lang.String,java.lang.String,boolean) throws java.io.IOException
2022-07-28 15:00:46.680  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{name}/{profiles}/{label:.*}],methods=[GET]}" onto public org.springframework.cloud.config.environment.Environment org.springframework.cloud.config.server.environment.EnvironmentController.labelled(java.lang.String,java.lang.String,java.lang.String)
2022-07-28 15:00:46.681  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{name}-{profiles}.yml || /{name}-{profiles}.yaml],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.lang.String> org.springframework.cloud.config.server.environment.EnvironmentController.yaml(java.lang.String,java.lang.String,boolean) throws java.lang.Exception
2022-07-28 15:00:46.682  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{label}/{name}-{profiles}.properties],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.lang.String> org.springframework.cloud.config.server.environment.EnvironmentController.labelledProperties(java.lang.String,java.lang.String,java.lang.String,boolean) throws java.io.IOException
2022-07-28 15:00:46.684  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{name}-{profiles}.json],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.lang.String> org.springframework.cloud.config.server.environment.EnvironmentController.jsonProperties(java.lang.String,java.lang.String,boolean) throws java.lang.Exception
2022-07-28 15:00:46.684  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{label}/{name}-{profiles}.json],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.lang.String> org.springframework.cloud.config.server.environment.EnvironmentController.labelledJsonProperties(java.lang.String,java.lang.String,java.lang.String,boolean) throws java.lang.Exception
2022-07-28 15:00:46.684  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{label}/{name}-{profiles}.yml || /{label}/{name}-{profiles}.yaml],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.lang.String> org.springframework.cloud.config.server.environment.EnvironmentController.labelledYaml(java.lang.String,java.lang.String,java.lang.String,boolean) throws java.lang.Exception
2022-07-28 15:00:46.685  INFO 8969 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/{name}/{profiles:.*[^-].*}],methods=[GET]}" onto public org.springframework.cloud.config.environment.Environment org.springframework.cloud.config.server.environment.EnvironmentController.defaultLabel(java.lang.String,java.lang.String)
2022-07-28 15:00:46.749  INFO 8969 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2022-07-28 15:00:46.749  INFO 8969 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2022-07-28 15:00:48.159  INFO 8969 --- [           main] o.s.s.web.DefaultSecurityFilterChain     : Creating filter chain: org.springframework.security.web.util.matcher.AnyRequestMatcher@1, [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@6e106680, org.springframework.security.web.context.SecurityContextPersistenceFilter@746cd757, org.springframework.security.web.header.HeaderWriterFilter@27c04377, org.springframework.security.web.csrf.CsrfFilter@70242f38, org.springframework.security.web.authentication.logout.LogoutFilter@4567e53d, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@f339eae, org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter@3b0ee03a, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@16c8b7bd, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@77b9d0c7, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@2532b351, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@576c5536, org.springframework.security.web.session.SessionManagementFilter@67403656, org.springframework.security.web.access.ExceptionTranslationFilter@6ed06f69, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@62e7dffa]
2022-07-28 15:00:48.979  INFO 8969 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 2 endpoint(s) beneath base path '/actuator'
2022-07-28 15:00:49.002  INFO 8969 --- [           main] s.b.a.e.w.s.WebMvcEndpointHandlerMapping : Mapped "{[/actuator/health],methods=[GET],produces=[application/vnd.spring-boot.actuator.v2+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(javax.servlet.http.HttpServletRequest,java.util.Map<java.lang.String, java.lang.String>)
2022-07-28 15:00:49.008  INFO 8969 --- [           main] s.b.a.e.w.s.WebMvcEndpointHandlerMapping : Mapped "{[/actuator/info],methods=[GET],produces=[application/vnd.spring-boot.actuator.v2+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(javax.servlet.http.HttpServletRequest,java.util.Map<java.lang.String, java.lang.String>)
2022-07-28 15:00:49.009  INFO 8969 --- [           main] s.b.a.e.w.s.WebMvcEndpointHandlerMapping : Mapped "{[/actuator],methods=[GET],produces=[application/vnd.spring-boot.actuator.v2+json || application/json]}" onto protected java.util.Map<java.lang.String, java.util.Map<java.lang.String, org.springframework.boot.actuate.endpoint.web.Link>> org.springframework.boot.actuate.endpoint.web.servlet.WebMvcEndpointHandlerMapping.links(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2022-07-28 15:00:49.121  INFO 8969 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2022-07-28 15:00:49.129  INFO 8969 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'dataSource' has been autodetected for JMX exposure
2022-07-28 15:00:49.144  INFO 8969 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'environmentManager' has been autodetected for JMX exposure
2022-07-28 15:00:49.153  INFO 8969 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'refreshScope' has been autodetected for JMX exposure
2022-07-28 15:00:49.154  INFO 8969 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'configurationPropertiesRebinder' has been autodetected for JMX exposure
2022-07-28 15:00:49.163  INFO 8969 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located managed bean 'environmentManager': registering with JMX server as MBean [org.springframework.cloud.context.environment:name=environmentManager,type=EnvironmentManager]
2022-07-28 15:00:49.181  INFO 8969 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located managed bean 'refreshScope': registering with JMX server as MBean [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope]
2022-07-28 15:00:49.211  INFO 8969 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=5fa07e12,type=ConfigurationPropertiesRebinder]
2022-07-28 15:00:49.227  INFO 8969 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located MBean 'dataSource': registering with JMX server as MBean [com.zaxxer.hikari:name=dataSource,type=HikariDataSource]
2022-07-28 15:00:49.351  INFO 8969 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8888 (http) with context path '/api/config'
2022-07-28 15:00:49.361  INFO 8969 --- [           main] c.t.pacman.config.ConfigApplication      : Started ConfigApplication in 13.421 seconds (JVM running for 14.921)
santhosh-challa commented 2 years ago

Thanks for the log.

As we can see: 022-07-28 15:00:49.361 INFO 8969 --- [ main] c.t.pacman.config.ConfigApplication : Started ConfigApplication in 13.421 seconds (JVM running for 14.921)

Config API started without any error.

Can you please restart the ECS task for config service and see if it is able to start? Please share the latest log if you see any errors.

Thanks.

estemendoza commented 2 years ago

No problem, what's the proper way of restarting that task? I haven't work with ECS for a while

santhosh-challa commented 2 years ago

You can follow the below steps:

  1. Navigate to ECS
  2. Select the Pladincloud cluster
  3. Click on the Tasks
  4. Select the paladincloud-config task
  5. click on the stop

As this is a FARGATE, ECS automatically starts the task again.

image

Also Please share the cluster screenshot once, just want to check the health.

Thanks

estemendoza commented 2 years ago

This is the status after restarting the task, I waited for a bit, but the task keeps failing

Screenshot 2022-07-28 at 17 32 55
santhosh-challa commented 2 years ago

Can you also share the latest log for it, from the starting?

estemendoza commented 2 years ago

it's basically the same error as before

Screenshot 2022-07-28 at 17 37 25
santhosh-challa commented 2 years ago

can you get the full log, from the start?

estemendoza commented 2 years ago

There is not much of it

Screenshot 2022-07-28 at 17 40 30
santhosh-challa commented 2 years ago

I will try to compare it with the internal log and see if there is anything different. Not sure why the same jar is not able to start inside the container.

santhosh-challa commented 2 years ago

Hi @estemendoza,

Sorry for the delay. I compared your log with the internal log and couldn't find anything interesting.

The error seems coming due to missing of hybetnate validator jar file and the same is in the config jar. We couldn't reproduce the same.

Can you please destroy and install the Paladin Cloud once and see if we can get past this issue?

Thanks

estemendoza commented 2 years ago

Hi @santhosh-challa,

I can try it again, no problem. Question, I am running the installer from an ubuntu instance, I think I didn't specify it before, but I mention it just in case, could this affect the installer somehow?

santhosh-challa commented 2 years ago

Not sure, we will try that once. Can you try with Amazon Linux this time and let us know the results.

estemendoza commented 2 years ago

Hi @santhosh-challa

Why is it recommended to use a t2.large instance type for the installer?

santhosh-challa commented 2 years ago

Hi @estemendoza,

Yes, t2.large is recommended. Also, we had a release happened yesterday. Please check and start the installation from the latest release.

Please let us know in case of any further information is needed.

Thanks, Santhosh

estemendoza commented 2 years ago

I have executed the installer again using Amazon Linux image and using the same auth creds than before getting this error:

Step 1: Checking if required tools are availables
    Tool: Maven, checking ................................................................ [Found]
    Tool: Git, checking .................................................................. [Found]
    Tool: MySQL client, checking ......................................................... [Found]
    Tool: Terraform, checking ............................................................ [Found]
    Tool: Nodejs, checking ............................................................... [Found]
    Tool: npm, checking .................................................................. [Found]
    Tool: Angular, checking .............................................................. [Found]
    Tool: Yarn, checking ................................................................. [Found]
    Tool: Docker, checking ............................................................... [Found]
    Required tools are available!!!

Step 2: Checking if required python packages are available
    Package: docker, Module: Client, checking ............................................ [Found]
    Module: boto3, checking .............................................................. [Found]
    Required python packages are available!!!

Step 3: Reading required inputs from user

    Select AWS authentication mechanism:

    1. Using access key and secret key

    2. Assuming an IAM role

    3. Using IAM role attached to the this instance

    Type 1 or 2 or 3 to continue to create services in AWS: 1
    Required inputs are available!!!

Step 4: Checking settings and inputs
    Checking VPC and CIDR Blocks ......................................................... [Valid]
    Checking VPC subnets ................................................................. [Valid]
    Checking user-attached policies ...................................................... [Not Present]
    Checking group-attached policies ..................................................... [Not Present]

    If you have added custom policies with all permissions, please type Yes or No: yes

Step 5: Checking resource existence
        Resource existence check completed!!!

Step 6: Terraform file generation started
    Error: Either docker service is not available or user do not have permission to run
    Error occured, please check error log for more details
santhosh-challa commented 2 years ago

Can you start the docker service and retry? Maybe it is not running.

sudo systemctl docker start

estemendoza commented 2 years ago

Yeah, the docker service was dead, sudo systemctl start docker fixed the issue, now the installer is running fine, will reply with results.

In the meantime, you should double check how the installer checks that docker is installed and running because even though the docker service was not running, the installer says that docker was found correctly.

santhosh-challa commented 2 years ago

Thanks for the feedback. I am thinking it is checking the availability of Docker in step#1. It is not checking the running state may be. We will check that.

Please post the results.

estemendoza commented 2 years ago

My ssh connection died in the middle of the installer and it failed. When I try to destroy it to install it again, I get this error:

Step 4: Terraform destroy started
        Time elapsed: 0m 7s [..Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ec2-user/CE-1.0.2/installer/core/providers/aws/destroy.py", line 142, in destroy_resources
    PyTerraform.save_terraform_output()
  File "/home/ec2-user/CE-1.0.2/installer/core/terraform/__init__.py", line 239, in save_terraform_output
    output_dict = cls.load_terraform_output()
  File "/home/ec2-user/CE-1.0.2/installer/core/terraform/__init__.py", line 264, in load_terraform_output
    working_dir=Settings.TERRAFORM_DIR,
  File "/home/ec2-user/CE-1.0.2/installer/core/lib/python_terraform/__init__.py", line 109, in __init__
    self.read_state_file(self.state)
  File "/home/ec2-user/CE-1.0.2/installer/core/lib/python_terraform/__init__.py", line 419, in read_state_file
    self.tfstate = Tfstate.load_file(file_path)
  File "/home/ec2-user/CE-1.0.2/installer/core/lib/python_terraform/__init__.py", line 54, in load_file
    json_data = json.load(f)
  File "/usr/lib64/python3.7/json/__init__.py", line 296, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib64/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

        Error occured, please check error log for more details

If I try to install again, I get this:

Step 5: Checking resource existence
        Resource: Aws Iam Role, name: `paladincloud_ro` ...................................... [Exists]
        Resource: Aws Iam Policy, name: `paladincloud_ecs_task_exec` ......................... [Exists]
        Resource: Aws Iam Role, name: `paladincloud_run_batch` ............................... [Exists]
        Resource: Aws Iam Role, name: `paladincloud_ecs_role` ................................ [Exists]
        Resource: Aws Iam Instance Profile, name: `paladincloud_ecs_role_profile` ............ [Exists]
        Resource: Aws Iam Role, name: `paladincloud_lambda_basic_execution` .................. [Exists]
        Resource: Aws Iam Policy, name: `paladincloud_ro` .................................... [Exists]
        Resource: Aws Iam Role, name: `paladincloud` ......................................... [Exists]
        Resource: Aws Iam Policy, name: `paladincloud_pacbot-autofix` ........................ [Exists]
        Resource: Aws Iam Policy, name: `paladincloud` ....................................... [Exists]
        Resource: Aws Security Group, name: `paladincloud` ................................... [Exists]
        Resource: Aws Db Option Group, name: `paladincloud-mysql` ............................ [Exists]
        Resource: Aws Db Parameter Group, name: `paladincloud-mysql` ......................... [Exists]
        Resource: Aws Db Subnet Group, name: `paladincloud-mysql` ............................ [Exists]
        Resource: Aws Db Instance, identifier: `paladincloud-data` ........................... [Exists]
        Resource: Aws Cloudwatch Log Group, name: `paladincloud/elasticsearch` ............... [Exists]
        Resource: Aws Elasticsearch Domain, domain_name: `paladincloud-data` ................. [Exists]
        Resource: Aws Lb, name: `paladincloud` ............................................... [Exists]
        Resource: Aws Alb Target Group, name: `paladincloud-admin` ........................... [Exists]
        Resource: Aws Alb Target Group, name: `paladincloud-asset` ........................... [Exists]
        Resource: Aws Alb Target Group, name: `paladincloud-auth` ............................ [Exists]
        Resource: Aws Alb Target Group, name: `paladincloud-compliance` ...................... [Exists]
        Resource: Aws Alb Target Group, name: `paladincloud-config` .......................... [Exists]
        Resource: Aws Alb Target Group, name: `paladincloud-ngnix` ........................... [Exists]
        Resource: Aws Alb Target Group, name: `paladincloud-notifications` ................... [Exists]
        Resource: Aws Alb Target Group, name: `paladincloud-statistics` ...................... [Exists]
        Resource: Aws Ecr Repository, name: `paladincloud-microservices` ..................... [Exists]
        Resource: Aws Ecr Repository, name: `paladincloud-webapp` ............................ [Exists]
        Resource: Aws Cloudwatch Log Group, name: `paladincloud/apis` ........................ [Exists]
        Resource: Aws Cloudwatch Log Group, name: `paladincloud/ui` .......................... [Exists]
        Resource: Aws Iam Policy, name: `paladincloud_s3` .................................... [Exists]
        Resource: Aws Batch Compute Environment, name: `paladincloud` ........................ [Exists]
        Resource: Aws Ecr Repository, name: `paladincloud-rule-engine` ....................... [Exists]
        Resource: Aws Batch Job Queue, name: `paladincloud-data` ............................. [Exists]
        Resource: Aws Batch Job Queue, name: `paladincloud-rule-engine` ...................... [Exists]
        Resource existence check failed
Please delete all the existing resources or change their names

What can I do to clean up the resources and install again?

santhosh-challa commented 2 years ago

We have pointed out the faqs in the readme as a troubleshooting step. FYI, https://github.com/PaladinCloud/CE/wiki/Installation-FAQs

Can you please run the redeploy and see if it is successful? You can run the commands in the background to avoid connection timeouts.

Please let us know in case you need any further help.

Thanks.

estemendoza commented 2 years ago

Getting this for redeploy:

$ sudo python3 manager.py redeploy

###########################################################################################################################
#######################################  ____                                  ____  ######################################
####################################### |/||\|           PacBot               |/||\| ######################################
####################################### . || .   Infra setup and deployment   . || . ######################################
#######################################   ||                                    ||   ######################################
###########################################################################################################################

Step 1: Checking if required tools are availables
    Tool: Maven, checking ................................................................ [Found]
    Tool: Git, checking .................................................................. [Found]
    Tool: MySQL client, checking ......................................................... [Found]
    Tool: Terraform, checking ............................................................ [Found]
    Tool: Nodejs, checking ............................................................... [Found]
    Tool: npm, checking .................................................................. [Found]
    Tool: Angular, checking .............................................................. [Found]
    Tool: Yarn, checking ................................................................. [Found]
    Tool: Docker, checking ............................................................... [Found]
    Required tools are available!!!

Step 2: Checking if required python packages are available
    Package: docker, Module: Client, checking ............................................ [Found]
    Module: boto3, checking .............................................................. [Found]
    Required python packages are available!!!

Step 3: Reading required inputs from user

    Select AWS authentication mechanism:

    1. Using access key and secret key

    2. Assuming an IAM role

    3. Using IAM role attached to the this instance

    Type 1 or 2 or 3 to continue to create services in AWS: 1
    Required inputs are available!!!
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ec2-user/CE-1.0.2/installer/custom/commands/redeploy.py", line 165, in run_reinstallation
    self.generate_terraform_files_and_upgrade_state(input_instance)
  File "/home/ec2-user/CE-1.0.2/installer/custom/commands/redeploy.py", line 153, in generate_terraform_files_and_upgrade_state
    PyTerraform.terrafomr12_upgrade()  # This is required only when terraform version 12 is used
  File "/home/ec2-user/CE-1.0.2/installer/core/terraform/__init__.py", line 352, in terrafomr12_upgrade
    working_dir=Settings.TERRAFORM_DIR,
  File "/home/ec2-user/CE-1.0.2/installer/core/lib/python_terraform/__init__.py", line 109, in __init__
    self.read_state_file(self.state)
  File "/home/ec2-user/CE-1.0.2/installer/core/lib/python_terraform/__init__.py", line 419, in read_state_file
    self.tfstate = Tfstate.load_file(file_path)
  File "/home/ec2-user/CE-1.0.2/installer/core/lib/python_terraform/__init__.py", line 54, in load_file
    json_data = json.load(f)
  File "/usr/lib64/python3.7/json/__init__.py", line 296, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib64/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
santhosh-challa commented 2 years ago

Can you please upload the logs from the installer/log directory?

estemendoza commented 2 years ago

Sure, here they are: logs.zip

santhosh-challa commented 2 years ago

Hi @estemendoza ,

Found below error on maven build log:

Failed to execute goal on project gcp-discovery: Could not resolve dependencies for project com.paladincloud:gcp-discovery:jar:0.0.1-SNAPSHOT: The following artifacts could not be resolved: io.grpc:grpc-netty-shaded:jar:1.44.1, io.grpc:grpc-xds:jar:1.47.0, io.opencensus:opencensus-proto:jar:0.2.0, com.google.api:gax:jar:2.12.2: Could not transfer artifact io.grpc:grpc-netty-shaded:jar:1.44.1 from/to central (https://repo.maven.apache.org/maven2): GET request of: io/grpc/grpc-netty-shaded/1.44.1/grpc-netty-shaded-1.44.1.jar from central failed: No space left on device -> [Help 1]

It seems, there is no space left on the installer machine. Can you please check and retry the process?

Thanks, Santhosh

estemendoza commented 2 years ago

Yes, I saw that and fixed the issue before, but I tried again the redeploy command and still getting that same error as before

santhosh-challa commented 2 years ago

Can you please confirm the free space on the instance once? What is the size you are using for installation machine?

estemendoza commented 2 years ago

Yep, I have this

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  576K  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/xvda1       20G   11G   10G  51% /
lakspp commented 2 years ago

@estemendoza can you try to install the latest release of Paladin Cloud release 1.1.0 and check if the issue persists?

estemendoza commented 2 years ago

@lakspp sure, no problem, do I need to run a "destroy" command or "install" directly? Because there are still a lot of resources created from the previous installation attempt

lakspp commented 2 years ago

yes, do destroy and install again.

estemendoza commented 2 years ago

@lakspp I am getting this error now with destroy:

****************************************************************************************************
*** Terraform Destroy  Started ***
DateTime: 2022-08-12 13:33:06
****************************************************************************************************Plugin reinitialization required. Please run "terraform init".
Reason: Could not satisfy plugin requirements.

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

1 error occurred:
        * provider.aws: no suitable version installed
  version requirements: "(any version)"
  versions installed: none

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints from each module, run "terraform providers".

Do I need to run "terraform init"?

santhosh-challa commented 2 years ago

@estemendoza,

Can you please confirm if you are you running the destroy command from the same machine used for installation?

Thanks.

estemendoza commented 2 years ago

Yes, I can confirm that, but what can I try to solve this? maybe running provision.sh script again to make sure?

santhosh-challa commented 2 years ago

Hi,

Can you share the screenshot of terrafirm directory?

estemendoza commented 2 years ago

your mean installer/data/terraform? it's empty

$ ll data/terraform/
total 0

but for installer/core/terraform is the following:

$ ll core/terraform/
total 20
-rw-rw-r-- 1 ec2-user ec2-user 12188 Aug  3 12:47 __init__.py
drwxr-xr-x 2 root     root        65 Aug 12 13:32 __pycache__
drwxrwxr-x 4 ec2-user ec2-user    89 Aug 12 13:32 resources
-rw-rw-r-- 1 ec2-user ec2-user  5461 Aug  3 12:47 utils.py
santhosh-challa commented 2 years ago

data/terraform should have the data incase of installation is done on the same machine.

If there is PaladinCloud installation running, you can get the terraform backup from Paladin s3 bucket. Can you please copy the terraform files from the s3 bucket to installer/data directory and try destroy once?

estemendoza commented 2 years ago

I tried to get the backup from S3 but I couldn't find it, here is the data inside paladin root folder on s3:

Screenshot 2022-08-16 at 18 41 22

and inside paladin cloud folder:

Screenshot 2022-08-16 at 18 41 41

Could this be because the installer was interrupted before finishing correctly and the files weren't uploaded to S3?

On the other hand, I have two folders, one for v1.0.2 and another for v1.1.0. I initially installed Paladin using v1.0.1 but I was checking the terraform folder of v1.1.0. So I copied the files from v1.0.2 to v1.1.0 and tried to run the destroy command again, but I am getting this:

Step 4: Terraform destroy started
        Terraform destroy encountered an error
    Error occured, please check error log for more details

and in terraform_destroy.log:

****************************************************************************************************
*** Terraform Destroy  Started ***
DateTime: 2022-08-16 16:58:27
****************************************************************************************************Plugin reinitialization required. Please run "terraform init".
Reason: Could not satisfy plugin requirements.

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

2 errors occurred:
    * provider.null: no suitable version installed
  version requirements: "(any version)"
  versions installed: none
    * provider.aws: no suitable version installed
  version requirements: "(any version)"
  versions installed: none

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints from each module, run "terraform providers".

On the terraform directory, I have the following then:

terraform]$ ll
total 772
-rw-r--r-- 1 ec2-user ec2-user   1794 Aug 16 16:26 batch_ecr_RuleEngineDockerImageBuild.tf.json
-rw-r--r-- 1 ec2-user ec2-user    846 Aug 16 16:26 batch_ecr_RuleEngineEcrRepository.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1916 Aug 16 16:26 batch_env_RuleEngineBatchJobEnv.tf.json
-rw-r--r-- 1 ec2-user ec2-user    839 Aug 16 16:26 batch_job_BatchJobsQueue.tf.json
-rw-r--r-- 1 ec2-user ec2-user    862 Aug 16 16:26 batch_job_RuleEngineJobQueue.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1955 Aug 16 16:26 batch_job_SubmitAndRuleEngineJobDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user    109 Aug 16 16:26 data_aws_info_AwsAccount.tf.json
-rw-r--r-- 1 ec2-user ec2-user     99 Aug 16 16:26 data_aws_info_AwsRegion.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1026 Aug 16 16:26 datastore_db_DBOptionGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user    982 Aug 16 16:26 datastore_db_DBParameterGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1054 Aug 16 16:26 datastore_db_DBSubnetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1976 Aug 16 16:26 datastore_db_MySQLDatabase.tf.json
-rw-r--r-- 1 ec2-user ec2-user    900 Aug 16 16:26 datastore_es_ESCloudWatchLogGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1279 Aug 16 16:26 datastore_es_ESCloudWatchLogResourcePolicy.tf.json
-rw-r--r-- 1 ec2-user ec2-user   2254 Aug 16 16:26 datastore_es_ESDomain.tf.json
-rw-r--r-- 1 ec2-user ec2-user    907 Aug 16 16:26 datastore_es_ESDomainPolicy.tf.json
-rw-r--r-- 1 ec2-user ec2-user      0 Aug 16 16:26 errored.tfstate
-rw-r--r-- 1 ec2-user ec2-user    852 Aug 16 16:26 iam_all_read_role_AllReadConfigRolePolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    850 Aug 16 16:26 iam_all_read_role_AllReadIAMFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    869 Aug 16 16:26 iam_all_read_role_AllReadLambdaFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    839 Aug 16 16:26 iam_all_read_role_AllReadOnlyAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1084 Aug 16 16:26 iam_all_read_role_AllReadRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user    950 Aug 16 16:26 iam_all_read_role_AllReadRoleAutoFixPolicy.tf.json
-rw-r--r-- 1 ec2-user ec2-user    869 Aug 16 16:26 iam_all_read_role_AllReadRoleAutoFixPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user   4112 Aug 16 16:26 iam_all_read_role_AllReadRoleAutoFixPolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user    849 Aug 16 16:26 iam_all_read_role_AllReadRoleLambdaPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    900 Aug 16 16:26 iam_all_read_role_AllReadRolePolicy.tf.json
-rw-r--r-- 1 ec2-user ec2-user    492 Aug 16 16:26 iam_all_read_role_AllReadRolePolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user    853 Aug 16 16:26 iam_all_read_role_AllReadSupportAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    601 Aug 16 16:26 iam_all_read_role_LambdaPolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user    806 Aug 16 16:26 iam_base_role_BaseAWSSupportPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    832 Aug 16 16:26 iam_base_role_BaseECSTaskExecPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    815 Aug 16 16:26 iam_base_role_BaseGuardDutyPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    820 Aug 16 16:26 iam_base_role_BaseReadOnlyAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1060 Aug 16 16:26 iam_base_role_BaseRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user    939 Aug 16 16:26 iam_base_role_ECSTaskExecutionRolePolicy.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1131 Aug 16 16:26 iam_base_role_ECSTaskExecutionRolePolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1057 Aug 16 16:26 iam_base_role_PolicyDocumentForBaseRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user    903 Aug 16 16:26 iam_base_role_policy_BaseRolePolicy.tf.json
-rw-r--r-- 1 ec2-user ec2-user    485 Aug 16 16:26 iam_base_role_policy_BaseRolePolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user    849 Aug 16 16:26 iam_base_role_policy_BatchBaseRolePolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    837 Aug 16 16:26 iam_base_role_policy_EcsBaseRolePolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    816 Aug 16 16:26 iam_batch_role_BatchAWSSupportPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    820 Aug 16 16:26 iam_batch_role_BatchIAMRolePolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    830 Aug 16 16:26 iam_batch_role_BatchReadOnlyAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1074 Aug 16 16:26 iam_batch_role_BatchRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user    660 Aug 16 16:26 iam_batch_role_BatchRolePolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user    836 Aug 16 16:26 iam_ecs_role_ECSAWSLambdaFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    796 Aug 16 16:26 iam_ecs_role_ECSAWSSupportPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    870 Aug 16 16:26 iam_ecs_role_ECSCloudWatchEventsFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    860 Aug 16 16:26 iam_ecs_role_ECSCloudWatchLogsFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    876 Aug 16 16:26 iam_ecs_role_ECSContainerServiceForEC2PolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    810 Aug 16 16:26 iam_ecs_role_ECSReadOnlyAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1053 Aug 16 16:26 iam_ecs_role_ECSRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user    755 Aug 16 16:26 iam_ecs_role_ECSRoleInstanceProfile.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1434 Aug 16 16:26 iam_ecs_role_ECSRolePolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user    848 Aug 16 16:26 iam_lambda_role_LambdaBatchFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    870 Aug 16 16:26 iam_lambda_role_LambdaCloudWatchFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user    830 Aug 16 16:26 iam_lambda_role_LambdaFullAccessPolicyAttach.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1097 Aug 16 16:26 iam_lambda_role_LambdaRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user    706 Aug 16 16:26 iam_lambda_role_LambdaRolePolicyDocument.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1147 Aug 16 16:26 lambda_rule_engine_function_EventRulesLambdaPermissions.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1254 Aug 16 16:26 lambda_rule_engine_function_RuleEngineCloudWatchEventTargets.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1346 Aug 16 16:26 lambda_rule_engine_function_RuleEngineEventRules.tf.json
-rw-r--r-- 1 ec2-user ec2-user   2141 Aug 16 16:26 lambda_rule_engine_function_RuleEngineLambdaFunction.tf.json
-rw-r--r-- 1 ec2-user ec2-user 250004 Aug 16 16:26 lambda_rule_engine_function_RulesListVariable.auto.tfvars
-rw-r--r-- 1 ec2-user ec2-user   1178 Aug 16 16:26 lambda_rule_engine_s3_upload_UploadLambdaRuleEngineZipFile.tf.json
-rw-r--r-- 1 ec2-user ec2-user   2331 Aug 16 16:26 lambda_submit_function_CloudNotificationCollectorCloudWatchEventTarget.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1118 Aug 16 16:26 lambda_submit_function_CloudNotificationCollectorEventRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1213 Aug 16 16:26 lambda_submit_function_CloudNotificationCollectorEventRuleLambdaPermission.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1869 Aug 16 16:26 lambda_submit_function_DataCollectorCloudWatchEventTarget.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1053 Aug 16 16:26 lambda_submit_function_DataCollectorEventRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1135 Aug 16 16:26 lambda_submit_function_DataCollectorEventRuleLambdaPermission.tf.json
-rw-r--r-- 1 ec2-user ec2-user   2368 Aug 16 16:26 lambda_submit_function_DataShipperCloudWatchEventTarget.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1138 Aug 16 16:26 lambda_submit_function_DataShipperEventRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1118 Aug 16 16:26 lambda_submit_function_DataShipperEventRuleLambdaPermission.tf.json
-rw-r--r-- 1 ec2-user ec2-user   2395 Aug 16 16:26 lambda_submit_function_RecommendationsCollectorCloudWatchEventTarget.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1108 Aug 16 16:26 lambda_submit_function_RecommendationsCollectorEventRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1201 Aug 16 16:26 lambda_submit_function_RecommendationsCollectorEventRuleLambdaPermission.tf.json
-rw-r--r-- 1 ec2-user ec2-user   2239 Aug 16 16:26 lambda_submit_function_SubmitJobLambdaFunction.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1138 Aug 16 16:26 lambda_submit_s3_upload_UploadLambdaSubmitJobZipFile.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1347 Aug 16 16:26 pacbot_app_alb_ApplicationLoadBalancer.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1194 Aug 16 16:26 pacbot_app_alb_listener_rules_AdminALBListenerRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1087 Aug 16 16:26 pacbot_app_alb_listener_rules_ApplicationLoadBalancerListener.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1194 Aug 16 16:26 pacbot_app_alb_listener_rules_AssetALBListenerRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1188 Aug 16 16:26 pacbot_app_alb_listener_rules_AuthALBListenerRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1224 Aug 16 16:26 pacbot_app_alb_listener_rules_ComplianceALBListenerRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1200 Aug 16 16:26 pacbot_app_alb_listener_rules_ConfigALBListenerRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1242 Aug 16 16:26 pacbot_app_alb_listener_rules_NotificationsALBListenerRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1224 Aug 16 16:26 pacbot_app_alb_listener_rules_StatisticsALBListenerRule.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1370 Aug 16 16:26 pacbot_app_alb_target_groups_AdminALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1370 Aug 16 16:26 pacbot_app_alb_target_groups_AssetALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1364 Aug 16 16:26 pacbot_app_alb_target_groups_AuthALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1400 Aug 16 16:26 pacbot_app_alb_target_groups_ComplianceALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1367 Aug 16 16:26 pacbot_app_alb_target_groups_ConfigALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1349 Aug 16 16:26 pacbot_app_alb_target_groups_NginxALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1418 Aug 16 16:26 pacbot_app_alb_target_groups_NotificationsALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1400 Aug 16 16:26 pacbot_app_alb_target_groups_StatisticsALBTargetGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user   2014 Aug 16 16:26 pacbot_app_build_ui_and_api_BuildUiAndApis.tf.json
-rw-r--r-- 1 ec2-user ec2-user    975 Aug 16 16:26 pacbot_app_cloudwatch_log_groups_ApiCloudWatchLogGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user    969 Aug 16 16:26 pacbot_app_cloudwatch_log_groups_UiCloudWatchLogGroup.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1780 Aug 16 16:26 pacbot_app_ecr_APIDockerImageBuild.tf.json
-rw-r--r-- 1 ec2-user ec2-user    840 Aug 16 16:26 pacbot_app_ecr_APIEcrRepository.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1784 Aug 16 16:26 pacbot_app_ecr_UIDockerImageBuild.tf.json
-rw-r--r-- 1 ec2-user ec2-user    829 Aug 16 16:26 pacbot_app_ecr_UIEcrRepository.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1873 Aug 16 16:26 pacbot_app_ecs_services_AdminEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user    876 Aug 16 16:26 pacbot_app_ecs_services_ApplicationECSCluster.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1873 Aug 16 16:26 pacbot_app_ecs_services_AssetEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1864 Aug 16 16:26 pacbot_app_ecs_services_AuthEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1918 Aug 16 16:26 pacbot_app_ecs_services_ComplianceEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1949 Aug 16 16:26 pacbot_app_ecs_services_ConfigEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1878 Aug 16 16:26 pacbot_app_ecs_services_NginxEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1945 Aug 16 16:26 pacbot_app_ecs_services_NotificationsEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1918 Aug 16 16:26 pacbot_app_ecs_services_StatisticsEcsService.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1118 Aug 16 16:26 pacbot_app_ecs_services_WaitConfigServiceToUp.tf.json
-rw-r--r-- 1 ec2-user ec2-user   2576 Aug 16 16:26 pacbot_app_ecs_task_defintions_AdminEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user   2576 Aug 16 16:26 pacbot_app_ecs_task_defintions_AssetEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user   2567 Aug 16 16:26 pacbot_app_ecs_task_defintions_AuthEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user   2621 Aug 16 16:26 pacbot_app_ecs_task_defintions_ComplianceEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user   2592 Aug 16 16:26 pacbot_app_ecs_task_defintions_ConfigEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user   2074 Aug 16 16:26 pacbot_app_ecs_task_defintions_NginxEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user   2647 Aug 16 16:26 pacbot_app_ecs_task_defintions_NotificationsEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user   2621 Aug 16 16:26 pacbot_app_ecs_task_defintions_StatisticsEcsTaskDefinition.tf.json
-rw-r--r-- 1 ec2-user ec2-user   1215 Aug 16 16:26 pacbot_app_import_db_ImportDbSql.tf.json
-rw-r--r-- 1 ec2-user ec2-user   6777 Aug 16 16:26 pacbot_app_import_db_ReplaceSQLPlaceHolder.tf.json
-rw-r--r-- 1 ec2-user ec2-user    889 Aug 16 16:26 s3_bucket_BucketStorage.tf.json
-rw-r--r-- 1 ec2-user ec2-user    858 Aug 16 16:26 s3_bucket_S3ResourcePolicy.tf.json
-rw-r--r-- 1 ec2-user ec2-user    822 Aug 16 16:26 s3_bucket_S3ResourcePolicyAttachToBaseRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user    856 Aug 16 16:26 s3_bucket_S3ResourcePolicyAttachToBaseRoleToECSRole.tf.json
-rw-r--r-- 1 ec2-user ec2-user    589 Aug 16 16:26 s3_bucket_S3ResourcePolicyDocument.tf.json
drwxr-xr-x 6 ec2-user ec2-user    176 Aug 16 16:26 scripts_and_files
drwxr-xr-x 2 ec2-user ec2-user      6 Aug 16 16:26 upload_to_s3
-rw-r--r-- 1 ec2-user ec2-user   2125 Aug 16 16:26 vpc_security_group_InfraSecurityGroupResource.tf.json