search

SecurityRiskAdvisors / VECTR

VECTR is a tool that facilitates tracking of your red and blue team testing activities to measure detection and prevention capabilities across different attack scenarios
1.34k stars 156 forks source link

Exception thrown on Organisation creation #239

Closed kimberleyhallifax closed 1 year ago

kimberleyhallifax commented 1 year ago

Hey all

So I just want to preface this by saying that I have read your documentation on operating systems such as macOS and kubernetes not being supported (which is indeed what I am running on here), but I'm raising this bug issue in confidence that this exception isn't related to my choice of OS (although I haven't attempted this on any other OS to verify that). But if I'm wrong and it is, please let me know.

Describe the bug

I'm running vectr locally on my macOS and whenever I attempt to create an organisation after logging in, I get this exception.

com.sra.vectr.model.mongo.migrations.exceptions.GoldStandardUnexpectedVersionException: Gold Standard version mismatch during webservice preTaskCheck.
        at com.sra.vectr.rest.service.task.SraWebServiceDatabaseTask.preTaskCheck(SraWebServiceDatabaseTask.java:39)
        at com.sra.vectr.rest.service.task.SraWebServiceTask.run(SraWebServiceTask.java:74)
        at com.sra.vectr.rest.service.impl.OrganizationResource.create(OrganizationResource.java:29)
        at com.sra.vectr.rest.service.impl.OrganizationResource.create(OrganizationResource.java:24)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
        at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:200)
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:103)
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265)
        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:233)
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:207)
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:159)
        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:224)
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:303)
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:216)
        at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590)
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:278)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at com.sra.vectr.libs.servlet.filters.CharacterSetFilter.doFilter(CharacterSetFilter.java:15)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at com.sra.vectr.libs.servlet.filters.CacheControlFilter.doFilter(CacheControlFilter.java:42)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:166)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:676)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:894)
        at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1664)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1219)
        at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$2.completed(Nio2Endpoint.java:630)
        at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$2.completed(Nio2Endpoint.java:608)
        at org.apache.tomcat.util.net.SecureNio2Channel$1.completed(SecureNio2Channel.java:1006)
        at org.apache.tomcat.util.net.SecureNio2Channel$1.completed(SecureNio2Channel.java:934)
        at java.base/sun.nio.ch.Invoker.invokeUnchecked(Unknown Source)
        at java.base/sun.nio.ch.Invoker$2.run(Unknown Source)
        at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(Unknown Source)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Unknown Source)

To Reproduce

You should be running on macOS.

  1. Download the docker compose and .env files by running 
wget https://github.com/SecurityRiskAdvisors/VECTR/releases/download/ce-8.8.1/sra-vectr-runtime-8.8.1-ce.zip`
  1. Update the .env file with the following:
VECTR_HOSTNAME=localhost
VECTR_CONTAINER_LOG_LEVEL=DEBUG
  1. Run the program
docker compose up -d
  1. Login to vectr by visiting https://localhost:8081/ and entering the default root credentials
  2. Create an organisation

Expected behavior

I expected an organisation to be created.

Desktop (please complete the following information):

I'm running macOS Monterey 12.2.1 on Apple M1 Pro.

Additional context

I have also received this exact same error when I've run vectr locally on minikube (kubernetes). I basically converted the docker compose file to a helm chart, reproduced the same steps, and received the same error at the same point. But I'm just raising the issue in relation to the docker compose method because there are less files involved.

I did find some additional logs printed during startup, relating to this GoldStandard (if this is of any help):

2023-07-04 06:41:54,841 INFO [com.sra.vectr.rest.service.impl.VectrService] - GoldStandard does not exist. Importing mongo dump if it exists.
2023-07-04 06:41:57,011 INFO [com.sra.vectr.rest.service.impl.VectrService] - Migrating up GoldStandard before starting VECTR. Please wait...

Here is my docker-compose.yml file:

version: '3.4'

services:
  vectr-mongo:
    image: mongo:4.2
    networks:
      - vectr_bridge
    volumes:
      #- ${VECTR_DATA_DIR:-./data/}${COMPOSE_PROJECT_NAME}:/data/db
      - vectr-db:/data/db
    env_file:
      - ./.env
    restart: unless-stopped

  vectr-tomcat:
    image: securityriskadvisors/vectr_tomcat:8.8.1
    networks:
      vectr_bridge:
        aliases:
          - ${VECTR_HOSTNAME}
    volumes:
      #- ./resources:/opt/vectr/resources
      - vectr-resources:/opt/vectr/resources
      - ./user:/opt/vectr/user
      #- vectr-user:/opt/vectr/user
      #- ./logs:/usr/local/tomcat/logs
      - vectr-logs:/usr/local/tomcat/logs
      - builder-runtimes:/opt/vectr/rta
    env_file:
      - ./.env
    ports:
      - "${VECTR_PORT:-8081}:8443"
    restart: unless-stopped
    depends_on:
      - vectr-mongo

  vectr-rta-webserver:
    image: securityriskadvisors/rta_webserver:1.5.6
    networks:
      - vectr_bridge
    depends_on:
      - vectr-rta-redis
    restart: unless-stopped

  vectr-rta-builder:
    image: securityriskadvisors/rta_builder:1.5.6
    networks:
      - vectr_bridge
    volumes:
      - builder-runtimes:/builds
    env_file:
      - ./.env
    depends_on:
      - vectr-rta-redis
    restart: unless-stopped

  vectr-rta-redis:
    image: redis:6.2-bullseye
    volumes:
      - redis-db:/data
    networks:
      - vectr_bridge
    restart: unless-stopped

networks:
  vectr_bridge:
    ipam:
      driver: default
      config:
        - subnet: "${VECTR_NETWORK_SUBNET:-10.0.27.0/24}"

volumes:
  vectr-logs:
    name: ${COMPOSE_PROJECT_NAME}-vectr-logs
  vectr-resources:
    name: ${COMPOSE_PROJECT_NAME}-vectr-resources
  vectr-user:
    name: ${COMPOSE_PROJECT_NAME}-vectr-user
  vectr-db:
    name: ${COMPOSE_PROJECT_NAME}-vectr-db
  builder-runtimes:
    name: ${COMPOSE_PROJECT_NAME}-builder-runtimes
  redis-db:
    name: ${COMPOSE_PROJECT_NAME}-redis-db

And here is my .env file:

# .env file

APP_NAME=VECTR

VECTR_HOSTNAME=localhost
VECTR_PORT=8081

# defaults to warn, debug useful for development
VECTR_CONTAINER_LOG_LEVEL=WARN

# If you supply a username:password in the URL, it will get overridden by MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_PASSWORD
MONGO_URL=mongodb://vectr-mongo:27017/?authSource=admin

# MongoDB authentication username
MONGO_INITDB_ROOT_USERNAME=admin

# Change and store in a safe place
MONGO_INITDB_ROOT_PASSWORD=Test1234

# PLEASE change this and store it in a safe place.  Encrypted data like passwords
# to integrate with external systems (like TAXII) use this key
VECTR_DATA_KEY=CHANGEMENOW

# JWT signing (JWS) and encryption (JWE) keys
# Do not use the same value for both signing and encryption!
# It is recommended to use at least 16 characters. You may use any printable unicode character
# PLEASE change these example values!
JWS_KEY=WSӠ$8É*X&*8HѲk!^£
JWE_KEY=VПlδ4x%vЋs$fIT@b€

# This sets the name of your project.  Will show up in the name of your containers.
COMPOSE_PROJECT_NAME=sandbox1

# This is where the mongodb mounts.
VECTR_DATA_DIR=/var/data/

Thank you in advance for your help!

SRAPSpencer commented 1 year ago

Hey, thanks for the report. Some follow up questions and comments,

The summary seems like there's an issue with your database. It either didn't migrate to the latest schema correct or somehow it was reverted.

  1. Is this blocking you from taking other actions in the application?
  2. Is this happening on the first login once you stand it up for the first time?
  3. Related, do you have meaningful data in the database you need?

FYI Organizations have no practical function in the system today. They're just a random attribute on a users profile.

If you don't have meaningful data in the system I recommend just deleting your volumes and starting over from scratch.

If this is happening on first stand up with fresh volumes, it's likely related to the deployment method. The system loads the "GoldStandard" you saw in the logs and then migrates it up to current. There's a chance it's loading correctly and some migration is failing, but the system still comes up and somewhat works.

If you have data you need to recover, that would require some offline discussion. Feel free to email the team at, vectrops@sra.io to discuss.

We're more than happy to see community members getting VECTR running on any flavor of k8s. Our "supported platforms" is more due to community support burden and functionality validation than a lack of compatibility. If you'd like to share your work feel free to start a topic under "Discussions" with examples of your Helm chart and such for other community members to engage with. If you don't feel comfortable sharing it publicly you can shoot it to our email above so we can consider rolling it into our public documentation.

kimberleyhallifax commented 1 year ago

Hey @SRAPSpencer, thank you for your speedy response!

Answering your questions:

  1. Is this blocking you from taking other actions in the application?

Yes it is. I can't progress beyond the organisation creation page.

  1. Is this happening on the first login once you stand it up for the first time?

Yep.

  1. Related, do you have meaningful data in the database you need?

No. I'm running this locally for testing purposes before I migrate to AWS EKS (in which case I'll be using EFS volumes for database storage - which persist).

Since you noted that it appears to be a database issue, I did notice a few mongo exceptions in the logs that occur every time I run docker compose, before the webapp launches. Here they are:

java.io.IOException: Cannot run program "mongorestore": error=0, Failed to exec spawn helper: pid: 408, exit value: 1
        at java.base/java.lang.ProcessBuilder.start(Unknown Source)
        at java.base/java.lang.ProcessBuilder.start(Unknown Source)
        at com.sra.vectr.libs.mongodb.MongoOperations.runCommand(MongoOperations.java:349)
        at com.sra.vectr.libs.mongodb.MongoOperations.mongorestore(MongoOperations.java:180)
        at com.sra.vectr.libs.mongodb.NewVectrMongoSetup.primeVectrMongoDatabases(NewVectrMongoSetup.java:63)
        at com.sra.vectr.rest.service.impl.VectrService.primeDatabases(VectrService.java:215)
        at com.sra.vectr.rest.service.impl.VectrService.init(VectrService.java:88)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1869)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1826)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1766)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:598)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:917)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:584)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:394)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:274)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:102)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4454)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4892)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:683)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:658)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:713)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:975)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1949)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
        at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
        at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:776)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:426)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1656)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309)
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
        at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
        at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:898)
        at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:846)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1332)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1322)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
        at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:871)
        at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:241)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:428)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:913)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:795)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:347)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:478)
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 408, exit value: 1
        at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
        at java.base/java.lang.ProcessImpl.<init>(Unknown Source)
        at java.base/java.lang.ProcessImpl.start(Unknown Source)
        ... 67 more
java.io.IOException: Cannot run program "mongorestore": error=0, Failed to exec spawn helper: pid: 411, exit value: 1
        at java.base/java.lang.ProcessBuilder.start(Unknown Source)
        at java.base/java.lang.ProcessBuilder.start(Unknown Source)
        at com.sra.vectr.libs.mongodb.MongoOperations.runCommand(MongoOperations.java:349)
        at com.sra.vectr.libs.mongodb.MongoOperations.mongorestore(MongoOperations.java:114)
        at com.sra.vectr.libs.mongodb.NewVectrMongoSetup.primeVectrMongoDatabase(NewVectrMongoSetup.java:91)
        at com.sra.vectr.rest.service.impl.VectrService.primeSystemData(VectrService.java:240)
        at com.sra.vectr.rest.service.impl.VectrService.init(VectrService.java:92)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1869)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1826)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1766)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:598)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:917)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:584)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:394)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:274)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:102)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4454)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4892)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:683)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:658)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:713)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:975)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1949)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
        at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
        at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:776)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:426)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1656)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309)
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
        at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
        at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:898)
        at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:846)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1332)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1322)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
        at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:871)
        at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:241)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:428)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:913)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:795)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:347)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:478)
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 411, exit value: 1
        at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
        at java.base/java.lang.ProcessImpl.<init>(Unknown Source)
        at java.base/java.lang.ProcessImpl.start(Unknown Source)
        ... 67 more
java.lang.RuntimeException: Cannot run program "mongodump": error=0, Failed to exec spawn helper: pid: 415, exit value: 1
        at com.sra.vectr.model.mongo.migrations.MongoMigrationCallable.orchestrateMigration(MongoMigrationCallable.java:159)
        at com.sra.vectr.model.mongo.migrations.MongoMigrationCallable.call(MongoMigrationCallable.java:94)
        at com.sra.vectr.model.mongo.migrations.MongoMigrationCallable.call(MongoMigrationCallable.java:21)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
2023-07-06 02:13:47,206 ERROR [com.sra.vectr.model.poji.migrations.MigrationCallable] - EXCEPTION IN MIGRATION RUNNER CALL, FAILING OUT.

I took your advice and deleted all volumes and then relaunched the app, and unfortunately still received the same original exception in this issue (as well as the ones above). I also noticed this error right before the exception.

2023-07-06 02:18:06,180 ERROR [com.sra.vectr.auth.pac4j.jwt.JwtManager] - JWT did not contain an SraProfile.class or schema was not valid
SRAPSpencer commented 1 year ago

Yeah, this looks like a environment specific issue. Something prevents the mongorestore from running, which is step 0 in setting up the database for first use. If it can't deploy the database there's no way to workaround that.

This is where other container runtimes and platforms get tricky. The "java.io.ioexception error=0" is a very generic error, it couldn't spawn the process with no indication as to why.

This is a non-exhaustive list of the potential issues.

Could be file permissions with the container runtime not reading mongorestore or being able to execute it.

Could be a run as issue, our containers have a VECTR user they run as. See details here, https://docs.vectr.io/upgrading/non-root-migration/faq/

Could be something specific to the Rosetta translation layer causing the process error. We ship VECTR as x86-64 so running on an ARM Mac wouldn't be translating or emulating in some manner.

If I had to guess it's the "run as" causing issues. K8s, namespaces and such can be a challenge for some containers.

I can leave the issue open if anyone else in the community wants to comment, but this goes beyond what we're able to troubleshoot for the community.

kimberleyhallifax commented 1 year ago

This makes sense @SRAPSpencer, thank you for your help. I'll play around with run as and see what I can do. I'm not all that familiar with mongo so thanks for putting it plainly. No worries at all, I'll close this issue!

SRAPSpencer commented 1 year ago

I'll link this back to our feature request for general k8s support,

https://github.com/SecurityRiskAdvisors/VECTR/issues/128

@kimberleyhallifax feel free to reach out to vectrops@sra.io if you have time to share some of your requirements for deployment and environments. We are actively seeking detailed feedback on the matter as we plan for the future.

kimberleyhallifax commented 1 year ago

Hey @SRAPSpencer, so I gave your advice a go and tried using root as the user in the Dockerfile instead of vectr for the securityriskadvisors/vectr_tomcat:8.8.1 container.

This is the Dockerfile I used:

FROM securityriskadvisors/vectr_tomcat:8.8.1

USER root

RUN chown -R root:root /usr/bin/ && chown -R root:root /usr/local/tomcat/bin/ && chmod 777 /usr/bin/* && chmod 777 /usr/local/tomcat/bin/*

As you can see, I changed ownership of all files used in execution (including the mongodb files) to root and set file permissions to be fully permissive. I then ran docker compose and execd into the container to verify that these config changes were successful. After verifying that, I then still received the same Java IO exception for mongo. I don't think this is related to my OS. If it was, there would be an issue running the docker compose and Dockerfiles, but the containers run fine. Do you have any other suggestions? I tried finding the com.sra.vectr.libs.mongodb.MongoOperations.runCommand(MongoOperations.java:349) file where the exception is thrown but I couldn't find it anywhere online. Is this file looking for the mongodb files in the right place? I'm struggling to see how this is platform related.

thebleucheese commented 1 year ago

This looks like an ARM issue that occurs with Java applications.

https://stackoverflow.com/questions/65962810/m1-mac-issue-bringing-up-elasticsearch-cannot-run-jdk-bin-java

https://github.com/airbytehq/airbyte/issues/9241

It looks like there are a few possible workarounds like changing the java_opts spawn method to vfork.

kimberleyhallifax commented 1 year ago

Omg that worked @thebleucheese! Thank you so much! I was wracking my brain with this for days. Much much appreciated!!