Inconsistencies in Environment Variables and Docker-Compose Configurations

[Thanaphat-Kalaya]

Issue Description I've encountered inconsistencies in the environment variables and Docker-compose configurations that are affecting the deployment and operational stability of the WKS Power platform on remote servers. These inconsistencies make it difficult to predict the behavior of the system under different configurations and can lead to failures during startup.

Steps to Reproduce Follow Installation Guide option 2 and use the provided Docker-compose and environment variable configurations. Attempt to use the docker-full-startup.sh script, expecting a seamless setup, but encounter configuration-related issues instead.

Expected Behavior Following the installation guide should lead to a smooth setup and startup of the WKS Power platform without needing to manually adjust or debug basic environment settings. Environment variables and Docker-compose configurations should align and work out-of-the-box for remote server setups.

Additional Context The issues seem to stem from a lack of synchronization between the Docker-compose files and the environment variables outlined in the .env files. This has led to problems such as incorrect database links, authentication errors with Keycloak, and CORS issues due to misconfigured security headers.

Possible Solution Revising the Docker-compose and .env sample files to ensure that all variables are correctly mapped and documented could resolve these inconsistencies. A review of all related documentation to include clear examples and corrective actions for common errors could also help mitigate these issues.

Tasks

• [ ] update docker-compose.event-hub.yaml references to docker-compose.event-hub.camunda7.yaml in docker-servers-startup.sh and docker-servers-shutdown.sh
• [ ] update Installation guide, Option 2, step 3, where it should be:

  docker-compose -f docker-compose.yaml -f docker-compose.camunda7.yaml -f docker-compose.demo-data-loader.camunda7.yaml -f docker-compose.event-hub.camunda7.yaml -f docker-compose.websocket.yaml -f docker-compose.novu.yaml -f docker-compose.email-sender.yaml up -d --build

• [ ] add env variables to .env-sample under CASE REST API section and add them to docker-compose.portal.yaml
  • REACT_APP_STORAGE_URL
  • REACT_APP_API_URL
  • REACT_APP_KEYCLOAK_URL

@Thanaphat-Kalaya

• the first and second tasks above should fix one issue I found when running Installation guide option 2, step 3. I didn't see you raising that issue yet, but this one instead. Make sure you have updated docker/docker-compose engines and wks-platform codebase develop branch
• Documentation fixes can be applied to https://github.com/wkspower/wks-platform/blob/develop/apps/react/docs/docs/02-Installation/index.mdx

[francav]

I got this result when going through Option 2 - step 3

docker-compose up minio mongodb opa camunda keycloak -d
no such service: camunda

This issue was introduced when refactoring the docker-compose files. The guide should be updated to reflect the changes. The correct command should be the one from https://github.com/wkspower/wks-platform/blob/develop/docker-servers-startup.sh

Is that one of the issues you found? I still didn't see missing variables.

[Thanaphat-Kalaya]

It's not kinda missing. But some variables just being hardcoded in the compose file and cannot set via .env file. To be specific, they are REACT_APP_XXX_URLs in docker-compose.portal.yaml file. Which got me a good time finding out what happened.

Also I got this additional error when trying through Option 2 - step 3. ERROR: The Compose file './docker-compose.yaml' is invalid because: 'name' does not match any of the regexes: '^x-'

[francav]

Insights:

1. Docker-Compose Files for Development/Test Environments:

   • The provided docker-compose files are recommended only for development/test environments.

2. Docker-Compose Files:

   • docker-compose.yaml: Includes essential servers (minio, mongodb, opa, keycloak, storage-api).
   • docker-compose.camunda[CAMUNDA_VERSION].yaml: Adds Camunda servers along with main WKS Platform backend components such as REST-API and External Tasks.
   • docker-compose.demo-data-loader.[CAMUNDA_VERSION].yaml: Introduces the WKS Platform Demo Data Loader app, loading demo case definitions, forms and processes into Camunda and MongoDB and realms, groups and users into Keycloak.
   • docker-compose.event-hub.[CAMUNDA_VERSION].yaml: Optional but necessary for integrating System Notifications and/or websockets for the case-portal.
     • Customizes Camunda with a plugin sending Kafka messages for registered events.
     • Starts Kafka and Zookeeper.
     • Enables Kafka communication from External Tasks.
   • docker-compose.websocket.yaml: Launches websockets for the case-portal.
   • docker-compose.novu.yaml: Initiates novu-publisher.
   • docker-compose.email-sender.yaml: Starts email-sender.
   • docker-compose.portal.yaml: Begins the React case-portal.

3. Shell Scripts for Local Development:

   • The shell scripts are utilities for local development.
   • docker-servers-[startup/shutdown].sh: Useful for loading all servers with Docker while running the React case-portal with yarn, typically for UI development.
   • docker-portal-[startup/shutdown].sh: Opposite of the above, handy for backend development.
   • docker-full-startup.sh: Loads all components from Docker images (defaults to Camunda 7), great for quick tests and demos.
   • docker-full-startup.sh: Similar to the previous script but with Camunda 8, suitable for quick tests and demos.

4. Environment Variables:

   • For Java:

     • Defined in the application.yml files at the root of each Java app (e.g., rest-api application.yml).
     • Can be passed into the app on application startup.
     • Can assume system environment variable values if defined like this one. For instance, camunda.bpm.client.base-url will assume CAMUNDA_BASE_URL if that system variable is available; otherwise, it defaults to http://localhost:8080/engine-rest, referenced within the Java code.
     • See this example of the previous variable overwritten by the docker-compose when including that Java service.
     • Here, the same variable is defined in a .env-sample file.
     • Hence, just as an example, we could apply the same structure to the logging related variables.

   • For React Case-Portal:

     • Defined here.
     • Passed into the app during build time.
     • If you want to start the React app with yarn, you need to copy .env-sample.
     • These variables will be overridden by the ones defined in the docker-compose.
     • To facilitate customization of variables such as this one, currently hard-coded, add it to .env-sample under the appropriate section and update it in the docker-compose file. See this example.

   • For Node Apps: (Using mail-sender app as an example, but the same applies to the other 2 Node apps - websocket publisher and novu-publisher)

     • Defined here.
     • .env-sample for standalone running with yarn here.
     • Docker-compose overrides here.
     • .env-sample for docker-compose here.