GKE Web UI 404 Archival Using bucket

[dhiaayachi]

Web UI 404 when accessing detail of workflow id from archival

Here is the English translation of your issue:

---

I am using Helm to deploy to GKE with the values.archival.gcloud.yaml as follows:

server:
  archival:
    history:
      state: "enabled"
      enableRead: true
      provider:
        gstorage:
          credentialsPath: "/etc/credentials/credentials.json"
    visibility:
      state: "enabled"
      enableRead: true
      provider:
        gstorage:
          credentialsPath: "/etc/credentials/credentials.json"

  namespaceDefaults:
    archival:
      history:
        state: "enabled"
        URI: "gs://kp-temporal-archival/temporal_archival"
      visibility:
        state: "enabled"
        URI: "gs://kp-temporal-archival/temporal_visibility"

For persistence, I am using PostgreSQL with the following values.postgresql.yaml:

server:
  config:
    persistence:
      default:
        driver: "sql"

        sql:
          driver: "postgres12"
          host: postgres.example.dev
          port: 5432
          database: temporal
          user: temporal
          password: temporal
          maxConns: 20
          maxConnLifetime: "1h"

      visibility:
        driver: "sql"

        sql:
          driver: "postgres12"
          host: postgres.example.dev
          port: 5432
          database: temporal_visibility
          user: temporal
          password: temporal
          maxConns: 20
          maxConnLifetime: "1h"

cassandra:
  enabled: false

mysql:
  enabled: false

postgresql:
  enabled: true

prometheus:
  enabled: false

grafana:
  enabled: false

elasticsearch:
  enabled: true

schema:
  createDatabase:
    enabled: false
  setup:
    enabled: true
  update:
    enabled: true

Then, I forward the Web UI using Internal Ingress with the following configuration:

web:
  enabled: true
  replicaCount: 1
  image:
    repository: temporalio/ui
    tag: 2.28.0
    pullPolicy: IfNotPresent
  service:
    type: ClusterIP
    port: 8080
    annotations: {}
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.regional-static-ip-name: ingress-temporal-web
      kubernetes.io/ingress.allow-http: "true"
      kubernetes.io/ingress.class: "gce-internal"
    hosts:
      - "temporal.example.dev"
    tls:
      - secretName: temporal
        hosts:
          - temporal.example.dev

The archival and visibility files are already in the bucket.

and this is my version server 1.24.2

[dhiaayachi]

Thanks for reporting this issue!

Can you please provide the following information so I can help you further:

• What version of the Temporal UI are you using?
• Is there anything specific you've done to configure your Temporal server?
• Do you have any custom components or configurations set up that could be contributing to this issue?

Once I have this information, I can try to find a solution based on the attached documents.

[dhiaayachi]

Let's break down the Temporal deployment process you described. Here's a comprehensive guide with explanations and recommendations:

Introduction

Temporal is a powerful workflow orchestration platform that helps build robust and scalable applications. The deployment process you described involves running Temporal as a standalone service with a reverse proxy for external access. This configuration provides flexibility and control over your workflow environment.

Prerequisites

Before you begin, make sure you have the following:

• A Linux Server: Temporal is primarily designed for Linux environments. You can use a cloud server (AWS EC2, Google Cloud VM, etc.) or a local Linux machine.
• A Domain Name: This is required for external access to your Temporal service. You can register a domain name through a registrar like GoDaddy, Namecheap, or Google Domains.
• SSL Certificates: For secure HTTPS connections, you'll need valid SSL certificates for your domain. You can obtain certificates through services like Let's Encrypt (using Certbot) or a domain registrar.
• Basic Linux Knowledge: Familiarity with the command line, file system navigation, and basic system administration tasks is helpful.

Obtaining the Temporal Binaries

• Temporal Server: Download the latest binary for your Linux system from the Temporal Github Releases page: https://github.com/temporalio/temporal/releases/
• Temporal UI Server: Download the latest binary for your Linux system from the Temporal UI Server Github Releases page: https://github.com/temporalio/ui-server/releases/

Configuring the Temporal Binaries

1. Temporal Server Configuration: Create a temporal-server.yaml file in a directory of your choice (e.g., /etc/temporal). Here's a basic configuration that uses SQLite for persistence:

   log:
    stdout: true
    level: info
   
   persistence:
    defaultStore: sqlite-default
    visibilityStore: sqlite-visibility
    numHistoryShards: 4
    datastores:
      sqlite-default:
        sql:
          pluginName: "sqlite"
          databaseName: "/etc/temporal/default.db"
          connectAddr: "localhost"
          connectProtocol: "tcp"
          connectAttributes:
            cache: "private"
            setup: true
   
      sqlite-visibility:
        sql:
          pluginName: "sqlite"
          databaseName: "/etc/temporal/visibility.db"
          connectAddr: "localhost"
          connectProtocol: "tcp"
          connectAttributes:
            cache: "private"
            setup: true
   
   global:
    membership:
      maxJoinDuration: 30s
      broadcastAddress: "127.0.0.1"
    pprof:
      port: 7936
   
   services:
    frontend:
      rpc:
        grpcPort: 7236
        membershipPort: 6933
        bindOnLocalHost: true
        httpPort: 7243
   
    matching:
      rpc:
        grpcPort: 7235
        membershipPort: 6935
        bindOnLocalHost: true
   
    history:
      rpc:
        grpcPort: 7234
        membershipPort: 6934
        bindOnLocalHost: true
   
    worker:
      rpc:
        membershipPort: 6939
   
   clusterMetadata:
    enableGlobalNamespace: false
    failoverVersionIncrement: 10
    masterClusterName: "active"
    currentClusterName: "active"
    clusterInformation:
      active:
        enabled: true
        initialFailoverVersion: 1
        rpcName: "frontend"
        rpcAddress: "localhost:7236"
        httpAddress: "localhost:7243"
   
   dcRedirectionPolicy:
    policy: "noop"

2. Temporal UI Server Configuration: Create a temporal-ui-server.yaml file in the same directory. Here's a basic configuration:

   temporalGrpcAddress: 127.0.0.1:7236
   host: 127.0.0.1
   port: 8233
   enableUi: true
   cors:
    allowOrigins:
      - http://localhost:8233
   defaultNamespace: default

Creating and Registering System Services

1. Temporal Server Service: Create a temporal.service file in /etc/systemd/system/. This file defines the Temporal server as a system service:

   [Unit]
   Description=Temporal Service
   After=network.target
   
   [Service]
   User=temporal
   Group=temporal
   ExecStart=/usr/bin/temporal-server -r / -c /etc/temporal/ -e temporal-server start
   
   [Install]
   WantedBy=multi-user.target

   • Important: Replace /usr/bin/temporal-server with the actual path where you placed the downloaded Temporal server binary.

2. Temporal UI Server Service: Create a temporal-ui.service file in /etc/systemd/system/:

   [Unit]
   Description=Temporal UI Server
   After=network.target
   
   [Service]
   User=temporal
   Group=temporal
   ExecStart=/usr/bin/temporal-ui-server -r / -c /etc/temporal/ -e temporal-ui-server start
   
   [Install]
   WantedBy=multi-user.target

   • Important: Replace /usr/bin/temporal-ui-server with the actual path where you placed the downloaded UI server binary.

3. Enable and Start Services:

   sudo systemctl enable temporal
   sudo systemctl start temporal
   
   sudo systemctl enable temporal-ui
   sudo systemctl start temporal-ui

Deploying an Nginx Reverse Proxy with an SSL Certificate

1. Install Nginx: If Nginx is not already installed, use your Linux distribution's package manager to install it. For example, on Ubuntu/Debian:

   sudo apt update
   sudo apt install nginx

2. Obtain SSL Certificates (Certbot): Use Certbot to automatically obtain and install SSL certificates from Let's Encrypt:

   sudo apt install certbot
   sudo certbot certonly --standalone -d YOUR_DOMAIN

   • Replace YOUR_DOMAIN with your actual domain name.

3. Nginx Configuration (Temporal UI): Create a temporal-ui configuration file in /etc/nginx/sites-available/:

   server {
      listen 80 default_server;
      listen [::]:80 default_server;
      server_name YOUR_DOMAIN www.YOUR_DOMAIN
   
      access_log /var/log/nginx/temporal.access.log;
      error_log /var/log/nginx/temporal.error.log;
   
      location / {
          proxy_pass http://127.0.0.1:8233;
          proxy_http_version 1.1;
          proxy_read_timeout 300;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection "upgrade";
          proxy_set_header Host $http_host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Real-PORT $remote_port;
          # Replace with your allowed client IPs
          allow 192.168.1.100;
          deny all;
      }
   
      listen 443 ssl;
      ssl_certificate /etc/letsencrypt/live/YOUR_DOMAIN/fullchain.pem;
      ssl_certificate_key /etc/letsencrypt/live/YOUR_DOMAIN/privkey.pem;
   
      # Redirect non-https traffic to https
      if ($scheme != "https") {
          return 301 https://$host$request_uri;
      }
   }

   • Important: Replace 192.168.1.100 with your allowed client IP address(es) or IP range.

4. Nginx Configuration (Temporal gRPC): Create a temporal configuration file in /etc/nginx/sites-available/:

   server {
      listen 7233 http2;
      listen [::]:7233 http2;
      server_name YOUR_DOMAIN
   
      http2 on;
   
      location / {
          grpc_pass localhost:7236;
          # Replace with your allowed client IPs
          allow 192.168.1.100;
          deny all;
      }
   }

   • Important: Replace 192.168.1.100 with your allowed client IP address(es) or IP range (same as for Temporal UI).

5. Enable and Reload Nginx:

   sudo ln -s /etc/nginx/sites-available/temporal-ui /etc/nginx/sites-enabled/temporal-ui
   sudo ln -s /etc/nginx/sites-available/temporal /etc/nginx/sites-enabled/temporal
   
   sudo systemctl reload nginx

Interacting with the Temporal Service

• Temporal Web UI: Open your web browser and navigate to https://YOUR_DOMAIN.
• Temporal CLI: Use the Temporal CLI to interact with your Temporal service from the command line. Remember to include the --address your_server:7233 flag in your CLI commands.

Conclusion

By following these steps, you've successfully deployed a Temporal service with an Nginx reverse proxy and SSL certificate. This configuration provides a secure and scalable environment for your workflows.

Key Points:

• Security: Always prioritize security. Use IP-based access control with Nginx, and consider using Temporal's authentication features for production deployments.
• Monitoring: Monitor your Temporal service logs regularly to identify any issues and ensure smooth workflow operation.
• Scalability: For large-scale deployments, consider using Temporal's high-availability (HA) options and a more robust persistence solution like MySQL or PostgreSQL.

Feel free to ask if you have any further questions! Let me know if you'd like help with specific workflows, the Temporal CLI, or other aspects of the deployment.

[dhiaayachi]

This is a great starting point for a Temporal Service deployment. You've covered the essential steps and configurations for a basic Temporal deployment with an SQLite backend. Here are some additional notes and improvements:

Important Notes:

• Security: While the guide mentions using an allow statement with an IP address, a more robust security approach is highly recommended for production environments. Consider using:
  • Authentication: Integrate user authentication (e.g., with Temporal's Identity feature) to control access to Temporal services.
  • Authorization: Implement fine-grained authorization to restrict users to specific workflows or resources within Temporal.
  • SSL: Use SSL certificates to encrypt communication between clients and Temporal. The guide already covers SSL for the UI, but this is vital for the Temporal Server itself, as it handles sensitive data.
• Persistence: Using SQLite for a production environment can be inefficient and unreliable. Temporal has a mysql persistence plugin that is a better choice. Even better, consider a scalable database solution like PostgreSQL.
• Observability: Incorporate monitoring tools (e.g., Prometheus, Grafana) to track Temporal performance and error rates. Temporal's metrics feature can be used to expose metrics for monitoring.

Improvements to the Guide:

1. Explicitly State Security Best Practices: Clearly explain that the provided allow statement is for demonstration and should not be used in production. Add sections emphasizing the need for authentication, authorization, and SSL.
2. Expand on Persistence Options: Describe alternative persistence plugins, including MySQL and PostgreSQL, as well as their advantages and disadvantages. Highlight potential performance and scaling considerations.
3. Integrate Monitoring: Show how to use Temporal's metrics feature and connect it to monitoring tools like Prometheus. Provide examples of metrics that should be monitored and what they reveal about Temporal's health.

Additional Considerations:

• Scaling: For production deployments, consider how you'll scale Temporal services to handle increased load. The guide focuses on a single server; for scaling, consider using multiple Temporal Servers and leveraging Temporal's built-in support for clustering and sharding.
• Workflow Design: The guide doesn't cover workflow design best practices. Incorporate patterns for handling retries, timeouts, and other error handling strategies that are crucial for robust workflows.
• Testing: Include sections on unit testing and integration testing your Temporal workflows and clients. This helps ensure the reliability of your application and catch potential errors during development.

Example Improvements:

Security Section (Added):

## Security Considerations
Securing your Temporal service is crucial for production environments.  The following best practices are recommended:

* **Authentication:**  Use Temporal's built-in `Identity` feature to authenticate users connecting to Temporal. This prevents unauthorized access to workflows and data.
* **Authorization:**   Implement fine-grained authorization mechanisms to restrict users to specific resources within Temporal. This ensures that users only have access to the data they need.
* **SSL:**  Use SSL certificates to encrypt communication between clients and Temporal servers. This helps protect sensitive data from interception during transmission. 

Persistence Section (Updated):

## Persistence
For production use, we recommend using a more robust persistence solution than SQLite.  Consider these alternatives:

* **MySQL:**  A popular, reliable database that Temporal supports.
* **PostgreSQL:**  A highly scalable and feature-rich database that is often preferred for Temporal's persistent state.

By incorporating these recommendations and best practices, you can create a secure, scalable, and reliable Temporal service for your production needs. The Temporal community is a valuable resource for additional guidance and examples!