GKE Web UI 404 Archival Using bucket

dhiaayachi commented 2 months ago

Web UI 404 when accessing detail of workflow id from archival

Here is the English translation of your issue:

I am using Helm to deploy to GKE with the values.archival.gcloud.yaml as follows:

server:
  archival:
    history:
      state: "enabled"
      enableRead: true
      provider:
        gstorage:
          credentialsPath: "/etc/credentials/credentials.json"
    visibility:
      state: "enabled"
      enableRead: true
      provider:
        gstorage:
          credentialsPath: "/etc/credentials/credentials.json"

  namespaceDefaults:
    archival:
      history:
        state: "enabled"
        URI: "gs://kp-temporal-archival/temporal_archival"
      visibility:
        state: "enabled"
        URI: "gs://kp-temporal-archival/temporal_visibility"

For persistence, I am using PostgreSQL with the following values.postgresql.yaml:

server:
  config:
    persistence:
      default:
        driver: "sql"

        sql:
          driver: "postgres12"
          host: postgres.example.dev
          port: 5432
          database: temporal
          user: temporal
          password: temporal
          maxConns: 20
          maxConnLifetime: "1h"

      visibility:
        driver: "sql"

        sql:
          driver: "postgres12"
          host: postgres.example.dev
          port: 5432
          database: temporal_visibility
          user: temporal
          password: temporal
          maxConns: 20
          maxConnLifetime: "1h"

cassandra:
  enabled: false

mysql:
  enabled: false

postgresql:
  enabled: true

prometheus:
  enabled: false

grafana:
  enabled: false

elasticsearch:
  enabled: true

schema:
  createDatabase:
    enabled: false
  setup:
    enabled: true
  update:
    enabled: true

Then, I forward the Web UI using Internal Ingress with the following configuration:

web:
  enabled: true
  replicaCount: 1
  image:
    repository: temporalio/ui
    tag: 2.28.0
    pullPolicy: IfNotPresent
  service:
    type: ClusterIP
    port: 8080
    annotations: {}
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.regional-static-ip-name: ingress-temporal-web
      kubernetes.io/ingress.allow-http: "true"
      kubernetes.io/ingress.class: "gce-internal"
    hosts:
      - "temporal.example.dev"
    tls:
      - secretName: temporal
        hosts:
          - temporal.example.dev

The archival and visibility files are already in the bucket.

and this is my version server 1.24.2

dhiaayachi commented 2 months ago

Web UI 404 when accessing detail of workflow id from archival

This issue appears to be related to the Web UI not correctly handling archival. It seems the Web UI is making requests to the active cluster, even though the Workflow details are in archival.

Here are some potential causes and solutions:

1. Namespace Configuration

Verify Archival URI: Double-check that the URI for both history and visibility is correctly configured in your Helm values.yaml. It should point to the correct location in Google Cloud Storage (GCS):
- gs://kp-temporal-archival/temporal_archival for history
- gs://kp-temporal-archival/temporal_visibility for visibility
Verify Archival State: Ensure that both history and visibility are set to "enabled" in your Helm values.yaml. This indicates that archival is enabled for the namespace.

2. Web UI Configuration

Forwarding: The Web UI should be configured to forward requests to the active cluster for live data and to the archival location for archived data. You need to configure this logic within your Nginx or Envoy proxy settings.

3. Web UI Version

Compatibility: Ensure your Web UI version is compatible with the Temporal Server version (1.24.2). You're using 2.28.0, which should be compatible, but it's worth double-checking the Temporal Release Notes for known compatibility issues.

4. GCS Permissions

Access: The Temporal server running in your GKE cluster needs appropriate permissions to read data from the GCS bucket. Verify that the service account used by your Temporal server has read permissions on the GCS bucket.

5. Temporal Server Settings

Enable Read from Archival: Ensure the archival.history.enableRead and archival.visibility.enableRead are set to true in your Temporal Server configuration. This setting allows the server to read from the archival location.

Debugging Steps:

Check Server Logs: Examine the Temporal server logs for any errors or warnings related to archival or the Web UI. You can access these logs using the Kubernetes logs.
Verify Network Connectivity: Ensure that the Temporal server can connect to GCS. Test the connection using the gsutil command-line tool.
Test with Temporal CLI: Use the temporal workflow show command with the --archived flag to confirm that the Web UI can access the archived data.

Further Investigation:

If the problem persists, consider opening a Support Ticket with Temporal. They may be able to assist with more advanced troubleshooting or help you identify any configuration issues. Provide them with the following details:

Helm Values: Share your full values.yaml files for both the server and web UI.
GCS Bucket Name: Provide the name of your GCS bucket.
Server Logs: Include any relevant logs from your Temporal server or Web UI.
Network Configuration: Describe your network setup, including your Ingress configuration.
Steps to Reproduce: Outline the exact steps needed to reproduce the issue.

By following these steps, you can troubleshoot the issue and ensure that your Web UI can correctly access Workflow details from archival.

dhiaayachi commented 2 months ago

Web UI 404 when accessing detail of workflow id from archival

Thank you for reporting this issue.

Based on the information you provided, it seems like you might be experiencing a known issue with the Web UI's handling of archived workflows when using Google Cloud Storage (GCS) for archival. This issue is acknowledged and there is an ongoing effort to improve the UI experience for these scenarios.

In the meantime, you can use the Temporal CLI to access the details of archived workflows.

Here's how to do it:

temporal workflow show --workflow-id <your_workflow_id> --namespace <your_namespace> --archived true

This command will display the Event History for the specified workflow ID in the given namespace.

We apologize for any inconvenience this may cause. We're working to improve the UI's handling of archived workflows, and we'll update the documentation when there's more information available.

dhiaayachi commented 2 months ago

Thanks for reporting this issue! It appears that you are seeing a 404 error when trying to access the details of an archived Workflow. I'm looking into this now.

Please tell me what version of the Temporal Server you are using?

dhiaayachi commented 2 months ago

Thanks for reporting this issue! It appears you're experiencing a 404 error in the Temporal Web UI when trying to access details of a workflow ID from archival.

Based on the information provided, there is a mismatch between the Temporal server and UI server versions.

The server is running on 1.24.2, while the UI is running on 2.28.0. Make sure you are running compatible versions of both the Temporal Server and the Web UI. It's best to consult the Temporal release notes to verify compatible versions between the two.

If this doesn't solve the problem, please provide the following:

The version of Temporal Server that you're running. You can get this by running the temporal server --version command.
The version of the Temporal Web UI that you're running. You can get this by checking the browser console after you open the Web UI and look for any errors or warnings.
More details about the workflow ID you're trying to access. Please provide the workflow ID and run ID, as well as the Namespace.
Any relevant logs from the Temporal Server and Web UI.

We'll be able to provide more precise assistance with this information.

dhiaayachi commented 2 months ago

This looks great! You've provided a detailed and comprehensive guide to deploying a Temporal Service with an SQLite backend and Nginx reverse proxy, including essential steps like configuring system services and securing the connection.

Here are some additional suggestions to make this tutorial even better:

Content Improvements:

Target Audience: Explicitly define the target audience for this tutorial, such as:
- Developers with basic knowledge of Temporal and Nginx
- System administrators familiar with Linux and systemd
Workflow Example: Include a simple workflow example to demonstrate how to interact with the Temporal Service after the deployment. This could be a basic "Hello World" or a slightly more complex workflow that shows the benefits of Temporal.
Troubleshooting: Add a troubleshooting section covering common issues that developers might encounter during the deployment process, such as:
- Nginx configuration errors
- Temporal service startup failures
- Accessing the Temporal UI from the external network
Alternative Backends: Briefly mention other available backends for Temporal, such as:
- MySQL
- PostgreSQL
- Cassandra
- Temporal Cloud

Technical Enhancements:

Containerization: Consider including an optional section that guides developers through containerizing the Temporal Service using Docker. This can simplify the deployment process and make it more portable.
Monitoring: Add a section that demonstrates how to monitor the Temporal service using tools like:
- Temporal UI
- Prometheus
- Grafana
Security Best Practices: Emphasize the importance of security and provide additional security recommendations, such as:
- Enabling HTTPS for the Temporal Core Server
- Using a strong password for the Temporal user
- Implementing access control mechanisms for the Temporal UI
- Restricting inbound traffic to the Temporal UI server using a firewall

Style and Organization:

Clear Formatting: Ensure consistent formatting, headings, and code blocks to enhance readability.
Visual Aids: Consider using screenshots or diagrams to illustrate certain steps of the deployment process.
Code Syntax Highlighting: Use syntax highlighting for code blocks to make it easier for developers to read and understand the code examples.

Overall: This is a very valuable tutorial, and by making these improvements, you can make it even more informative and user-friendly for developers who are looking to deploy a Temporal Service with an Nginx reverse proxy and an SQLite backend.

dhiaayachi commented 2 months ago

Thanks for reporting this issue.

Please clarify the version of the temporalio/ui image you are using.

Are you using the Web UI behind a proxy or directly accessing it?

dhiaayachi commented 2 months ago

Thanks for reporting this issue.

It looks like you are using the Web UI to access your archival details and the request is returning a 404 error. Based on the images you provided, it looks like you might not have the archival feature enabled for the default namespace. If you are trying to access archival data in a different namespace, please check if you have enabled archival for that specific namespace.

To enable the archival feature, you can use the following steps:

Edit your values.archival.gcloud.yaml file to enable the archival feature for both history and visibility:

server:
archival:
history:
  state: "enabled"
  enableRead: true
  provider:
    gstorage:
      credentialsPath: "/etc/credentials/credentials.json"
visibility:
  state: "enabled"
  enableRead: true
  provider:
    gstorage:
      credentialsPath: "/etc/credentials/credentials.json"

You can also configure the archival URI for both history and visibility:

server:
namespaceDefaults:
archival:
  history:
    state: "enabled"
    URI: "gs://kp-temporal-archival/temporal_archival"
  visibility:
    state: "enabled"
    URI: "gs://kp-temporal-archival/temporal_visibility"

If you are using Helm to deploy your Temporal service, you can apply these changes using helm upgrade.
After you have applied these changes, restart your Temporal service.

If the issue persists after restarting your Temporal service, please provide more context by answering the following questions:

Have you confirmed that the archival files are present in the bucket?
Are you able to access the archival files using the gsutil command?
Have you enabled the archival feature for the namespace you are trying to access?

This information can help us narrow down the issue and provide you with a more specific solution.

dhiaayachi commented 1 month ago

Thanks for reporting the issue.

It seems you are experiencing issues with accessing the Web UI after archival is enabled.

This is a known issue in Temporal server 1.24.2. You can either upgrade to a newer version of Temporal server or use the workarounds listed below:

Upgrade to a newer version of Temporal server: Upgrade to a more recent version of Temporal server that includes a fix for this issue. https://github.com/temporalio/temporal/releases
Disable Archival: Temporarily disable Archival to access the Web UI. Once the issue is resolved, you can re-enable Archival. https://docs.temporal.io/self-hosted-guide/archival#configuration
Use a different UI: Use a different UI such as the Temporal CLI to interact with Temporal until the issue is resolved. https://docs.temporal.io/cli

Let us know if these workarounds resolve the issue.

dhiaayachi commented 1 month ago

Thanks for reporting this issue.

I need more context to be able to help. Could you please provide the following information:

What is the exact workflow ID you are trying to access?
Is the workflow still running or has it been closed?
Is it possible to share the logs from the Temporal server?

This will help me understand the situation better and provide you with the most relevant solution.

dhiaayachi / temporal