kestra-io / kestra

Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
https://kestra.io
Apache License 2.0
7.58k stars 462 forks source link

HTTP plugin icon doesn't render in the new Outputs #4437

Closed anna-geller closed 1 month ago

anna-geller commented 1 month ago

Describe the issue

image

id: microservices_and_apis
namespace: tutorial
description: Microservices and APIs

inputs:
  - id: server_uri
    type: URI
    defaults: https://kestra.io

  - id: slack_webhook_uri
    type: URI
    defaults: https://reqres.in/api/slack

tasks:
  - id: http_status_check
    type: io.kestra.plugin.core.flow.AllowFailure
    tasks:
      - id: http_request
        type: io.kestra.plugin.core.http.Request
        uri: "{{ inputs.server_uri }}"

      - id: check_status
        type: io.kestra.plugin.core.flow.If
        condition: "{{ outputs.http_request.code != 200 }}"
        then:
          - id: unhealthy
            type: io.kestra.plugin.core.log.Log
            message: "Server is unhealthy! Response {{ outputs.http_request.body }}"
          - id: send_slack_alert
            type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
            url: "{{ inputs.slack_webhook_uri }}"
            payload: |
              {
                "channel": "#alerts",
                "text": "The server {{ inputs.server_uri }} is down!"
              }
        else:
          - id: healthy
            type: io.kestra.plugin.core.log.Log
            message: Everything is fine!
    errors:
      - id: server_unreachable
        type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
        url: "{{ inputs.slack_webhook_uri }}"
        payload: |
          {
            "channel": "#alerts",
            "text": "The server {{ inputs.server_uri }} is unreachable!"
          }

triggers:
  - id: daily
    type: io.kestra.plugin.core.trigger.Schedule
    disabled: true
    cron: "0 9 * * *"

Environment

anna-geller commented 1 month ago

same for dbt, python and SQLite

image

id: dwh_and_analytics
namespace: tutorial
description: Data Warehouse and Analytics

tasks:
  - id: dbt
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
    - id: clone_repository
      type: io.kestra.plugin.git.Clone
      url: https://github.com/kestra-io/dbt-demo
      branch: main

    - id: dbt_build
      type: io.kestra.plugin.dbt.cli.DbtCLI
      taskRunner:
        type: io.kestra.plugin.scripts.runner.docker.Docker
      containerImage: ghcr.io/kestra-io/dbt-duckdb:latest
      commands:
        - dbt deps
        - dbt build
      profiles: |
        jaffle_shop:
          outputs:
            dev:
              type: duckdb
              path: dbt.duckdb
              extensions: 
                - parquet
              fixed_retries: 1
              threads: 16
              timeout_seconds: 300
          target: dev      

    - id: python
      type: io.kestra.plugin.scripts.python.Script
      outputFiles:
        - "*.csv"
      taskRunner:
        type: io.kestra.plugin.scripts.runner.docker.Docker
      containerImage: ghcr.io/kestra-io/duckdb:latest
      script: |
        import duckdb
        import pandas as pd

        conn = duckdb.connect(database='dbt.duckdb', read_only=False)

        tables_query = "SELECT table_name FROM information_schema.tables WHERE table_schema = 'main';"
        tables = conn.execute(tables_query).fetchall()

        # Export each table to CSV, excluding tables that start with 'raw' or 'stg'
        for table_name in tables:
            table_name = table_name[0]
            # Skip tables with names starting with 'raw' or 'stg'
            if not table_name.startswith('raw') and not table_name.startswith('stg'):
                query = f"SELECT * FROM {table_name}"
                df = conn.execute(query).fetchdf()
                df.to_csv(f"{table_name}.csv", index=False)

        conn.close()