Setup only clickhouse backend

[b17k0]

Hello! Is it possible to set up quesma without elasticsearch? I mean, to have the following scheme: Kibana -> Quesma -> ClickHouse With this configuration:

#licenseKey: {your-quesma-license-key-if-needed} # license key is required for paid features such as `hydrolix` backend conector
frontendConnectors:
  - name: elastic-query
    type: elasticsearch-fe-query
    config:
      listenPort: 8080
backendConnectors:
  - name: my-clickhouse-data-source
    type: clickhouse-os
    config:
      url: "clickhouse://localhost:9000"
      adminUrl: "http://localhost:8123/play"
ingestStatistics: true
disableTelemetry: true
logging:
  path: "logs"
  level: "info"
  disableFileLogging: false
processors:
  - name: my-query-processor
    type: quesma-v1-processor-query
    config:
      indexes:
        some_transactions:
          target: [ my-clickhouse-data-source ]
          schemaOverrides:
            fields:
              timestamp:
                type: alias
                targetColumnName: "started_at"
              id:
                type: "keyword"
              started_at:
                type: "date"
              uuid:
                type: "keyword"
              account_id:
                type: "keyword"
              state:
                type: "keyword"
              category:
                type: "keyword"
              balance_at_start:
                type: "float"
              balance_at_end:
                type: "float"
              finished_at:
                type: "date"
              currency_id:
                type: "keyword"
              merchant_id:
                type: "keyword"
              user_login:
                type: "keyword"
              # more fields
        "*":       # DO NOT remove, always required
          target: [ my-clickhouse-data-source ]
pipelines:
  - name: my-pipeline-elasticsearch-query-clickhouse
    frontendConnectors: [ elastic-query ]
    processors: [ my-query-processor ]
    backendConnectors: [ my-clickhouse-data-source ]

I get the following error:

2024/11/13 15:51:54 Config validation failed: 2 errors occurred:
    * pipeline my-pipeline-elasticsearch-query-clickhouse has a processor of type quesma-v1-processor-query which requires having one elasticsearch backend connector
    * query processor requires two backend connectors

[ptbrzoska]

Hi, thanks for you question, Quesma always requires a minimal Elasticsearch/OpenSearch cluster for:

• storing Kibana/OSD metadata
• storing user data and authentication (unless its explicitly disabled)
• storing any additional indexes appearing at ingest endpoint and not explicitly listed in Quesma ingest processor no data needs to reside there though, everything can be in ClickHouse. Check this page for more information: https://docs.quesma.com/example-2-0-clickhouse-specific.html

[b17k0]

Thank you for the answer. Is it possible to add a processor type where elasticsearch is not needed? That is, to store data such as Kibana/OSD metadata, storing user data, in a separate clickhouse table? I'm looking for options to connect kibana to clickhouse, but I really don't want to keep even minimal elasticsearch.

[ptbrzoska]

Understood. That's currently a roadmap item - feel free to come back in some time to check if that's available in one of the next releases.

[mieciu]

Closing for now as this is going to be a part of way bigger initiative.

Thank you for your feedback and detailed explanation, it is truly appreciated! ❤️

At this moment, we do not have publicly-facing roadmap for people to follow, so I'd encourage you to subscribe to our newsletter (there's form at the bottom of this page) to stay up to date with the project!

[avelanarius]

Changed the closing reason from "completed" to more appropriate "not planned" (not planned in the short term)

[ptbrzoska]

Mistake. Re-opened