search

opensearch-project / sql

Query your data using familiar SQL or intuitive Piped Processing Language (PPL)
https://opensearch.org/docs/latest/search-plugins/sql/index/
Apache License 2.0
120 stars 139 forks source link

[BUG] Error "Async Query APIs are disabled" when trying to access prometheus #2601

Open shotor opened 7 months ago

shotor commented 7 months ago

Describe the bug

I updated to 2.13.0. Now when I go to datasources -> prometheus I get the following error when trying to query:

Async Query APIs are disabled. Please configure plugins.query.executionengine.spark.config in cluster settings to enable them.
GET /_cluster/settings?include_defaults=true&flat_settings=true
=> 
    "plugins.query.executionengine.async_query.enabled": "true",
    "plugins.query.executionengine.spark.config": "",

For some reason it's an empty string. I don't see any documentation on how to configure this. Trying to set it to an empty object yields an error:

PUT /_cluster/settings
{
  "persistent": {
    "plugins.query.executionengine.spark.config": {}
  }
}
=> 
{
  "error": {
    "root_cause": [
      {
        "type": "action_request_validation_exception",
        "reason": "Validation Failed: 1: no settings to update;"
      }
    ],
    "type": "action_request_validation_exception",
    "reason": "Validation Failed: 1: no settings to update;"
  },
  "status": 400
}

Unfortunately I can't downgrade to 2.12.0 because it gives me a formatTooNew error.

Related component

Search:Query Capabilities

To Reproduce

  1. Add prometheus datasource to dashboard
  2. Click on it
  3. See error

Expected behavior

It should allow me to query the datasource

Additional Details

Plugins

Screenshots

Host/Environment (please complete the following information):

Additional context

Full opensearch api log:

[2024-04-03T17:13:51,816][ERROR][o.o.s.s.r.RestAsyncQueryManagementAction] [9548ee319f74] Error happened during request handling
java.lang.IllegalArgumentException: Async Query APIs are disabled. Please configure plugins.query.executionengine.spark.config in cluster settings to enable them.
    at org.opensearch.sql.spark.client.EMRServerlessClientFactoryImpl.validateSparkExecutionEngineConfig(EMRServerlessClientFactoryImpl.java:52) ~[spark-2.13.0.0.jar:?]
    at org.opensearch.sql.spark.client.EMRServerlessClientFactoryImpl.getClient(EMRServerlessClientFactoryImpl.java:36) ~[spark-2.13.0.0.jar:?]
    at org.opensearch.sql.spark.dispatcher.SparkQueryDispatcher.dispatch(SparkQueryDispatcher.java:62) ~[spark-2.13.0.0.jar:?]
    at org.opensearch.sql.spark.asyncquery.AsyncQueryExecutorServiceImpl.createAsyncQuery(AsyncQueryExecutorServiceImpl.java:43) ~[spark-2.13.0.0.jar:?]
    at org.opensearch.sql.spark.transport.TransportCreateAsyncQueryRequestAction.doExecute(TransportCreateAsyncQueryRequestAction.java:67) [spark-2.13.0.0.jar:?]
    at org.opensearch.sql.spark.transport.TransportCreateAsyncQueryRequestAction.doExecute(TransportCreateAsyncQueryRequestAction.java:28) [spark-2.13.0.0.jar:?]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.13.0.jar:2.13.0]
    at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
    at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
    at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:395) [opensearch-security-2.13.0.0.jar:2.13.0.0]
    at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.13.0.0.jar:2.13.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
    at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:77) [opensearch-performance-analyzer-2.13.0.0.jar:2.13.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.13.0.jar:2.13.0]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.13.0.jar:2.13.0]
    at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.13.0.jar:2.13.0]
    at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.13.0.jar:2.13.0]
    at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.13.0.jar:2.13.0]
    at org.opensearch.sql.spark.rest.RestAsyncQueryManagementAction.lambda$executePostRequest$1(RestAsyncQueryManagementAction.java:126) [spark-2.13.0.0.jar:?]
    at org.opensearch.sql.datasources.utils.Scheduler.lambda$withCurrentContext$0(Scheduler.java:30) [datasources-2.13.0.0.jar:?]
    at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) [opensearch-2.13.0.jar:2.13.0]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
    at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

Dashboard logs:

Error in running direct query: StatusCodeError: Bad Request
    at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (node:events:529:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1400:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {
  status: 400,
  displayName: 'BadRequest',
  path: '/_plugins/_async_query',

  query: {},
  body: '{\n' +
    '  "status": 400,\n' +
    '  "error": {\n' +
    '    "type": "IllegalArgumentException",\n' +
    '    "reason": "Invalid Request",\n' +
    '    "details": "Async Query APIs are disabled. Please configure plugins.query.executionengine.spark.config in cluster settings to enable them."\n' +
    '  }\n' +
    '}',
  statusCode: 400,
  response: '{\n' +
    '  "status": 400,\n' +
    '  "error": {\n' +
    '    "type": "IllegalArgumentException",\n' +
    '    "reason": "Invalid Request",\n' +
    '    "details": "Async Query APIs are disabled. Please configure plugins.query.executionengine.spark.config in cluster settings to enable them."\n' +
    '  }\n' +
    '}',
  toString: [Function (anonymous)],
  toJSON: [Function (anonymous)]
}
peternied commented 7 months ago

[Triage - attendees 1 2 3 4 5 6 7 8] @opensearch-project/admin Could you please transfer this SQL repo - it looks like they will be in the best position to look into this 2.13 impacting issue.

HungryHowies commented 7 months ago

I have the same issue, Upgrade to version 2.12.0 was when i first noticed this and then upgraded to version 2.13.0 still same issue. Using either API command or Prometheus Plugin Web UI I get a connection when querying Data I see the metrics but if I click on any one of them screen goes blank. I posted here

AdaptiveStep commented 6 months ago

have this problem too. Get the same "Async Query APIs are disabled" error. Its really hard to get opensearch to work with any kind of prometheus stuff..

szydell commented 5 months ago

In 2.14.0 i can connect to Thanos again. It does not work too well, but that's for another ticket. :)