[Inference API] Making a request to a rerank endpoint without a query timesout

[maxhniebergall]

Elasticsearch Version

main git:b5d6fa0130dd987cea0c215751553478dc38f632

Installed Plugins

No response

Java Version

bundled

OS Version

macos

Problem Description

When making a request to cohere rerank endpoint without including both documents and query, there should be an illegal argument exception / bad request, but instead the request times out as the error is swallowed inside elasticsearch somewhere.

Steps to Reproduce

1. put inference endpoint with cohere rerank
2. send a perform inference request with the following body:

{
    "input": ["test1", "test2", "test3"]

}

Logs (if relevant)

[2024-10-08T10:57:27,016][WARN ][o.e.x.i.r.ModelRegistry  ] [runTask-0] Failed to store inference endpoint [testss] index: [.secrets-inference] bulk failure message [[.secrets-inference/RCiwHbAUQECxRUWxJ7MkZw][[.secrets-inference][0]] org.elasticsearch.index.engine.VersionConflictEngineException: [model_testss]: version conflict, document already exists (current version [1])]
[2024-10-08T10:58:13,734][WARN ][o.e.x.i.e.h.s.R.RateLimitingEndpointHandler] [runTask-0] Executor service grouping [-1200277257] failed to execute request java.lang.IllegalArgumentException: Unsupported inference inputs type: [class org.elasticsearch.xpack.inference.external.http.sender.DocumentsOnlyInput]
        at org.elasticsearch.inference@9.0.0-SNAPSHOT/org.elasticsearch.xpack.inference.external.http.sender.InferenceInputs.createUnsupportedTypeException(InferenceInputs.java:14)
        at org.elasticsearch.inference@9.0.0-SNAPSHOT/org.elasticsearch.xpack.inference.external.http.sender.QueryAndDocsInputs.of(QueryAndDocsInputs.java:17)
        at org.elasticsearch.inference@9.0.0-SNAPSHOT/org.elasticsearch.xpack.inference.external.http.sender.CohereRerankRequestManager.execute(CohereRerankRequestManager.java:51)
        at org.elasticsearch.inference@9.0.0-SNAPSHOT/org.elasticsearch.xpack.inference.external.http.sender.RequestExecutorService$RateLimitingEndpointHandler.executeEnqueuedTaskInternal(RequestExecutorService.java:416)
        at org.elasticsearch.inference@9.0.0-SNAPSHOT/org.elasticsearch.xpack.inference.external.http.sender.RequestExecutorService$RateLimitingEndpointHandler.executeEnqueuedTask(RequestExecutorService.java:388)
        at org.elasticsearch.inference@9.0.0-SNAPSHOT/org.elasticsearch.xpack.inference.external.http.sender.RequestExecutorService.handleTasks(RequestExecutorService.java:237)
        at org.elasticsearch.inference@9.0.0-SNAPSHOT/org.elasticsearch.xpack.inference.external.http.sender.RequestExecutorService.start(RequestExecutorService.java:192)
        at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:925)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1570)

[2024-10-08T10:58:43,709][WARN ][r.suppressed             ] [runTask-0] path: /_inference/testss, params: {task_type_or_id=testss}, status: 500 org.elasticsearch.ElasticsearchTimeoutException: Request timed out waiting to be sent after [30s]
        at org.elasticsearch.inference@9.0.0-SNAPSHOT/org.elasticsearch.xpack.inference.external.http.sender.RequestTask.lambda$getListener$2(RequestTask.java:67)
        at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.ListenerTimeouts$TimeoutableListener.run(ListenerTimeouts.java:103)
        at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:925)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1570)

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[davidkyle]

@maxhniebergall is this still an issue? I tried to reproduce the issue but the POST inference request returned a validation error when the query was not present

    "type": "action_request_validation_exception",
    "reason": "Validation Failed: 1: Field [query] cannot be null for task type [rerank];"

It looks like the request validation was fixed in #110147 but that fix predates this issue.

[maxhniebergall]

@davidkyle, I just tried it out and its still timing out for me. I don't see that validation exception at all in my test.

[davidkyle]

Thanks @maxhniebergall I've got it now. The validation exception is thrown if using the URL that includes the task type, if not you get the timeout.

# This call will not pass the validation checks
POST _inference/rerank/my-reranker
{
    "input": ["test1", "test2", "test3"]
}

"Validation Failed: 1: Field [query] cannot be null for task type [rerank];"

# This call will time out
POST _inference/my-reranker
{
    "input": ["test1", "test2", "test3"]
}