[ML] Creating an ML related detection rule in Security with the recognizer API times out

[qn895]

Kibana version: 7.16.2

Elasticsearch version:

Server OS version:

Browser version:

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.):

Describe the bug: Hitting the api/ml/modules/recognize occasionally gives 502 error with the following issue EOF, backend closed connection.

Steps to reproduce:

1. Go to Security app, create a new detection rule with Machine Learning
2. If the index pattern/data view set in Security's advanced setting contains a remote cluster index that takes a long time to search, there will be an error popping up Security job fetch failure

Also, currently there's no way to modify what indices are used in an ML rule directly in the Security UI. The only option is through Advanced settings.

Expected behavior: Jobs and rules are created successfully.

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)