change-metrics / monocle

Monocle helps teams and individual to better organize daily duties and to detect anomalies in the way changes are produced and reviewed.
https://demo.changemetrics.io/
GNU Affero General Public License v3.0
365 stars 56 forks source link

Regex limit reached for project definitions #572

Closed morucci closed 2 years ago

morucci commented 2 years ago

The current master hits this issue when a project regexps is too long. ElasticSearch limits to 1000 chars.

openstack: searching QueryRequest_QueryTypeQUERY_CHANGE with `(from:2021-05-01 repo_regex: .*-distgit) and project:rdo-ui`: [{"bool":{"must":[{"bool":{"must":[{"range":{"created_at":{"gt":"2021-05-01T00:00:00Z","boost":1}}},{"regexp":{"repository_fullname":{"flags":"ALL","value":".*-distgit"}}}]}},{"bool":{"must":[{"regexp":{"repository_fullname":{"flags":"ALL","value":"^rdo/openstack/octavia-dashboard-distgit|^openstack/oct
avia-dashboard|^rdo/openstack/vitrage-dashboard-distgit|^openstack/vitrage-dashboard|^rdo/openstack/horizon-distgit|^openstack/horizon|^rdo/openstack/django_openstack_auth-distgit|^openstack/django_openstack_auth|^rdo/openstack/heat-dashboard-distgit|^openstack/heat-dashboard|^rdo/openstack/ironic-ui-distgit|^openstack/ironic-ui|^rdo/openstack/cloudkitty-dashboard-distgit|^openstack/cloudkitty-dashboard|^rdo/openstack/manila
-ui-distgit|^openstack/manila-ui|^rdo/openstack/neutron-lbaas-dashboard-distgit|^openstack/neutron-lbaas-dashboard|^rdo/openstack/trove-dashboard-distgit|^openstack/trove-dashboard|^rdo/openstack/sahara-dashboard-distgit|^openstack/sahara-dashboard|^rdo/openstack/tripleo-ui-distgit|^openstack/tripleo-ui|^rdo/openstack/app-catalog-ui-distgit|^openstack/app-catalog-ui|^rdo/openstack/mistral-dashboard-distgit|^openstack/mistral
-dashboard|^rdo/openstack/magnum-ui-distgit|^openstack/magnum-ui|^rdo/openstack/murano-dashboard-distgit|^openstack/murano-dashboard|^rdo/openstack/tempest-horizon-distgit|^openstack/tempest-horizon|^rdo/openstack/karbor-dashboard-distgit|^openstack/karbor-dashboard|^rdo/openstack/designate-dashboard-distgit|^openstack/designate-dashboard"}}}]}}]}}]                                                                             
Response {responseStatus = Status {statusCode = 400, statusMessage = "Bad Request"}, responseVersion = HTTP/1.1, responseHeaders = [("content-type","application/json; charset=UTF-8"),("content-encoding","gzip"),("content-length","422")], responseBody = "{\"error\":{\"root_cause\":[{\"type\":\"query_shard_exception\",\"reason\":\"failed to create query: The length of regex [1251] used in the Regexp Query request has exceeded 
the allowed maximum of [1000]. This maximum can be set by changing the [index.max_regex_length] index level setting.\",\"index_uuid\":\"YjcdhqIVRRSh4qu9xfeqvQ\",\"index\":\"monocle.changes.1.openstack\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"monocle.changes.1.openstack\",\"node\":\"oTQanDwjT2SNQNv9uaDRKA
\",\"reason\":{\"type\":\"query_shard_exception\",\"reason\":\"failed to create query: The length of regex [1251] used in the Regexp Query request has exceeded the allowed maximum of [1000]. This maximum can be set by changing the [index.max_regex_length] index level setting.\",\"index_uuid\":\"YjcdhqIVRRSh4qu9xfeqvQ\",\"index\":\"monocle.changes.1.openstack\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\"
:\"The length of regex [1251] used in the Regexp Query request has exceeded the allowed maximum of [1000]. This maximum can be set by changing the [index.max_regex_length] index level setting.\"}}}]},\"status\":400}", responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose, responseOriginalRequest = Request {                                                                                                         
  host                 = "localhost"                                                                                                                                                                                                                                                                                                                                                                                                        
  port                 = 9200                                                                                                                                                                                                                                                                                                                                                                                                               
  secure               = False                                                                                                                                                                                                                                                                                                                                                                                                              
  requestHeaders       = [("Content-Type","application/json")]                                                                                                                                                                                                                                                                                                                                                                              
  path                 = "/monocle.changes.1.openstack/_search"                                                                                                                                                                                                                                                                                                                                                                             
  queryString          = ""                                                                                                                                                                                                                                                                                                                                                                                                                 
  method               = "POST"                                                                                                                                                                                                                                                                                                                                                                                                             
  proxy                = Nothing                                                                                                                                                                                                                                                                                                                                                                                                            
  rawBody              = False                                                                                                                                                                                                                                                                                                                                                                                                              
  redirectCount        = 10                                                                                                                                                                                                                                                                                                                                                                                                                 
  responseTimeout      = ResponseTimeoutDefault                                                                                                                                                                                                                                                                                                                                                                                             
  requestVersion       = HTTP/1.1                                                                                                                                                                                                                                                                                                                                                                                                           
  proxySecureMode      = ProxySecureWithConnect                                                                                                                                                                                                                                                                                                                                                                                             
}                                                                                                                                                                                                                                                                                                                                                                                                                                           
} 

The settings could be set at index creation, however bloodhound does not support this setting. A change in bloodhound upstream might be needed.

In the meantime this issue must be stated in the troubleshooting section.

morucci commented 2 years ago

This https://github.com/change-metrics/monocle/pull/573 adds workaround in README.md