Handle wrong store names in search request headers more gracefully

[RasonJ]

If a non-existent store is passed in as X-ubi-store, the search client errors out with the following error message:

HTTP/1.1 404 Not Found
content-type: application/json; charset=UTF-8
content-length: 351

{"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"","phase":"expand","grouped":true,"failed_shards":[],"caused_by":{"type":"index_not_found_exception","reason":"no such index [.ubi_log_queries]","index":".ubi_log_queries","resource.id":".ubi_log_queries","resource.type":"index_or_alias","index_uuid":"_na_"}},"status":404}

Maybe return the search results regardless with a warning? Not sure what the best way to handle it would be.

[jzonthemtn]

I can change it to accept the query but log a warning on the OS side that a received search did not include the header. There's probably cases where someone would legitimately not want the query logged and are omitting the header on purpose.

[epugh]

Yeah... We do need to think through hardening as people start to use it.. what are the common "Oops, I goofed in configuration" and how do we communicate that!

[jzonthemtn]

We can either not let anything break, or we can let it break. I can see the benefit from both sides. If you are sending a search with a bad UBI store, breaking might be ideal so you know something is not right. But I can also see not allowing it to break as a fail-safe to prevent an expected user experience.

Looking at it from an OpenSearch perspective, I lean to the side of not letting things break. OpenSearch and its Elasticsearch roots go to lengths to "just work", e.g. automatically creating an index if one does not exist or automatically creating a mapping if none is given. So my thought is the same here - don't let a bad header value break the search. Log a warning and move on.