look at breaking changes in newest es versions and ingest/config-mgmt es code; start a plan to support upgrade

[vjeffrey]

User Story

we're going to need to upgrade to es 7 or 8 by the end of the year, bc 6 will be eol. https://github.com/chef/automate/issues/4165

let's look at the breaking changes and see if there's anything that affects the ingest/config mgmt code

Aha! Link: https://chef.aha.io/epics/SH-E-738

[vjeffrey]

current version: https://github.com/chef/automate/blob/master/components/automate-elasticsearch/habitat/plan.sh#L7

es: https://www.elastic.co/guide/en/elasticsearch/reference/current/es-release-notes.html

theoretically, we should be able to simply update the version in the automate-elasticsearch plan, rebuild, and see the results of upgrading.

plan is to update to ES 7.9

[vjeffrey]

also need to look at plan for bringing customers from A1 (es5) to latest (which will now be es7) see https://chefio.slack.com/archives/C6JED4R8W/p1596221710414500

[lancewf]

Elasticsearch breaking changes https://www.elastic.co/guide/en/elasticsearch/reference/7.0/release-notes-7.0.0.html#breaking-7.0.0

[lancewf]

Time formats with timezones need updated https://www.elastic.co/guide/en/elasticsearch/reference/current/migrate-to-java-time.html#java-time-migration-incompatible-date-formats.

"yyyy-MM-dd'T'HH:mm:ssZ" need to be changed to "yyyy-MM-dd'T'HH:mm:ssX" for times like "2017-09-01-07:58:06Z". But for time like "2010-01-01T01:02:03-07:00" we need to add more X like "yyyy-MM-dd'T'HH:mm:ssXXX"

[lancewf]

All "elastic.NewTypeQuery(mappings.DocType)" filters can be removed because there is not a doctype anymore for documents. So All documents in indexes have the same doctype.

All elastic.NewIdsQuery(mappings.DocType) can remove the doctype because all the documents have the same doctypes.

[lancewf]

compliance-service reporting test

  1) Failure:
07_suggestions_spec.rb#test_0001_works [/Users/lancewf/workspace/automate/components/compliance-service/api/tests/07_suggestions_spec.rb:141]:

--- expected
+++ actual
@@ -1 +1,3 @@
-[]
+["Apache should be enabled--apache-02--",
+ "Apache should be running--apache-01--",
+ "Apache should start max. 1 root-task--apache-03--"]

1 runs, 20 assertions, 1 failures, 0 errors, 0 skips

11_wonky_profiles_spec.rb#test_0001_works [/Users/lancewf/workspace/automate/components/compliance-service/api/tests/11_wonky_profiles_spec.rb:28]:
Expected /InSpec archive failed for /
 to match "Received RST_STREAM with error code 2"
.

  1) Failure:
07_suggestions_w_filters_spec.rb#test_0001_works [/Users/lancewf/workspace/automate/components/compliance-service/api/tests/07_suggestions_w_filters_spec.rb:50]:
--- expected
+++ actual
@@ -1 +1,10 @@
-["Disable neighbor solicitations to send out per address--sysctl-27--"]
+["Apache should start max. 1 root-task different--apache-03--",
+ "Assign one global unicast IPv6 addresses to each interface--sysctl-28--",
+ "Disable Accept Router Preference from router advertisement--sysctl-22--",
+ "Disable Apache’s follows Symbolic Links for directories in alias.conf--apache-11--",
+ "Disable Core Dumps--sysctl-31--",
+ "Disable Directory Listing for directories in alias.conf--apache-12--",
+ "Disable IPv6 autoconfiguration--sysctl-26--",
+ "Disable IPv6 if it is not needed--sysctl-18--",
+ "Disable Source Routing--sysctl-13--",
+ "Disable TRACE-methods--apache-09--"]

1 runs, 2 assertions, 1 failures, 0 errors, 0 skips

[lancewf]

After adding 100 compliance nodes with "chef_load_compliance_nodes 100" the automate-elasticsearch's status is "unknown" When running curl "$ELASTICSEARCH_URL/_cat/indices?pretty" I get the below error message.

{
  "error" : {
    "root_cause" : [
      {
        "type" : "circuit_breaking_exception",
        "reason" : "[parent] Data too large, data for [<http_request>] would be [1582945680/1.4gb], which is larger than the limit of [1478832947/1.3gb], real usage: [1582945680/1.4gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b, model_inference=0/0b, accounting=1601192/1.5mb]",
        "bytes_wanted" : 1582945680,
        "bytes_limit" : 1478832947,
        "durability" : "PERMANENT"
      }
    ],
    "type" : "circuit_breaking_exception",
    "reason" : "[parent] Data too large, data for [<http_request>] would be [1582945680/1.4gb], which is larger than the limit of [1478832947/1.3gb], real usage: [1582945680/1.4gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b, model_inference=0/0b, accounting=1601192/1.5mb]",
    "bytes_wanted" : 1582945680,
    "bytes_limit" : 1478832947,
    "durability" : "PERMANENT"
  },
  "status" : 429
}

[lancewf]

Because the default number of shards has changed from 5 in es6 to 1 in es7 each time an index or template is created we need to set the number of shards to 5.

https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#_static_index_settings https://www.elastic.co/guide/en/elasticsearch/reference/6.8/index-modules.html#_static_index_settings

[lancewf]

For the automate-cs-oc-erchef component, the "_all" is not a valid field in ES 7. https://github.com/chef/automate/blob/master/components/automate-cs-oc-erchef/habitat/config/elasticsearch-index-init.json#L18

https://www.elastic.co/guide/en/elasticsearch/reference/6.8/mapping-all-field.html