Attaching an ILM - Githubissues

atefhaloui commented 1 year ago

Hi, It would be nice to add a default ILM in the script to avoid blocking elasticsearch if the index becomes too big. I've added a simple ILM manually by running the following commands from devtools on kibana:

Create new policy

PUT _ilm/policy/hot_warm_delete_policy
{
"policy": {
"phases": {
  "hot": {
    "min_age": "0ms",
    "actions": {
      "rollover": {
        "max_primary_shard_size": "100gb",
        "max_age": "7d"
      }
    }
  },
  "warm": {
    "min_age": "7d",
    "actions": {
      "shrink": {
          "number_of_shards": 1
        },
        "forcemerge": {
          "max_num_segments": 1
        }
    }
  },
  "delete": {
    "min_age": "15d",
    "actions": {
      "delete": {
        "delete_searchable_snapshot": true
      }
    }
  }
}
}
}

Update the template by adding settings.index.lifecycle.name:

PUT /_index_template/logs-akamai.datastream2
{
"index_patterns": [
"datastream2*"
],
"template": {
"settings": {
  "index.lifecycle.name": "hot_warm_delete_policy"
},
"mappings": {
  "_routing": {
    "required": false
  },
  "numeric_detection": false,
  "dynamic": true,
  "_source": {
    "excludes": [],
    "includes": [],
    "enabled": true
  },
  "dynamic_templates": [],
  "date_detection": false,
  "properties": {
    "country": {
      "type": "keyword"
    },
    "referer": {
      "type": "keyword"
    },
    "reqPort": {
      "type": "keyword"
    },
    "rspContentLen": {
      "coerce": true,
      "ignore_malformed": true,
      "type": "integer"
    },
    "reqMethod": {
      "type": "keyword"
    },
    "streamId": {
      "type": "keyword"
    },
    "city": {
      "type": "keyword"
    },
    "lastByte": {
      "type": "keyword"
    },
    "reqTimeSec": {
      "format": "epoch_second",
      "ignore_malformed": true,
      "type": "date"
    },
    "securityRules": {
      "type": "keyword"
    },
    "tlsOverheadTimeMSec": {
      "coerce": true,
      "ignore_malformed": true,
      "type": "integer"
    },
    "errorCode": {
      "type": "keyword"
    },
    "range": {
      "type": "keyword"
    },
    "UA": {
      "type": "keyword"
    },
    "accLang": {
      "type": "keyword"
    },
    "billingRegion": {
      "type": "keyword"
    },
    "objSize": {
      "coerce": true,
      "ignore_malformed": true,
      "type": "integer"
    },
    "rspContentType": {
      "type": "keyword"
    },
    "transferTimeMSec": {
      "coerce": true,
      "ignore_malformed": true,
      "type": "integer"
    },
    "dnsLookupTimeMSec": {
      "coerce": true,
      "ignore_malformed": true,
      "type": "integer"
    },
    "overheadBytes": {
      "coerce": true,
      "ignore_malformed": true,
      "type": "integer"
    },
    "totalBytes": {
      "coerce": true,
      "ignore_malformed": true,
      "type": "integer"
    },
    "cacheable": {
      "type": "keyword"
    },
    "state": {
      "type": "keyword"
    },
    "reqEndTimeMSec": {
      "coerce": true,
      "ignore_malformed": true,
      "type": "integer"
    },
    "reqHost": {
      "type": "keyword"
    },
    "reqPath": {
      "type": "keyword"
    },
    "cliIP": {
      "type": "ip"
    },
    "cookie": {
      "type": "keyword"
    },
    "turnAroundTimeMSec": {
      "coerce": true,
      "ignore_malformed": true,
      "type": "integer"
    },
    "tlsVersion": {
      "type": "keyword"
    },
    "cacheStatus": {
      "type": "keyword"
    },
    "serverCountry": {
      "type": "keyword"
    },
    "cp": {
      "type": "keyword"
    },
    "ewExecutionInfo": {
      "type": "text"
    },
    "version": {
      "type": "version"
    },
    "customField": {
      "type": "keyword"
    },
    "edgeIP": {
      "type": "ip"
    },
    "reqId": {
      "type": "keyword"
    },
    "maxAgeSec": {
      "coerce": true,
      "ignore_malformed": true,
      "type": "integer"
    },
    "ewUsageInfo": {
      "type": "text"
    },
    "bytes": {
      "coerce": true,
      "ignore_malformed": true,
      "type": "integer"
    },
    "proto": {
      "type": "keyword"
    },
    "uncompressedSize": {
      "coerce": true,
      "ignore_malformed": true,
      "type": "integer"
    },
    "xForwardedFor": {
      "type": "keyword"
    },
    "breadcrumbs": {
      "type": "text"
    },
    "queryStr": {
      "type": "keyword"
    },
    "statusCode": {
      "type": "keyword"
    }
  }
}
},
"composed_of": [],
"priority": 300
}

Cheers,

hokamoto commented 11 months ago

Hi @atefhaloui,

Thank you for your comments. Also, sorry for the delay in responding.

I have modified that StackScript so that Elasticsearch uses dynamic index names. https://cloud.linode.com/stackscripts/1059555

That StackScript also automatically assigns an ILM policy named datastream2-ilm to that index. You can control your old data by editing this ILM policy. The alias name is still datastream2, so you do not need to change your Akamai DS2 configuration. Unfortunately, migrating existing Elasticsearch setup built with the old StackScript to the dynamic index setup is not easy and beyond my capabilities. Sorry for the inconvenience, but you will need to build a new visualization environment from scratch with the new StackScript.

https://github.com/hokamoto/stackscripts/commit/945c77da758298466f09a57e58fefb04b97e4cdb

atefhaloui commented 11 months ago

Hi @hokamoto, I've just forked a new VM with the updated script and I will check if all is working correctly after the stream rollover. I've juste made 2 modifications in the script to avoid duplication of system parameters in case of reinstall:

cat > /etc/sysctl.d/20-elasticsearch.conf <<EOF
vm.max_map_count=262144
net.ipv4.tcp_retries2=5
EOF

cat > /etc/security/limits.d/20-elasticsearch.conf <<EOF
elasticsearch  -  nofile  65535
EOF

Thank you

hokamoto commented 11 months ago

Hi @atefhaloui Thank you for the good suggestion. I reflected your advice in the StackScript and committed as https://github.com/hokamoto/stackscripts/commit/25c443b364086392db28593d6544707db3df2f42

atefhaloui commented 11 months ago

Hi, Just as a follow up: the index is correctly rolled up without any problem when 50GB is reached as specified by the ILM policy.

[2023-11-29T14:40:04,207][INFO ][o.e.x.c.i.WaitForRolloverReadyStep] [xxx-xxx-xxx-xxx.ip.linodeusercontent.com] index [datastream2-000001] is ready for rollover, conditions: [{[max_age: 30d]=false, [max_primary_shard_docs: 200000000]=false, [min_docs: 1]=true, [max_primary_shard_size: 50gb]=true}]
[2023-11-29T14:40:04,208][INFO ][o.e.x.i.IndexLifecycleTransition] [xxx-xxx-xxx-xxx.ip.linodeusercontent.com] moving index [datastream2-000001] from [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] to [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] in policy [datastream2-ilm]
[2023-11-29T14:40:04,247][INFO ][o.e.c.m.MetadataCreateIndexService] [xxx-xxx-xxx-xxx.ip.linodeusercontent.com] [datastream2-000002] creating index, cause [rollover_index], templates [logs-akamai.datastream2], shards [1]/[1]
[2023-11-29T14:40:04,321][INFO ][o.e.i.m.MapperService    ] [xxx-xxx-xxx-xxx.ip.linodeusercontent.com] [datastream2-000002] reloading search analyzers
[2023-11-29T14:40:04,327][INFO ][o.e.x.i.IndexLifecycleTransition] [xxx-xxx-xxx-xxx.ip.linodeusercontent.com] moving index [datastream2-000002] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [datastream2-ilm]
[2023-11-29T14:40:04,328][INFO ][o.e.x.i.IndexLifecycleTransition] [xxx-xxx-xxx-xxx.ip.linodeusercontent.com] moving index [datastream2-000001] from [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] to [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] in policy [datastream2-ilm]
[2023-11-29T14:40:04,373][INFO ][o.e.x.i.IndexLifecycleTransition] [xxx-xxx-xxx-xxx.ip.linodeusercontent.com] moving index [datastream2-000001] from [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] to [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] in policy [datastream2-ilm]
[2023-11-29T14:40:04,374][INFO ][o.e.x.i.IndexLifecycleTransition] [xxx-xxx-xxx-xxx.ip.linodeusercontent.com] moving index [datastream2-000001] from [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] to [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] in policy [datastream2-ilm]
[2023-11-29T14:40:04,424][INFO ][o.e.c.m.MetadataMappingService] [xxx-xxx-xxx-xxx.ip.linodeusercontent.com] [datastream2-000002/DZHXeGRjTKWSka0w3ebg7w] update_mapping [_doc]
[2023-11-29T14:40:04,446][INFO ][o.e.x.i.IndexLifecycleTransition] [xxx-xxx-xxx-xxx.ip.linodeusercontent.com] moving index [datastream2-000002] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"set_priority","name":"set_priority"}] in policy [datastream2-ilm]
[2023-11-29T14:40:04,447][INFO ][o.e.x.i.IndexLifecycleTransition] [xxx-xxx-xxx-xxx.ip.linodeusercontent.com] moving index [datastream2-000001] from [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] to [{"phase":"hot","action":"complete","name":"complete"}] in policy [datastream2-ilm]
[2023-11-29T14:40:04,493][INFO ][o.e.x.i.IndexLifecycleTransition] [xxx-xxx-xxx-xxx.ip.linodeusercontent.com] moving index [datastream2-000002] from [{"phase":"hot","action":"set_priority","name":"set_priority"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [datastream2-ilm]
[2023-11-29T14:40:04,511][INFO ][o.e.x.i.IndexLifecycleTransition] [xxx-xxx-xxx-xxx.ip.linodeusercontent.com] moving index [datastream2-000002] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [datastream2-ilm]

Thank you

hokamoto / stackscripts

Attaching an ILM #1