Elasticsearch migration from 7.17 to 8.14

[jphilou]

ElasticSearch Upgrade from v7.22 to V8.14 for docker env. like dev one.

The docker-compose file has been updated in order to facilitate and automotive plugins installation and prepare the environment for migration. The ElasticSearch 7.22 is still build and run but would not be the main ES for the service. It could be decommissioned after migration. docker-compose file contains as well :

• api building with integration of the requirements.txt for python pip installation (component updates)
• ES 7.22 listening on port 9201 instead of 9200 : > elasticsearch72
• redis no change
• postgresql no change
• ES 8.14 listening on port 9200 and 9300 (elasticsearch8.Dockerfile) > elasticsearch
• Logstash 8.14 for data migration purpose. (logstash8.Dockerfile)

migration steps

1. build the environment : docker-compose build
2. Launch the environment : docker-compose up -d
3. Create the index and its mapping : ./scripts/esUpdateMappings78.sh
4. Data Migration : docker-compose start logstash8 It will stop by itself when migration would be achieved
5. postgresql shema up to date: docker-compose exec api .build/venv/bin/alembic upgrade head

post migration

if all the process has been passed without issues. you can comment or remove in the docker-compose.yml services :

• logstash
• elasticsearch72

special note

ElasticSearch 8 integrate new security settings and configuration by design like:

• https
• user account (id, pwd)

in the docker for dev, I let a basic authentification working and keep http.

Base upon the way you make the configuration some adjustments have to be realized.

• with the docker provided:

  • elasticsearch listen on http => app config file : elasticsearch_sheme = http && tests_elasticsearch_scheme = http https instead for tls
  • elastic user elastic have a bootstrap passwd define : elasticsearch/bin/elasticsearch-keystore add "bootstrap.password"value elastic2024

• in app config file, it is necessary to report this authentification :

  • elasticsearch_user = elastic & elasticsearch_passwd = elastic2024
  • tests_elasticsearch_user = elastic & tests_elasticsearch_passwd = elastic2024

• in the script ./scripts/esUpdateMapSettings78.sh for mapping update

• in the logstash pipeline file for data migration purpose, it is also required to report this authentification as well :

  
  output
  {
  elasticsearch
    {
        hosts => [elasticsearch]
        index => "c2corg_%{[c2corg_doc_type]}"
        #document_type => "_doc"
        document_id => "%{[@metadata][_id]}"
        user => 'elastic'
        password => 'elastic2024'
    }
  } 

[codacy-production[bot]]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
:white_check_mark: +0.04%	:white_check_mark: 99.11%

Coverage variation details

| | Coverable lines | Covered lines | Coverage | | ------------- | ------------- | ------------- | ------------- | | Common ancestor commit (24e010e996d803ee11afc6937955a1e56339ac5d) | 19853 | 18407 | 92.72% | | | Head commit (efb3ddaeb795dd1c8bf86cc4fd54010ebb7d3b7e) | 19960 (+107) | 18515 (+108) | 92.76% (**+0.04%**) | **Coverage variation** is the difference between the coverage for the head and common ancestor commits of the pull request branch: ` - `

Diff coverage details

| | Coverable lines | Covered lines | Diff coverage | | ------------- | ------------- | ------------- | ------------- | | Pull request (#1826) | 225 | 223 | **99.11%** | **Diff coverage** is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: `/ * 100%`

See your quality gate settings    Change summary preferences

---

_{:rocket: Don’t miss a bit, follow what’s new on Codacy.}

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}