panic: runtime error: invalid memory address or nil pointer dereference 这个错误是什么意思？

[wajika]

 ./esm --source=http://192.168.10.141:9200 --dest=http://192.168.10.26:9200 --source_auth=elastic:elastic --src_indexes=* --copy_mappings --copy_settings --shards=3 --log=debug
[01-23 11:56:24] [DBG] [main.go:536,ClusterVersion] {
  "name" : "es-node03",
  "cluster_name" : "es",
  "cluster_uuid" : "Bi43wnN8QECy8o9lmxJ1IQ",
  "version" : {
    "number" : "7.12.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "78722783c38caa25a70982b5b042074cde5d3b3a",
    "build_date" : "2021-03-18T06:17:15.410153305Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

[01-23 11:56:24] [DBG] [main.go:120,main] source es is V7,7.12.0
[01-23 11:56:35] [DBG] [scroll.go:131,Next] scroll result is empty
[01-23 11:56:35] [DBG] [main.go:197,func1] closing doc chan
[01-23 11:56:35] [DBG] [main.go:536,ClusterVersion] {
  "name" : "es-node01",
  "cluster_name" : "es",
  "cluster_uuid" : "GHAC3nVXTkmyba_6MYCXIA",
  "version" : {
    "number" : "7.16.3",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "4e6e4eab2297e949ec994e688dad46290d018022",
    "build_date" : "2022-01-06T23:43:02.825887787Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

[01-23 11:56:35] [DBG] [main.go:264,main] target es is V7,7.16.3
[01-23 11:56:35] [DBG] [main.go:294,main] start process with mappings
[01-23 11:56:35] [DBG] [v0.go:53,ClusterHealth] http://192.168.10.141:9200/_cluster/health
[01-23 11:56:35] [DBG] [v0.go:54,ClusterHealth] {"cluster_name":"es","status":"green","timed_out":false,"number_of_nodes":4,"number_of_data_nodes":4,"active_primary_shards":280,"active_shards":560,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}
[01-23 11:56:35] [DBG] [v0.go:53,ClusterHealth] http://192.168.10.26:9200/_cluster/health
[01-23 11:56:35] [DBG] [v0.go:54,ClusterHealth] {"cluster_name":"es","status":"green","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":398,"active_shards":530,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}
Scroll 10000 / 1755555077 [>-----------------------------------------------------------------------------------------------------------------------------------------------------]   0.00% 0s
Output  0 / 1755555077 [------------------------------------------------------------------------------------------------------------------------------------------------------------]   0.00%
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x90 pc=0x585820]

goroutine 1 [running]:
regexp.(*Regexp).doExecute(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc02c7f9ca0, 0x11, 0x0, 0x0, ...)
    /usr/local/go/src/regexp/exec.go:527 +0x560
regexp.(*Regexp).doMatch(...)
    /usr/local/go/src/regexp/exec.go:514
regexp.(*Regexp).MatchString(...)
    /usr/local/go/src/regexp/regexp.go:525
main.(*ESAPIV7).GetIndexMappings(0xc0000963c0, 0xac3800, 0x7ffd9b061707, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/medcl/go/src/infini.sh/esm/v7.go:180 +0x825
main.main()
    /Users/medcl/go/src/infini.sh/esm/main.go:325 +0xb2a

[wajika]

我换了一条命令 执行到一半就被killed了，也没有报原因

./esm --source=http://192.168.10.141:9200 --dest=http://192.168.10.26:9200 --source_auth=elastic:elastic  --copy_mappings --copy_settings --log=debug --force --src_indexes=aliyun-k8s-prod-cluster-2022.01
.......
[01-23 13:29:30] [INF] [main.go:458,main] settings/mappings migration finished.
[01-23 13:29:30] [INF] [main.go:474,main] start data migration..
Scroll 1090000 / 1798471 [=========================================================================================>---------------------------------------------------------]  60.61% 00m35s
Bulk 207340 / 1798471 [=================>------------------------------------------------------------------------------------------------------------------------------------]  11.53% 06m55s
Killed

[wajika]

另外几个问题，我需要备份全部的索引到另一个集群上，好像 --src_indexes=_all 不是备份所有索引。 是否有什么其他办法呢？ 还有 可视化，template，dashboard 这些非index数据如何备份？

[wajika]

我试了直接同步 7.12 > 7.16 会有Can't update non dynamic settings的问题 sudo /home/wajika/esm --workers=5 --bulk_size=5 --source=http://192.168.10.141:9200 --source_auth=elastic:elastic --src_indexes=$INDEX --dest=http://192.168.10.26:9200 --dest_index=$INDEX --log=debug --copy_mappings --copy_settings | sudo tee >> /home/wajika/esm.log

[01-24 01:19:31] [ERR] [main.go:421,main] server error: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't update non dynamic settings [[index.codec]] for open indices [[apm-7.14.0-onboarding-2021.08.26/7JHivLXMS7ukWUX6NGvjIg]]"}],"type":"illegal_argument_exception","reason":"Can't update non dynamic settings [[index.codec]] for open indices [[apm-7.14.0-onboarding-2021.08.26/7JHivLXMS7ukWUX6NGvjIg]]"},"status":400}

然后尝试直接导出，再导入。 但是导入的时候没有任何日志出现，es上也没有收到任何新增数据

#!/bin/bash
for line in $(curl -u elastic:elastic -sXGET 'http://192.168.10.141:9200/_cat/indices'); do
          INDEX=$(echo $line | (awk '{print $3}'))
          #  SHARD=$(echo $line | (awk '{print $2}'))
            sudo /home/wajika/esm -o dump.json --workers=5 --bulk_size=1 --source=http://192.168.10.141:9200 --source_auth=elastic:elastic --src_indexes=$INDEX --log=debug --copy_mappings --copy_settings
done

导入

#!/bin/bash
for line in $(curl -u elastic:elastic -sXGET 'http://192.168.10.141:9200/_cat/indices'); do
          INDEX=$(echo $line | (awk '{print $3}'))
          #  SHARD=$(echo $line | (awk '{print $2}'))
            sudo /home/wajika/esm -i dump.json --workers=5 --bulk_size=5 --dest=http://192.168.10.26:9200 --dest_index==$INDEX --log=debug
done

[medcl]

--copy_mappings --copy_settings 不适应跨版本的情况，建议只拷贝数据，mapping 和 setting 通过模板提前做好，避免工具拷贝完才发现配置不对

[medcl]

esm 所在机器内存不够的话，设置 --buffer_count=1000 控制下 buffer 的内存占用

[wajika]

esm 所在机器内存不够的话，设置 --buffer_count=1000 控制下 buffer 的内存占用

我的需求是想把7.12的所有数据，包括 dashboard，template，各种设置都同步到 7.16集群上去，有没有什么最佳实践？

[wajika]

我尝试 使用 在7.12 集群上制作 snapshot，然后挂载nfs disk 到7.16集群上，但是7.16上无法直接识别到这个snapshot。 我也试了先在7.16上create 一个snapshot，然后狸猫换太子，用7.12的数据替换掉 7.16产生的snapshot，但是7.16集群 会提示 cluster id 找不到。

[wajika]

esm 所在机器内存不够的话，设置 --buffer_count=1000 控制下 buffer 的内存占用

分别试了8G和16G的机器，我观察了内存使用量，不是很大。 你的意思是 如果是将 dump.json 导入到ES 的话，内存需要更大？ 我的dump.json200多G，那么大概需要给多少内存？ 顺便一提，如果是 导出dump.json的话，那么应该不需要很多内存把？

[medcl]

更新了最新的包，可以试试

[wajika]

@medcl 好的