search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
29.78k stars 2.86k forks source link

[Bug]: standalone panic with error `invalid memory address or nil pointer dereference` during test after chaos #34375

Closed zhuwenxing closed 3 months ago

zhuwenxing commented 3 months ago

Is there an existing issue for this?

Environment

- Milvus version:master-20240701-87bccb1a-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

[2024/07/02 20:52:50.293 +00:00] [INFO] [compaction/load_stats.go:113] ["Successfully load pk stats"] [segmentID=450875350949038792] [time=12.124248ms] [size=1429504]
[2024/07/02 20:52:50.293 +00:00] [INFO] [metacache/meta_cache.go:289] ["metacache does not have segment, add it"] [segmentID=450875350949038792]
[2024/07/02 20:52:50.293 +00:00] [INFO] [metacache/meta_cache.go:301] ["remove dropped segment"] [segmentID=450874971742697894]
[2024/07/02 20:52:50.293 +00:00] [INFO] [metacache/meta_cache.go:301] ["remove dropped segment"] [segmentID=450874971741697884]
[2024/07/02 20:52:50.293 +00:00] [INFO] [metacache/meta_cache.go:301] ["remove dropped segment"] [segmentID=450874971742898927]
[2024/07/02 20:52:50.293 +00:00] [INFO] [metacache/meta_cache.go:301] ["remove dropped segment"] [segmentID=450875350949438085]
[2024/07/02 20:52:50.293 +00:00] [INFO] [datacoord/session_manager.go:256] ["success to sync segments"] [nodeID=2] [planID=0]
[2024/07/02 20:52:50.293 +00:00] [INFO] [datacoord/sync_segments_scheduler.go:149] ["sync segments success"] [collectionID=450874971741889147] [partitionID=450874971741889148] [channelName=by-dev-rootcoord-dml_10_450874971741889147v1] [nodeID=2] [segments="[450874971741695845,450875350949036470,450875350949443485,450874971741692920,450875350949037202,450874971741692292,450875350949038792,450875350949443372,450874971741694456,450874971741697849]"]
[2024/07/02 20:52:50.293 +00:00] [INFO] [datanode/services.go:283] ["DataNode receives SyncSegments"] [traceID=c0595c782344373541fca702166dd0c2] [planID=0] [nodeID=2] [collectionID=450874971742092756] [partitionID=450874971742092757] [channel=by-dev-rootcoord-dml_13_450874971742092756v0]
[2024/07/02 20:52:50.294 +00:00] [ERROR] [conc/options.go:54] ["Conc pool panicked"] [panic="runtime error: invalid memory address or nil pointer dereference"] [stack="github.com/milvus-io/milvus/pkg/util/conc.(*poolOption).antsOptions.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/conc/options.go:54\ngithub.com/panjf2000/ants/v2.(*goWorker).run.func1.1\n\t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:54\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\ngithub.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1.1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:74\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:261\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:861\ngithub.com/milvus-io/milvus/internal/metastore/kv/binlog.DecompressBinLog\n\t/go/src/github.com/milvus-io/milvus/internal/metastore/kv/binlog/binlog.go:143\ngithub.com/milvus-io/milvus/internal/datanode.(*DataNode).SyncSegments.func1\n\t/go/src/github.com/milvus-io/milvus/internal/datanode/services.go:320\ngithub.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\ngithub.com/panjf2000/ants/v2.(*goWorker).run.func1\n\t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67"]
I20240702 20:52:50.295473  3582 InvertedIndexTantivy.cpp:127] [SERVER][Upload][milvus] index file: /var/lib/milvus/data/querynode/index_files/450875350949440007/1/ef7b18e79be24d21a432b0b0f3509630.idx added
I20240702 20:52:50.295585  3582 InvertedIndexTantivy.cpp:123] [SERVER][Upload][milvus] trying to add index file: /var/lib/milvus/data/querynode/index_files/450875350949440007/1/ef7b18e79be24d21a432b0b0f3509630.pos
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x429b494]

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-kafka-cron/detail/chaos-test-kafka-cron/15404/pipeline log: artifacts-standalone-pod-failure-15404-server-logs.tar.gz

Anything else?

No response

weiliu1031 commented 3 months ago

should be fixed by https://github.com/milvus-io/milvus/pull/34389

weiliu1031 commented 3 months ago

please verify this with latest images

weiliu1031 commented 3 months ago

/assign @zhuwenxing

zhuwenxing commented 3 months ago

verified and fixed in master-20240704-fcafdb6d-amd64