search

qdrant / qdrant

Qdrant - High-performance, massive-scale Vector Database for the next generation of AI. Also available in the cloud https://cloud.qdrant.io/
https://qdrant.tech
Apache License 2.0
20.19k stars 1.37k forks source link

ERROR qdrant::startup: Panic occurred hs.commit 64 is out of range [0, 0], raft_id: 548108420710242 #4683

Closed muzian666 closed 2 months ago

muzian666 commented 3 months ago

Current Behavior

Get an error when restart one of the deploy when using distribute deployment mode.

ERROR qdrant::startup: Panic occurred in file /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/raft-0.7.0/src/raft.rs at line 2786: hs.commit 64 is out of range [0, 0], raft_id: 548108420710242

Steps to Reproduce

  1. I'm currently using Kubernetes, and using Azure Blob storage to mount the /qdrant/storage
  2. Everything is fine when the first time deployment.
  3. After deployment, no need to do anything, just restart the pods, then the error occur.

Environment

  1. Kubernetes with Ubuntu as system
  2. Qdrant version: 1.10.1 (I also have the issue when using 1.10.0, 1.8.0, not test 1.9 serious)
  3. Service are using ClusterIP

Error Detail:

          _                 _    
  __ _  __| |_ __ __ _ _ __ | |_  
 / _` |/ _` | '__/ _` | '_ \| __| 
| (_| | (_| | | | (_| | | | | |_  
 \__, |\__,_|_|  \__,_|_| |_|\__| 
    |_|                           

Version: 1.10.1, build: 4aac0231
Access web UI at http://localhost:8080/dashboard

2024-07-17T09:57:59.863067Z  INFO storage::content_manager::consensus::persistent: Loading raft state from ./storage/raft_state.json    
2024-07-17T09:57:59.863809Z DEBUG storage::content_manager::consensus::persistent: State: Persistent { state: RaftState { hard_state: HardState { term: 94, vote: 548108420710242, commit: 64 }, conf_state: ConfState { voters: [527504786059992, 1128910046478821, 548108420710242], learners: [], voters_outgoing: [], learners_next: [], auto_leave: false } }, latest_snapshot_meta: SnapshotMetadataSer { term: 0, index: 0 }, apply_progress_queue: EntryApplyProgressQueue(Some((65, 64))), peer_address_by_id: RwLock { data: {527504786059992: http://alt-qdrant-db-follower-1.qdrant.svc.cluster.local:8082/, 548108420710242: http://alt-qdrant-db-follower-2.qdrant.svc.cluster.local:8082/, 1128910046478821: http://alt-qdrant-db-leader.qdrant.svc.cluster.local:8082/} }, peer_metadata_by_id: RwLock { data: {1128910046478821: PeerMetadata { version: Version { major: 1, minor: 10, patch: 1 } }, 548108420710242: PeerMetadata { version: Version { major: 1, minor: 10, patch: 1 } }, 527504786059992: PeerMetadata { version: Version { major: 1, minor: 10, patch: 1 } }} }, this_peer_id: 548108420710242, path: "./storage/raft_state.json", dirty: false }    
2024-07-17T09:58:00.313799Z  INFO storage::content_manager::toc: Loading collection: alt_record_document_pdf_embeddings    
2024-07-17T09:58:46.879778Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Segment vectors: 1420    
2024-07-17T09:58:46.879815Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Estimated segment size 8 MB    
2024-07-17T09:58:46.916302Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Segment vectors: 1420    
2024-07-17T09:58:46.916319Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Estimated segment size 16 MB    
2024-07-17T09:58:47.246784Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Segment vectors: 1426    
2024-07-17T09:58:47.246811Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Estimated segment size 16 MB    
2024-07-17T09:58:47.301508Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Segment vectors: 1426    
2024-07-17T09:58:47.301529Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Estimated segment size 8 MB    
2024-07-17T09:58:47.443425Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Segment vectors: 1431    
2024-07-17T09:58:47.443465Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Estimated segment size 16 MB    
2024-07-17T09:58:47.487477Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Segment vectors: 1431    
2024-07-17T09:58:47.487494Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Estimated segment size 8 MB    
2024-07-17T09:58:47.774356Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Segment vectors: 1431    
2024-07-17T09:58:47.774379Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Estimated segment size 16 MB    
2024-07-17T09:58:47.815953Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Segment vectors: 1431    
2024-07-17T09:58:47.815970Z DEBUG segment::vector_storage::dense::simple_dense_vector_storage: Estimated segment size 8 MB    
2024-07-17T09:58:48.704046Z  INFO collection::shards::local_shard: Recovering collection alt_record_document_pdf_embeddings: 0/1 (0%)    
2024-07-17T09:59:03.005550Z  INFO collection::shards::local_shard: Recovered collection alt_record_document_pdf_embeddings: 1/1 (100%)    
2024-07-17T09:59:05.888417Z DEBUG storage::content_manager::toc: Auto adjusting update rate limit to 4 parallel update requests    
2024-07-17T09:59:05.888495Z DEBUG qdrant: Loaded collection: alt_record_document_pdf_embeddings    
2024-07-17T09:59:09.441618Z DEBUG qdrant::consensus: Local raft state found - bootstrap and uri cli arguments were ignored    
2024-07-17T09:59:09.441640Z DEBUG qdrant::consensus: Local raft state found - skipping initialization    
2024-07-17T09:59:09.441924Z ERROR qdrant::startup: Panic backtrace: 
   0: qdrant::startup::setup_panic_hook::{{closure}}
   1: std::panicking::rust_panic_with_hook
   2: std::panicking::begin_panic_handler::{{closure}}
   3: std::sys_common::backtrace::__rust_end_short_backtrace
   4: rust_begin_unwind
   5: core::panicking::panic_fmt
   6: qdrant::consensus::Consensus::new
   7: qdrant::consensus::Consensus::run
   8: qdrant::main
   9: std::sys_common::backtrace::__rust_begin_short_backtrace
  10: main
  11: <unknown>
  12: __libc_start_main
  13: _start

2024-07-17T09:59:09.441939Z ERROR qdrant::startup: Panic occurred in file /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/raft-0.7.0/src/raft.rs at line 2786: hs.commit 64 is out of range [0, 0], raft_id: 548108420710242    
2024-07-17T09:59:09.443077Z DEBUG reqwest::connect: starting new connection: https://telemetry.qdrant.io/
generall commented 2 months ago

I'm currently using Kubernetes, and using Azure Blob storage to mount the /qdrant/storage

Is that right, that /qdrant/storage is shared between machines in the cloud? Likely, that Blob storage is not POSIX-compatible, so it is not supported

timvisee commented 2 months ago

We list the above mentioned requirement amongst others here: https://qdrant.tech/documentation/guides/installation/

muzian666 commented 2 months ago

OK, Understand. Thanks.