milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
31.06k stars 2.95k forks source link

Can't make time travel search work #21717

Closed Mahanchello closed 1 year ago

Mahanchello commented 1 year ago

Is there an existing issue for this?

Environment

- Milvus version: 2.2.x
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.2.1
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

i'm trying to make time travel search work using this example . The code is the same as in the example, the only thing before loading the collection, i'm creating an index, otherwise search won't work. When i run the code, i get the following error :

raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=only support to travel back to 0s so far, but got 2s)>

Expected Behavior

I'm expecting to get data back that fits the timestamp specified in the search event.

Steps To Reproduce

Run the following code 

from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
import random

collection_name = "test_time_travel"

conns = connections.connect(
  alias=<alias>,
  host=<host>
  port=<port>
) 

schema = CollectionSchema([
  FieldSchema("pk", DataType.INT64, is_primary=True),
  FieldSchema("example_field", dtype=DataType.FLOAT_VECTOR, dim=2)
])

collection = Collection(collection_name, schema)

data = [
  [i for i in range(10)],
  [[random.random() for _ in range(2)] for _ in range(10)],
]
batch1 = collection.insert(data)

data = [
  [i for i in range(10, 20)],
  [[random.random() for _ in range(2)] for _ in range(9)],
]
data[1].append([1.0,1.0])
batch2 = collection.insert(data)

index = {
    "index_type": "IVF_FLAT",
    "metric_type": "L2",
    "params": {"nlist": 128},
}
collection.create_index("example_field", index)
collection.load()
search_param = {
  "data": [[1.0, 1.0]],
  "anns_field": "example_field",
  "param": {"metric_type": "L2"},
  "limit": 10,
  "travel_timestamp": batch1.timestamp,
}
res = collection.search(**search_param)

get error:

raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=only support to travel back to 0s so far, but got 2s)>


### Milvus Log

Exception gets thrown here https://github.com/milvus-io/pymilvus/blob/master/pymilvus/client/grpc_handler.py#L431 

### Anything else?

_No response_
filip-halt commented 1 year ago

Hi, I think the issue is that time travel is disabled in 2.2 by default.

image

Release notes found here.

xiaofan-luan commented 1 year ago

Hi, I think the issue is that time travel is disabled in 2.2 by default.

image

Release notes found here.

You r right, need to change common.retentionDuration to at least 3 seconds if we are traveling back to 2 sec

yanliang567 commented 1 year ago

/assign @Mahanchello Please retry as suggested above.

Mahanchello commented 1 year ago

/assign @Mahanchello Please retry as suggested above.

@yanliang567 @xiaofan-luan thank you for pointing out, totally missed that. Maybe it makes sense to add a note in the docs for time travel that it's disabled in v2.2? anyway, thank you! ❤️

yanliang567 commented 1 year ago

@Mahanchello great to hear that. Any breaking changes would be highlighted in release notes.