milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.56k stars 2.92k forks source link

[Enhancement]: Optimize expression parsing efficiency #36672

Open xiaocai2333 opened 1 month ago

xiaocai2333 commented 1 month ago

Is there an existing issue for this?

What would you like to be added?

Optimize expression parsing efficiency.

  1. Optimize the time of parsing Chinese characters.
  2. Pass in the expression parameters to be queried through the template.
    • such as expr=pk > {age} and city in {city}, search_params={age:3, city:['beijing","shanghai"]}

Why is this needed?

No response

Anything else?

No response

xiaofan-luan commented 1 month ago

Excited about the template feature!

zhengbuqian commented 3 weeks ago

we now support nullable fields, how do we handle null in template?

xiaocai2333 commented 3 weeks ago

we now support nullable fields, how do we handle null in template?

Currently, our expression does not support null values. We can discuss it further when null support is added in the future. /cc @smellthemoon

xiaofan-luan commented 3 weeks ago

null support is already added?

xiaofan-luan commented 3 weeks ago

did you mean isNull and isNotNull? can we do a == null?

zhengbuqian commented 3 weeks ago

did you mean isNull and isNotNull?

yes. we should be able to filter based on null. chatted with @smellthemoon offline, we don't currently support this.

Maybe we should not allow null in template, we can simply use a NULL keyword:

smellthemoon commented 3 weeks ago

did you mean isNull and isNotNull?

yes. we should be able to filter based on null. chatted with @smellthemoon offline, we don't currently support this.

Maybe we should not allow null in template, we can simply use a NULL keyword:

  • to use null only: a == NULL or a != NULL
  • to use null with non null values: a == 100 || a == NULL
  • to use null with template: a in [{accepted_values}] || a == NULL

In fact, in the future, we will still not support expressions like == null and != null, which will directly report errors at the parse layer. Exposing the NULL keyword will bring a lot of complexity. It needs to be identified in all expressions, and the respective logical relationships need to be considered in complex expressions. Therefore, in the future, we will only support the form of isNull(field) and isNotNull(field).

zhengbuqian commented 3 weeks ago

did you mean isNull and isNotNull?

yes. we should be able to filter based on null. chatted with @smellthemoon offline, we don't currently support this. Maybe we should not allow null in template, we can simply use a NULL keyword:

  • to use null only: a == NULL or a != NULL
  • to use null with non null values: a == 100 || a == NULL
  • to use null with template: a in [{accepted_values}] || a == NULL

In fact, in the future, we will still not support expressions like == null and != null, which will directly report errors at the parse layer. Exposing the NULL keyword will bring a lot of complexity. It needs to be identified in all expressions, and the respective logical relationships need to be considered in complex expressions. Therefore, in the future, we will only support the form of isNull(field) and isNotNull(field).

that is fine. as long as we have a way of checking null, we can remove the need to support null in expression template.