[Bug] Query es catalog error

lsy3993 commented 1 year ago

Search before asking

[X] I had searched in the issues and found no similar issues.

Version

doris master

What's Wrong?

query es catalog error.

 ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.6)[CANCELLED][RUNTIME_ERROR]Expected value of type: STRING; but found type: Array; Document slice is : []

What You Expected?

query normally

How to Reproduce?

At first, apply the two PRs in master, because the PRs solve other es problem.

  https://github.com/apache/doris/pull/22046/files
  https://github.com/apache/doris/pull/22279/files

Then start es.

mapping

curl -XPUT "http://127.0.0.1:9200/test_table8"  -H "Content-Type:application/json" -d '{
      "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 0,
        "index": {
          "max_result_window": 864000000
        }
      },
      "mappings": {
        "dynamic": "strict",
        "properties": {
          "id": {
            "type": "long"
          },
          "uuid": {
            "type": "keyword"
          },
          "version": {
            "type": "keyword"
          },
          "desc": {
            "type": "keyword"
          },
          "reserve": {
            "type": "keyword"
          },
          "is_new": {
            "type": "keyword"
          }
        }
      }
    }'

add column

curl -X POST "http://127.0.0.1:9200/test_table8/_mappings"  -H "Content-Type:application/json" -d '{
      "properties": {
          "company_id": {
            "type": "long"
          },
          "dacs_license_type": {
            "type": "keyword"
          },
          "os": {
            "type": "keyword"
          },
          "dacs_version": {
            "type": "keyword"
          },
          "session_id": {
            "type": "keyword"
          },
          "original_data_trace_id": {
            "type": "keyword"
          }
      }
    }'

put data in es

curl "http://127.0.0.1:9200/test_table8/_doc/1" -H "Content-Type:application/json" -X POST -d '{
    "company_id": 0,
    "dacs_license_type": "abc",
    "os": "Windows 10 Pro 2009",
    "is_new": 0,
    "reserve": 0,
    "id": 12,
    "uuid": "A52C86027FCF3E44719C62E1EB4E30CDDC403464",
    "desc": 2724,
    "dacs_version": 0,
    "original_data_trace_id": 0,
    "session_id": 0,
    "version": "0123"
}'

put another data in es, but this data does not contain all columns

curl "http://127.0.0.1:9200/test_table8/_doc/2" -H "Content-Type:application/json" -X POST -d '{
    "company_id": 0,
    "os": "Windows 10 Pro 2009",
    "is_new": 0,
    "reserve": 0,
    "uuid": "A52C86027FCF3E44719C62E1EB4E30CDDC403464",
    "desc": 2724,
    "dacs_version": 0,
    "original_data_trace_id": 0,
    "session_id": 0
}'

query es catalog

select * from test_table8;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.6)[CANCELLED][RUNTIME_ERROR]Expected value of type: STRING; but found type: Array; Document slice is : []

but if query a column which is in step 4, it's OK
```
select session_id from test_table8;
```

Anything Else?

No response

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

liugddx commented 1 year ago

Use the latest version and the problem has been solved.

bulolo commented 11 months ago

Use the latest version and the problem has been solved.

哪个版本？

liugddx commented 11 months ago

Use the latest version and the problem has been solved.

哪个版本？

2.0.2

bulolo commented 11 months ago

Use the latest version and the problem has been solved.

哪个版本？

2.0.2

今天我使用2.0.2一样有问题

select * from materials_cdc_v8 limit 1

errCode = 2, detailMessage = (172.16.0.4)[CANCELLED][RUNTIME_ERROR]Expected value of type: STRING; but found type: Array; Document slice is : []

/root/src/doris-2.0/be/src/common/stack_trace.cpp:302: StackTrace::tryCapture() @ 0x000000000ba1f197 in /root/doris/be/lib/doris_be

/root/src/doris-2.0/be/src/common/stack_trace.h:0: doris::get_stack_trace[abi:cxx11]() @ 0x000000000ba1d72d in /root/doris/be/lib/doris_be

/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11 Time: 0.053s

bulolo commented 11 months ago

还有更奇怪的

CREATE CATALOG es PROPERTIES ( "type"="es", "hosts"="http://ip:9200", );

esquery查询语句

select * from test where esquery(k4, '{ "match_phrase": { "k4": "doris on es" } }');

报错： [CANCELLED][NOT_IMPLEMENTED_ERROR]esquery only supported on es table

文档内不是支持扩展的 esquery？ https://doris.apache.org/zh-CN/docs/lakehouse/multi-catalog/es/#%E6%89%A9%E5%B1%95%E7%9A%84-esqueryfield-querydsl

liugddx commented 11 months ago

Please send me the table structure of your es

bulolo commented 11 months ago

Use the latest version and the problem has been solved.

哪个版本？

2.0.2

今天我使用2.0.2一样有问题

select * from materials_cdc_v8 limit 1

errCode = 2, detailMessage = (172.16.0.4)[CANCELLED][RUNTIME_ERROR]Expected value of type: STRING; but found type: Array; Document slice is : []

/root/src/doris-2.0/be/src/common/stack_trace.cpp:302: StackTrace::tryCapture() @ 0x000000000ba1f197 in /root/doris/be/lib/doris_be

/root/src/doris-2.0/be/src/common/stack_trace.h:0: doris::get_stack_traceabi:cxx11 @ 0x000000000ba1d72d in /root/doris/be/lib/doris_be

/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11

Time: 0.053s

{
  "materials" : {
    "mappings" : {
      "dynamic_templates" : [
        {
          "message_full" : {
            "match" : "message_full",
            "mapping" : {
              "fields" : {
                "keyword" : {
                  "ignore_above" : 2048,
                  "type" : "keyword"
                }
              },
              "type" : "text"
            }
          }
        },
        {
          "message" : {
            "match" : "message",
            "mapping" : {
              "type" : "text"
            }
          }
        },
        {
          "strings" : {
            "match_mapping_type" : "string",
            "mapping" : {
              "type" : "keyword"
            }
          }
        }
      ],
      "properties" : {
        "isPublic" : {
          "type" : "boolean"
        },
        "mfrId" : {
          "type" : "keyword"
        },
        "mfrName" : {
          "type" : "text",
          "analyzer" : "ngram_lower"
        },
        "name" : {
          "type" : "text",
          "analyzer" : "ngram_lower"
        },
        "sn" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          },
          "analyzer" : "ngram_lower"
        },
        "status" : {
          "type" : "boolean"
        }
      }
    }
  }
}

bulolo commented 11 months ago

还有更奇怪的

CREATE CATALOG es PROPERTIES ( "type"="es", "hosts"="http://ip:9200", );

esquery查询语句

select * from test where esquery(k4, '{ "match_phrase": { "k4": "doris on es" } }');

报错： [CANCELLED][NOT_IMPLEMENTED_ERROR]esquery only supported on es table

文档内不是支持扩展的 esquery？ https://doris.apache.org/zh-CN/docs/lakehouse/multi-catalog/es/#%E6%89%A9%E5%B1%95%E7%9A%84-esqueryfield-querydsl

PUT test
{
   "settings": {
      "index": {
         "number_of_shards": "1",
         "number_of_replicas": "0"
      }
   },
  "mappings": {
    "properties": {
      "k1": {
        "type": "long",
        "index": "true"
      },
      "k2": {
        "type": "date",
        "format": "yyyy-MM-dd"
      },
      "k3": {
        "type": "text",
        "analyzer": "ik_max_word",
        "search_analyzer": "ik_max_word"
      },
      "k4": {
        "type": "text",
        "analyzer": "ik_max_word",
        "search_analyzer": "ik_max_word"
      },
      "k5": {
        "type": "float"
      }
    }
  }
}

@liugddx

liugddx commented 11 months ago

还有更奇怪的 CREATE CATALOG es PROPERTIES ( "type"="es", "hosts"="http://ip:9200", ); esquery查询语句 select * from test where esquery(k4, '{ "match_phrase": { "k4": "doris on es" } }'); 报错： [CANCELLED][NOT_IMPLEMENTED_ERROR]esquery only supported on es table 文档内不是支持扩展的 esquery？ https://doris.apache.org/zh-CN/docs/lakehouse/multi-catalog/es/#%E6%89%A9%E5%B1%95%E7%9A%84-esqueryfield-querydsl
PUT test
{
   "settings": {
      "index": {
         "number_of_shards": "1",
         "number_of_replicas": "0"
      }
   },
  "mappings": {
    "properties": {
      "k1": {
        "type": "long",
        "index": "true"
      },
      "k2": {
        "type": "date",
        "format": "yyyy-MM-dd"
      },
      "k3": {
        "type": "text",
        "analyzer": "ik_max_word",
        "search_analyzer": "ik_max_word"
      },
      "k4": {
        "type": "text",
        "analyzer": "ik_max_word",
        "search_analyzer": "ik_max_word"
      },
      "k5": {
        "type": "float"
      }
    }
  }
}
@liugddx

use 2.0.3 version

liugddx commented 11 months ago

bulolo commented 11 months ago

请问2.0.3在哪里下载？我看官网最新2.0.2

liugddx commented 11 months ago

请问2.0.3在哪里下载？我看官网最新2.0.2

Will be released soon

bulolo commented 11 months ago

请问2.0.3在哪里下载？我看官网最新2.0.2

Will be released soon

所以2.0.2是不行的？

liugddx commented 11 months ago

请问2.0.3在哪里下载？我看官网最新2.0.2

Will be released soon

所以2.0.2是不行的？

yes

bulolo commented 10 months ago

2.0.3.1 selectdb0doris 并没有解决

@liugddx

bulolo commented 10 months ago

"inventory" : {
          "dynamic" : "true",
          "properties" : {
            "BJ001" : {
              "type" : "long"
            },
            "SX001" : {
              "type" : "long"
            }
          }
        },

发现是这个定义导致的，有什么解决方法吗

liugddx commented 10 months ago

inventory

This may require some adaptation. Wait a minute, i will deal with it

bulolo commented 10 months ago

inventory

This may require some adaptation. Wait a minute, i will deal with it

有什么临时的解决办法？

qidaye commented 10 months ago

@bulolo esquery is working normal in the latest branch-2.0 and master branch You can try the 2.0.4 rc release.

apache / doris