query autocompletion – add handler to viewer http backend

[antonkovalenko]

Handler interfaces:

1) Database list (optional handler, required by YQ).

params:

  prefix STRING?,
  limit NUMBER

returns:

  success: BOOLEAN
  result?: {
      entites: {name: STRING}[]
      total: NUMBER
  },
  error?: {
    message: STRING
  }

Questions:

• fuzzy-search

2) Schema object list - must return directories, tables, views, topics, etc

params:

  database?: STRING,
  prefix STRING?,
  limit NUMBER

returns:

  success: BOOLEAN,
  result?: {
      entites: {name: STRING, type: STRING}[],
      total: NUMBER
  },
  error?: {
    message: STRING
  }

3) Table schema.

Params:
  table: STRING[],
  prefix STRING?,
  limit NUMBER

Returns:

  success: BOOLEAN
  result?: {
      entites: {name: STRING, type: ANY, tableName: STRING}[] *(when fields are displayed initial sort order must be preserved)*
  },
  error?: {
    message: STRING
  }

Questions:

• do we need prefix and limit? what about column tables and olap?
• fuzzy-поиск

4) List of global functions, UDF, table functions

Returns:
  success: BOOLEAN,
  result?: {
      entites: {name: STRING, type: 'global' | 'table' | 'udf'}[] *(type is requires for :: suggestions)*
  },
  error?: {
    message: STRING
  }

Questions:

• is one handler enough?
• do we need parameters like in other functions? does list of functions depend on database, path?

[StekPerepolnen]

one handler for database, tables and scheme autocomplete /viewer/json/autocomplete schema. Params:

  database: STRING?,
  table: STRING[],
  prefix STRING?,
  limit NUMBER

Returns:

  success: BOOLEAN
  result?: {
      entites: {name: STRING, type: STRING, tableName: STRING}[] *(when fields are displayed initial sort order must be preserved)*
  },
  error: STRING[]

[StekPerepolnen]

https://github.com/ydb-platform/ydb/pull/2677

[Raubzeug]

After testing on live cluster I have a few questions: 1) Handler works different if use post and get methods. For example: GET /viewer/json/autocomplete?database=%2Flocal&prefix=local%2F - response Error: ["Inner errors while collected information"] POST /viewer/json/autocomplete with params {database: '/local', prefix: 'local/'} crashes cluster.

2) If request without params, I expect list of databases with default limit, but receive empty result.

3) If request with only prefix, handler doesn't response with databases list: GET /viewer/json/autocomplete?prefix=local

response:

{
  "Success": true,
  "Result": {}
}

4) If request with params {database: '/local'}, receive list with one directory '.sys_health'. Where is .sys?

request: GET /viewer/json/autocomplete?database=%2Flocal

response:

{
  "Success": true,
  "Result": {
    "Total": 1,
    "Entities": [
      {
        "Name": ".sys_health",
        "Type": "directory",
        "Parent": "/local"
      }
    ]
  }
}

5) GET method not working with slashes. GET /viewer/json/autocomplete?database=%2Flocal&prefix=local%2F - response Error: ["Inner errors while collected information"]

6) It would be great to have tests with handler's behavior.

For example typical user story.

This is my cluster's structure.

a) request with empty params. Receive list with database local/ b) params: {database: 'local/'}. Receive list with two dirs - .sys_health and .sys. c) params: {database: 'local/', prefix: 'local/.sys_hea'}. Receive list one dir - .sys_health. d) params: {database: 'local/', prefix: 'local/.sys_health/'}. Receive list one table test.

[Raubzeug]

Agreed with @StekPerepolnen to add enum for Entity type in TQueryAutocomplete.

[StekPerepolnen]

@Raubzeug https://github.com/ydb-platform/ydb/pull/3784 I have made the following changes based on comments:

• In case of failure with path resolution, there should be an error indicating the reason and the path that the backend is trying to check. Error receiving Navigate response:
• Changed the handling logic for bad paths. Items 2-5 require a re-check.
• Added tests and an enum with all types.

For requests to the autocomplete endpoint, it is expected that:

• The database parameter must always start with a /, and there should be no / at the end; I will remove these strict restrictions.
• In prefix, there is no need to include the part of the path that relates to the database.

In case of problems, the error will indicate bad constructed path.

[Raubzeug]

@StekPerepolnen there's still a few questions: 1) Limit seems not to work. I pass 1000, but receive 10 (100% that total amount is more then 10) 2) Search should be case insensitive. 3) Fuzzy-search works a little bit strange. As for me if the substring is a strict part of the string (especially if string starts from the substring), such results should come first. 4) .sys folder is not in results, check this please. 5) It will be handy to have a boolean flag if an entity has children.