search

cloudera / hue

Open source SQL Query Assistant service for Databases/Warehouses
https://cloudera.com
Apache License 2.0
1.17k stars 366 forks source link

How can you use spark SQL to display databases and tables in the left-hand column of the database display? #850

Closed IVitamin-C closed 5 years ago

IVitamin-C commented 5 years ago

Hello, How can you use spark SQL to display databases and tables in the left-hand column of the database display? When I used spark thriftserver, port 10000, the hive metastore I used, but I can see which databases are available through SQL query, but only the default database is displayed in the left-hand column. Thanks.

TimKingNF commented 5 years ago

How you solve this question?

IVitamin-C commented 5 years ago

This is done by modifying the code in hiveserver.py, but because I did not write the code, the programmer who modified it is submitting it for official review

TimKingNF commented 5 years ago

Could you show me the code or links about this error? I can't find it in PR.

IVitamin-C commented 5 years ago

https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/hiveserver2.py

TimKingNF commented 5 years ago

I try to rebuild my hue. But the problem is still.

When I access the url http://localhost:8000/notebook/api/autocomplete/default with request's body like

snippet: {"type":"sparksql","source":"data"}
cluster: {"interface":"all","server_host":"all","type":"direct","id":"default","name":"default"}

It calls TCLIService.GetTables (https://github.com/cloudera/hue/blob/fd5cbb9edf88ace944d021ada8e0a6d3972686ca/apps/beeswax/src/beeswax/server/hive_server2_lib.py#L790), But return empty results.rows in results.

And Spark-hive-thrift2 log like this.

2019-05-20 17:43:11 INFO  HiveMetaStore:746 - 3: get_databases: default
2019-05-20 17:43:11 INFO  audit:371 - ugi=hadoop        ip=unknown-ip-addr      cmd=get_databases: default
2019-05-20 17:43:11 INFO  HiveMetaStore:746 - 3: get_databases: default
2019-05-20 17:43:11 INFO  audit:371 - ugi=hadoop        ip=unknown-ip-addr      cmd=get_databases: default
2019-05-20 17:43:11 INFO  HiveMetaStore:746 - 3: get_tables: db=default pat=*
2019-05-20 17:43:11 INFO  audit:371 - ugi=hadoop        ip=unknown-ip-addr      cmd=get_tables: db=default pat=*
2019-05-20 17:43:11 INFO  HiveMetaStore:746 - 3: get_multi_table : db=default tbls=
2019-05-20 17:43:11 INFO  audit:371 - ugi=hadoop        ip=unknown-ip-addr      cmd=get_multi_table : db=default tbls=

I'm not sure what caused the problem, it looks like some configuration error or others.

IVitamin-C commented 5 years ago

This is because hiveserver2 and spark-thriftserver are different when it comes to library table status updates, so you need an additional method to get the library and table status updates

TimKingNF commented 5 years ago

How can I resolve this problem? Thx !!!

IVitamin-C commented 5 years ago

The programmer who modified it is submitting it for official review.Please wait for the official code modification

TimKingNF commented 5 years ago

Okay! Thx for your answer.

IVitamin-C commented 5 years ago

Hue-8843 is available through code review, but currently only runs spark- SQL on spark-thriftserver.If you configure hive to run on the same port, there will still be problems, but the basic problem has been solved @TimKingNF