search

trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
https://trino.io
Apache License 2.0
10.42k stars 3k forks source link

Document for add hive.metastore=file explanation in hive connector #5714

Open allendang001 opened 4 years ago

allendang001 commented 4 years ago

Currently, presto supports directly using files as the configuration of hive.metastore, but there is no introduction in the hive-connector document.

findepi commented 4 years ago

This is intentionally not documented, because file metastore is not a supported feature.

findepi commented 4 years ago

@allendang001 :

We have tested in the local file system and Tencent Cloud's cos, both are worked.

I know it works, we use it for testing purposes. In fact, the implementation is tailored for testing purposes too. We make no backward compatibility guarantees for the internal structures used by the FileHiveMetastore. In fact, we may need to introduce breaking changes to overcome existing limitations in that metastore. I would not want to cause breakage for the users. If we were to document the file metastore, it would need to come with all these considerations as warnings / disclaimers.

allendang001 commented 4 years ago

@allendang001 :

We have tested in the local file system and Tencent Cloud's cos, both are worked.

I know it works, we use it for testing purposes. In fact, the implementation is tailored for testing purposes too. We make no backward compatibility guarantees for the internal structures used by the FileHiveMetastore. In fact, we may need to introduce breaking changes to overcome existing limitations in that metastore. I would not want to cause breakage for the users. If we were to document the file metastore, it would need to come with all these considerations as warnings / disclaimers.

If it is the realization of the test purpose, we can introduce this usage, and then declare the follow-up support and disclaimer in the document. It just so happens that the function we want is similar to this one. It was only after reading the code that we realized that we can still play like this. Documented this can also help those who need it later understand this implementation and improve this feature.

XuQianJin-Stars commented 4 years ago

hi @allendang001 @findepi This function is very helpful for landing the results of DQL on the public cloud on cos. Because it does not need to create an entity table in the hive metastore, and output the results in various formats.

allendang001 commented 4 years ago

@findepi What should i do for merging this issue？ Change something in document or add something? The realization of file metastore is very useful for many people like us.

findepi commented 4 years ago

@electrum PTAL

XuQianJin-Stars commented 3 years ago

hi @electrum PTAL

electrum commented 3 years ago

I think it's fine if we document this. We might want to mention its limitations and say that it is mainly for testing purposes.

allendang001 commented 3 years ago

Now, could this issue be merged?