search

trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
https://trino.io
Apache License 2.0
10.42k stars 3k forks source link

Load catalogs concurrently #15234

Closed BOFA1ex closed 1 year ago

BOFA1ex commented 1 year ago

It took a long time to startup trino when catalog files were too many.

As I know, load plugins have been supported concurrently already. see https://github.com/trinodb/trino/pull/9521

so why does ConnectManager#createCatalog modify with sync? or does it have another way to improve catalog initialization?

hashhar commented 1 year ago

cc: @dain since you're working on dynamic catalogs already

findepi commented 1 year ago

As I know, load plugins have been supported concurrently already. see #9521

When I was working on #9521 the whole catalog creation was synchronized https://github.com/trinodb/trino/blob/4b2565a98d622a5419263ba40fd0ae12ef077888/core/trino-main/src/main/java/io/trino/connector/ConnectorManager.java#L215, so it didn't make sense to run it in multiple threads.

This is no longer the case though, so -- https://github.com/trinodb/trino/pull/15295

BOFA1ex commented 1 year ago

As I know, load plugins have been supported concurrently already. see #9521

When I was working on #9521 the whole catalog creation was synchronized

https://github.com/trinodb/trino/blob/4b2565a98d622a5419263ba40fd0ae12ef077888/core/trino-main/src/main/java/io/trino/connector/ConnectorManager.java#L215

, so it didn't make sense to run it in multiple threads. This is no longer the case though, so -- #15295

But in tag:389, it's still an issue. Does it have another way to solve the issue without upgrading the version? (e.g. erasing the synchronized keyword from createCatalog)