search

trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
https://trino.io
Apache License 2.0
10.39k stars 2.98k forks source link

[In-memory connector] non-persist metadata #1452

Open 51yu opened 5 years ago

51yu commented 5 years ago

when coordinator restart, coordinate lose all previous in-memory fragment index info (i.e we could no longer read data though it still exists I guess) then how to clean up those fragment cached at nodes other than coordinator ? restart all would be painful... or, we persist meta info at least? then it's not pure in-memory ...

findepi commented 5 years ago

You're correct, @l0yu. This is truly in-memory and there is no cleanup implemented. We should have something that syncs workers' state with coordinator serving as source of truth about tables.

51yu commented 5 years ago

You're correct, @l0yu. This is truly in-memory and there is no cleanup implemented. We should have something that syncs workers' state with coordinator serving as source of truth about tables.

@findepi ^^ reminds me of sync node state with serviceDescriptor/Announcement in discovery manager ... are we applying the same mechanism for syncing in-memory table states ?

findepi commented 5 years ago

@l0yu no, due to how connectors are plugged in, they don't talk to discovery manager.

51yu commented 5 years ago

@l0yu no, due to how connectors are plugged in, they don't talk to discovery manager.

@findepi seems like when connector manager create connector, node manager is injected as a member variable of connector context (but announcer is not). And in-memory connector just leverage that for enough worker nodes for create table pre-check (cluster has memory). Yeah, Maybe we have more general approach.