Open 51yu opened 5 years ago
You're correct, @l0yu. This is truly in-memory and there is no cleanup implemented. We should have something that syncs workers' state with coordinator serving as source of truth about tables.
You're correct, @l0yu. This is truly in-memory and there is no cleanup implemented. We should have something that syncs workers' state with coordinator serving as source of truth about tables.
@findepi ^^ reminds me of sync node state with serviceDescriptor/Announcement in discovery manager ... are we applying the same mechanism for syncing in-memory table states ?
@l0yu no, due to how connectors are plugged in, they don't talk to discovery manager.
@l0yu no, due to how connectors are plugged in, they don't talk to discovery manager.
@findepi seems like when connector manager create connector, node manager is injected as a member variable of connector context (but announcer is not). And in-memory connector just leverage that for enough worker nodes for create table pre-check (cluster has memory). Yeah, Maybe we have more general approach.
when coordinator restart, coordinate lose all previous in-memory fragment index info (i.e we could no longer read data though it still exists I guess) then how to clean up those fragment cached at nodes other than coordinator ? restart all would be painful... or, we persist meta info at least? then it's not pure in-memory ...