search

vkuznet / transfer2go

Distributed, loosely couple agent-based transferring system
MIT License
8 stars 2 forks source link

Implement central catalog #16

Open vkuznet opened 7 years ago

vkuznet commented 7 years ago

PhEDEx uses central catalog, while transfer2go does not. We need feasibility studies if we can use distributed model at scale. E.g., what will happen if we need to update catalog, corrupted catalog at one site, etc.

rishiloyola commented 7 years ago

I guess we need to test this thing to implement push and pull model. Can you tell me more about this test?

vkuznet commented 7 years ago

Since TFC (trivial file catalog) now resides in each agent we need to test what will happen in agent will go down or its catalog get corrupted. There are few ways to avoid this problem. We may follow "amazon" approach for distributed store where we keep a copy of catalog elsewhere, e.g. each agent keep a catalog of its neighbors. Or, we may still sync agent's catalog to central one.

But in any case this will be second priority until you implement push model and we perform integration tests.

On 0, Rishi notifications@github.com wrote:

I guess we need to test this thing to implement push and pull model. Can you tell me more about this test?

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/vkuznet/transfer2go/issues/16#issuecomment-313196803

vkuznet commented 7 years ago

The main-agent should be able to answer questions about participating agents. For that we need to have central catalog. The main-agent will periodically collect data from agents and merge them into central catalog. The system should work similar to git model where there is a central repository and forked ones. The forked repositories (agent's catalog) will keep current status, while central repository (main-agent) will keep info from all of them