search

getavalon / docker

Avalon in a Dockerfile
MIT License
5 stars 2 forks source link

CGWire Synchronisation #23

Open mottosso opened 6 years ago

mottosso commented 6 years ago

Goal

Fast, accurate and bi-directional synchronisation between CGWire and Avalon.

At the moment, python a:\sync.py does one-way synchronisation from CGWire to Avalon with a 10 second interval. But it is flawed.

  1. It does not send data already in Avalon, or new data, to CGWire
  2. It does not have any knowledge about applications or path templates
  3. It synchronises via polling, which puts unnecessary stress on the CPU and disk unless there has been a change. Preferably we would only synchronise when a change has been made (via e.g. an event handler from CGWire).
mottosso commented 6 years ago

From https://github.com/getavalon/docker/pull/21#issuecomment-401173561

I was thinking the syncing daemon would be part of the docker container?

It could be.. but the problem is that it's a little inverted. See, CGWire is a container, so is Mongo. So we can either:

  1. Have a third container to synchronise the two containers
  2. Have CGWire synchronise itself with Mongo
  3. Have Mongo synchronise itself with CGWire

For (1), that's another container to manage for a simple task. For (2) and (3) it's not as clean as I'd like.

I expect lots more container to be part of this setup, so odds are we either bite this bullet, or bite the bullet of requiring a docker-compose template..

tokejepsen commented 6 years ago

Personally I don't mind going with option 2 and 3.

In the future with other trackers like Ftrack, the synchronization from tracker > Avalon will need to be custom anyways, so keeping it separate might actually be cleaner in the end.

Uncleanliness will come from Mongo > tracker, but could that not also be in the tracker container? That would keep all tracker based code inside the tracker container.

tokejepsen commented 6 years ago

I also think there should be an option to rescan for missing data.

I have seen a lot of times with event handlers, that events are dropped or missed somehow. This can happen in any number of ways, but frequently it happens due to connection issues.

What we could have is a rescan at startup and maybe at an hourly interval.

tokejepsen commented 6 years ago

The sync python script is getting larger and becoming unorganized. Even worse is that there are no tests to ensure consistency.

How about a syncing module in bin/pythonpath? Would be nice to start having a specific cgwire namespace as well, to anticipate other trackers.

mottosso commented 6 years ago

Yeah, might be time for a cgwire-sync repo even. Should I make one? Then we can add it to the git/ dir.

tokejepsen commented 6 years ago

Sure, that might be even better.

mottosso commented 6 years ago

https://github.com/getavalon/cgwire-sync

tokejepsen commented 6 years ago

Actually come to think about it, would it not be better to keep it in avalon-docker repository?

Testing already happens here, where all the components for CGWire <> Avalon syncing are accounted for. Also issue tracking would might be confusing as well.

mottosso commented 6 years ago

That would solve the immediate issue, but I expect we'd be delaying the inevitable; which is managing issues and features specific to CGWire.

For testing, what we've got going is positive in that it has access to all of Avalon to perform tests. But it's negative in that we haven't yet considered how to trigger these tests on updating any of the submodules. For example, I expect we'll want to update core, and test it during a PR, prior to merging. But using the strategy we've got here, we wouldn't be able to actually trigger these tests until after a merge, and after an update of the submodule here. (Unless we jump through hoops setting up parallel branches, cross-repository? :S)

How about we start the Docker container from the CGWire project? That way:

  1. We'd reap the benefits of this project for tests
  2. Lay the groundwork for testing in other projects, like Core and Launcher
  3. Encourage others to provide (and test) their own (independent, or in-house) sync projects
tokejepsen commented 6 years ago

How about we start the Docker container from the CGWire project?

Do you mean in the cgwire-sync repository, we use the container from here to test with?

tokejepsen commented 6 years ago

Looks like Zou has an event table we could use for syncing; https://github.com/cgwire/zou/pull/92

tokejepsen commented 6 years ago

Trying to make a start on it; https://github.com/getavalon/cgwire-sync/pull/1