CGWire / Kitsu Integration

BigRoy commented 4 years ago

This is about a "connection" integration with CGWire / Kitsu. The nice thing about CG-Wire and Kitsu is that, like Avalon, it is open source and you can run it completely free.

Implementation references

Some prototype work was done on connecting Kitsu with Avalon in the past:

Example API for linking Avalon with CGWire / Kitsu
Issue on getavalon/docker on CG-Wire syncing which resulted in this syncing code and some details discussed in its PR
For Qt interfaces for CG-Wire/Kitsu I set up the qtazu repository some time ago. Having Qt interfaces ready for Kitsu would be helpful for exposing the connection with Kitsu to artists right in the host applications where Avalon is integrated.
There is a cgwire-sync repository however this currently seems completely unused.

This is NOT a host integration

This is not a host integration like Maya, Houdini, Fusion Instead this would be a "connection" to CG-Wire. Similar to #280 is for Ftrack. The code doesn't necessarily need to end up in Avalon core. But I'm creating this issue to track the status of any ongoing work on integrations and eventually allow others to work together on getting Avalon + CG-Wire working well togeher.

References

There's also a dedicated #avalon channel in the CGWire Slack but of course the Avalon Gitter lobby is perfectly suited to chat about it too.
Avalon Issues #280: Ftrack Integration

frankrousseau commented 4 years ago

I coded sync functions to synchronize two Kitsu instances. The code written for this may be helpful if you want to develop synchronization for Avalon.

https://github.com/cgwire/zou/blob/master/zou/app/services/sync_service.py

tokejepsen commented 4 years ago

What might be good is to get an outline of goals for this "connection".

For example when working with Ftrack/Pype, they have integration plugins so any publish is immediate and there is no waiting for a "sync".

cameronpd commented 4 years ago

I just saw this issue today, so I've just finished off the final outstanding task for my kitsu -> avalon sync utils, but after coming back to it after not touching it for 5+ months I've concluded that it's probably not very much use to most people.

The implementation is flawed due to my lack of understanding about the Avalon db scheme and what could be stored in it when I started the project.

It was also my first python project and the first programming I’ve done in 15+ years other than a few simple pyblish test, but if anyone wants to poke at it and/or laugh/cry at it, I can put it up somewhere.

tokejepsen commented 4 years ago

I can put it up somewhere.

Would always be useful to see and maybe learn from you mistakes :)

I've started looking at this as well (sparetime though so slow progress), and formulated a general plan of action:

Full sync from MongoDB to CGWire and vice versa. This is to clarify how the mapping works between the two databases. This will also help if the communication between the databases goes down.
On demand syncing. Instead of running a full sync every n time, we want to update the databases only when needed. CGWire has an event framework. Avalon could have a register_sync where we forward the write requests in io.py to all registered sync modules which handles the logic of mapping to CGWire.

tokejepsen commented 4 years ago

Forgot one:

I would also suggest that we register the write events in an event log in MongoDB. Any syncing modules could record which event they have synced, so if the communication goes down, the syncing module could resume from where it ended.

cameronpd commented 4 years ago

Ok, I've cloned it to github. We ran it for several months without issue (I'm in the process of setting it up again on new infrastructure after moving offices). But as noted at the top of the readme, has some major limitations. Which were fine for us as we were just starting out, but not much use for anyone with an existing setup.

Bidirectional syncing would of course be great, but was a lot more complication than I wanted to take on or had time to deal with and besides having to create assets/projects only in Kitsu wasn't much of an negative for us, it's interfaces is arguably better/easier to use than Avalons.

cameronpd commented 4 years ago

Having spent some more time thinking about it over the weekend, I can't currently see how it's possible to do two way or even one way syncing without either Avalon or Kitsu needing to have a clean db/install, as you'd have no way of knowing how to link the assets in one to the other (the only possible exception would be if asset names etc exactly matches up between the two).

In that regard, it at least makes my Avalon sync project not so flawed. Still has the flaw of storing all the syncing data in an external data folder rather than in the data fields of Kitsu and Avalon though.

tokejepsen commented 4 years ago

I'm currently working through Avalon to CGWire where I'm storing the ID of the Avalon entity in CGWire on the data JSON member. If I find this ID, I match by that, else it's by the name.

tokejepsen commented 4 years ago

For anyone interested my work-in-progress for syncing is here; https://github.com/tokejepsen/docker/tree/cgwire_sync

Currently writing tests so experiment with the code afterwards.

tokejepsen commented 4 years ago

Currently writing tests so experiment with the code afterwards.

Should also warn people not to run the tests on a production database since it rinses all data.

getavalon / core