search

incraigulous / contentful-sdk

A modern PHP SDK for Contentful delivery and management APIs.
MIT License
14 stars 11 forks source link

Build out Synchronization support #5

Closed janpio closed 7 years ago

janpio commented 9 years ago

Hey,

I was really happy when I found this new project via the Contentful twitter account as I have a really bad hacked together implementation in production right now and was hoping to replace it using this library.

Looking at it again it seems to be complete enough for me to actually try, but now I see you don't plan to implement the sync endpoint - I use it to get a complete "download" of my (quite big) spaces to start and later keep it updated.

You write that caching is a better solution for this problem in PHP. Can you elaborate a bit how one should start to get all the initial data down to build a representation of what is already in Contentful?

Thanks, Jan

PS: Are you generally not interested in having a sync implementation or are you open for someone else to submit a PR of it?

incraigulous commented 9 years ago

Hi Jan,

I'm glad you posted this. Honestly, I didn't implement synchronization because I'm not familiar enough with it. Not just Contentful's implementation, but even as a concept. Caching queries on demand seems much more simple. Especially since most PHP devs already cache query results (from DB calls and web services). I assumed that synchronization was geared mostly toward client-side development like iOS or Javascript. Obviously, if I'm already hearing otherwise and the package has only been out for a week, I'm probably wrong.

I would be more than happy to accept a pull request to add this in. I don't think I'm the one to build it, but I would love for someone else to. Are you up to the task? I would be happy to assign it to you, or I can mark the issue as "help wanted".

I am curious, can you give me a bit more info on how you're using it? Do you just store the data in NoSQL or memcached or maybe flat files? Do you query for updates on a cron? I'm considering softening my language on synchronization in the readme. Before I do, I would really like to understand how one would implement it a bit better.

janpio commented 9 years ago

Hey Craig.

Sure: We have a tool that fills a relational database with all the stuff we have in Contentful [1]. So Contentful is not our database, as you do when you use the normal entries and assets endpoints, but an external data store that we synchronize with. [2]

We use webhooks from Contentful to get changes sent to this tool, but as they can get lost and you almost never start with an empty space, we need a way to get the updates since a specific date or since forever - and sync is perfect for that.

In a sync you get a list of all content_types, entries, assets and deletions (new concept!), and they include all locales, so the structure of the objects is a bit different. There are also some parameters, but they are not too important and easy to figure out.

Most important are sync tokens and the paging: Your first request is with no sync token, and you get everything that is in the space right now. If it doesn't fit on one page, you have to request a second (third, forth...) one. At the end, you get a sync token you save.

Next time you already have this token, and the sync request only returns the changed objects since your last request. (Note: not the changes, but the updates objects). Again paging, again new sync token.

And that's it. (Solves the problem of when to invalidate your cache, too, by the way...)

I'ld love to help, but my PHP is PHP4 pre classes, so I'm not sure I would be a great help. Biggest problem would also be the first: Where to put this. Right now everything is neatly OO, but sync delivers all kinds of different objects in a slightly different format than what you built already.

[1] and the other way around - here I could already use this library. [2] We also do the exact same thing on mobile, but for different reasons: There our app should work offline, so we have no other way than to download all the data - and use sync for it

incraigulous commented 9 years ago

That sounds like a cool system! I love the idea behind it, being able to use MySQL with Contentful content would be the best of both worlds! Just from looking at the API docs, I think it could be accomplished with a sync resource class (along the lines of the other resource classes). It's not too far off the current structure I think, but I don't know for sure without digging in to it. I'll go ahead and mark this as help wanted and hopefully someone will step up. You would just need some custom methods in the resource class for setting the type, etc.

If anyone wants to take this, let me know! I would be happy to get you started or field any questions.