search

LockerProject / Locker

Locker - the "me" platform
http://lockerproject.org/
BSD 3-Clause "New" or "Revised" License
1.07k stars 123 forks source link

Collection data access APIs --> core #878

Closed smurthas closed 12 years ago

smurthas commented 12 years ago

The collections can now (optionally) expose an api.js file in their src dir and it will be required in by core and add it's endpoints (GET only right now). This is intended to enable collection processes to turn themselves off more often (only need to run for data processing) and (hopefully) enabled better process priority control (nice down processing, while leaving core at normal priority).

This also add Common/node/collectionDataStore, which abstracts several common data store patterns in collections. Next step is to make some of the API methods common, but I'd rather merge (and deploy) this first.

This passes all tests, but they're aren't really a ton of collection tests. I'd be happy to write more, but if we are moving away from vows it feels like a waste (tell me I'm wrong!). I did auth all connectors to ensure data processing works properly and hit all API endpoints to make sure they work.

smurthas commented 12 years ago

I need at least one person to test this:

  1. on master, clean me dir, connect all services
  2. once all data is in, hit /state on all collections, record the result (mostly count)
  3. repeat process from clean me dir on this branch
    • again, check /state, compare results -- should be the same as master!
    • check the logs closely for for any errors.
    • hit all collection API endpoints, listed below.

Contacts

/Me/contacts /Me/contacts/ /Me/contacts?limit=10&offset=25 /Me/contacts/state /Me/contacts/since?id= /Me/contacts/id/:someid /Me/contacts/update (and check logs again for errors!)

Photos

/Me/photos /Me/photos/ /Me/photos?limit=10&offset=25 /Me/photos/state /Me/photos/since?id= /Me/photos/id/:someid /Me/photo/update (and check logs again for errors!) /Me/photos/image/:someid /Me/photos/thumbnail/:someid

Places

/Me/places /Me/places/ /Me/places?limit=10&offset=25 /Me/places/state /Me/places/since?id= /Me/places/id/:someid /Me/places/update (and check logs again for errors!) /Me/places/from/:network/:from (eg /from/foursquare/someones4sqIdNum) /Me/places/from/geo/:network (eg /geo/foursquare) (and check logs again for errors!)

Links

/Me/links /Me/links/ /Me/links?limit=10&offset=25 /Me/links/state /Me/links/since?id= /Me/links/id/:someid /Me/links/encounters/:id /Me/links/update (and check logs again for errors!) /Me/links/search?q=singly /Me/links/embed?url=http://www.youtube.com/watch?v=1bJxYRtGUM4

quartzjer commented 12 years ago

Dude, kudos, this was quite the undertaking, moving a lot of logic around, thanks! It feels like a solid step towards some generalized collection functionality too, a pattern that could start to be applied to the data processing as well (I've had some ideas here, making this look more like synclets).

One caveat is this will increase the ram in the core process, and any bugs will bring the whole ship down, but the work @temas is doing with synclets using vm contexts as isolation might work perfectly here too.

Now to do some testing :)

quartzjer commented 12 years ago

I ran this with every app, and I also tested the memory impact on my mac between the two for lockerd node process. By including the apis in the main process it only bumped up RSS by 2MB, not bad.