OpenStack Swift / CloudFiles support

[mikermcneil]

Adding this for reference-- thinking about tackling this one myself since we have a working implementation we built for another project. If you guys have a certain way you'd like to see this done, please let me know!

Love the philosophy and purpose of the project. Thanks for all your hard work!

[ryan-roemer]

Hi @mikermcneil

And OpenStack / CloudFiles provider would be great. The model I'd use is what has been done for the AWS provider.

The biggest issue I see with OS/CF (at least as I remember it last time I worked with the raw REST API) is that getting a connection requires a REST call / network event, and the current Sunny configuration class assumes a connection can be synchronously retrieved. (This is because AWS / GSFD in AWS legacy mode use "one-shot" signed requests). OS/CF and even APIs using OAuth will need async for this, and this means breaking the existing Sunny API in at least that place.

That said, I'm totally fine with the API changing to support OS/CF (in of course the minimal fashion needed), and I used to be pretty familiar with Rackspace Cloud Files (not sure how different swift is these days), so I can help out with high level guidance, particularly if you have questions of what parts of the Sunny API should be broken, extended, etc.

Looking forward to branches / pull requests!

Best, Ryan

[mikermcneil]

Ryan-- I'll run this by the stakeholders involved to be sure it's ok we're open-sourcing this, and then I'll work on getting our OS/CF adapter integrated in a early pass way. Then maybe you can take a look and we can work on what a 2.0 api would like like.

Thanks for the advice! Mike

[rupakg]

@ryan-roemer, I had started discussions with you for implementing HP Cloud Services which are based on Openstack a while back, but I got busy. I maintain the HP Cloud fog (the Ruby multi-cloud provider library) provider as well. I think we had talked about the core changes but never got to it. I will get back on this soon. Good to see some more interest develop since then.

[ryan-roemer]

Hi @rupakg -- good to hear from you again. I think it might be a great opportunity for you and @mikermcneil to collaborate on geting out an OS/CF provider (assuming both of you have available time and support). And, again, I can help out around the edges with API design and things like getting all the unit tests to pass when we hook in the new providers...

[mikermcneil]

@rupakg Nice to meet you!

Talked to the interested parties and sure enough, we're good to go. I'll commit what I have this evening.

[mikermcneil]

Here's the fork: https://github.com/balderdashy/node-sunny

I set up npm test to run the existing tests. Is there an example file for the configuration to make live tests work? I couldn't find it, but I didn't look too hard.

I also dumped the current OpenStack adapter implementation we're using (SwiftAPIService.js) in a new provider dir (lib/provider/openstack)

Currently, download is using a stream, but upload is using a blob of binary. The reason for this is that Express provides access to uploaded files in req.files (on the disk).

[ryan-roemer]

There's a full development guide that shows how to set up secrets for running the live tests. Also, keep an eye towards jshint on the code (more for the eventual pull request, we'll want that to pass). There's also an error in package.json that has a dep. for jslint instead of jshint. I'll fix this soon.

I haven't checked the abstraction straight up, but the goal for the Blob class is a fully-compliant stream both ways according to the existing API (although extra convenience methods are fine). Put another way, a Blob from any provider should be able to be pipe'ed either way -- read or write.

Thanks for the start, and I'll try and look over the changes when I get more time to help with pointers.

Update: Perhaps put more helpfully, I'd like the new provider to fully support the existing abstract interfaces (the methods and events):

• http://sunnyjs.org/api/symbols/base.Connection.html
• http://sunnyjs.org/api/symbols/base.blob.Container.html
• http://sunnyjs.org/api/symbols/base.blob.Blob.html

And, I suspect we will need to break / augment the following abstract interface:

• http://sunnyjs.org/api/symbols/Configuration.html

Apologies for the rather stream-of-conscious series of commentary -- it's been a while since I fully reviewed the API, and this is the first new provider in a while. Hope everything I'm thinking makes sense and seems reasonable, but please let me know if you're thinking differently.

Thanks!

[ryan-roemer]

And thanks for starting work on the provider!

[ryan-roemer]

Another comment is that I'd like to watch the addition of extra dependencies -- Sunny only has one real dep. right now, Xml2JS, using standard lib HTTP for all communication. For requests as an example, there is the sunny wrapper library (https://github.com/ryan-roemer/node-sunny/blob/master/lib/request.js) that has NOP requests, and AWS-specific authenticated requests, etc.

The branch has two underscore deps and a separate request library -- which so far that looks OK, as I use underscore extensively in other projects, and I imagine that the OS/CF requests are going to be a lot different (especially wrt authentication) which may favor having a dedicated and better library than raw Node http.

But, I've taken pains to keep the deps really slim on Sunny, and I'd like to see that continue as much as possible.

[mikermcneil]

Makes a lot of sense-- I appreciate that. request has sort of become the de facto standard for simple stream manipulation and HTTP requests IMO, and _ is pretty light. How about we start by getting it working, and then we can axe deps if necessary? I won't do a pull request until it all looks good.

I'll be messing with it a bit more, mainly focused on using duplex streams-- @techpines is working on better file upload middleware for our purposes, so we'll actually be able to take advantage of that for our project.

[ryan-roemer]

That sounds like a good plan. If request works out well, I'll look into to backporting it into the base Sunny request wrapper for consistency.

Feel free to ask for periodic feedback along the way before the pull request(s) if you'd like, and in particular any issues with getting all the live tests to pass against a real OS/CF service (for example even the existing Google Storage live tests are finicky and where kind of pain to get into place when I originally wrote them). I think I do still have a Rackspace CF account around, so I can run your fork's tests if that helps.

[mikermcneil]

Lets both- we have a couple of customer OS instances ourselves. The more tests against different instances the better since there are so many ways to configure these things. For instance, we're running into a bug that locks down all IO when you try to play an mp3 in the browser, but we think its specific to a devstack instance

Mike's phone

On Jan 16, 2013, at 14:56, Ryan Roemer notifications@github.com wrote:

That sounds like a good plan. If request works out well, I'll look into to backporting it into the base Sunny request wrapper for consistency.

Feel free to ask for periodic feedback along the way before the pull request(s) if you'd like, and in particular any issues with getting all the live tests to pass against a real OS/CF service (for example even the existing Google Storage live tests are finicky and where kind of pain to get into place when I originally wrote them). I think I do still have a Rackspace CF account around, so I can run your fork's tests if that helps.

— Reply to this email directly or view it on GitHub.