Steps to split Fog into modules

[tokengeek]

It's been mentioned before #473 but splitting Fog into modules is back on the agenda since we discussed a rough roadmap for Fog in #1250.

Rather than rushing into possible ways it is probably worth listing exactly what people want to get out of this. Based on discussions so far I've got:

• Standardised service interfaces with canonical tests to check (#1252)
• Reduce burden on @geemus for Fog releases
• Allow providers to update their services without Wes being (unintentional) blocker
• Cleaner separation between core issues and provider issues
• Reduce dependencies for single provider when using Fog as library (no libxml for JSON API)
• Minimal memory usage when using single provider
• Reduce number of files when used on PaaS services like Heroku
• Cleaner way to version APIs
• Maintain community without fragmenting to provider ghettos (https://github.com/fog/fog/issues/473#issuecomment-1903823)
• Still simple for users to pick up and start using
• Reduce "churn" updating hardcoded references to new Regions or Images should not warrant a release of Fog but people should be able to update to use them (So provider point release)

So there's lots of technical gains we can achieve but we need to make sure we don't damage the community.

Next up is discussing possible ways to do this. Then we can get issues for the smaller parts.

Can we just break everything into gems? Do we have dependencies in fog-core we need to break first?

Is this Fog 2.0 so we can be backwards incompatible?

Threads, autoloading... etc.

[geemus]

In order to still have all the same benefits I think several meta-gems may also be needed. So fog would have all fog projects as dependencies, but maybe fog-aws could have just all the aws ones, fog-compute could have all the compute ones, etc. Otherwise some of the interoperability/simplicity may lose out (also this may provide for a better, backwards compatible experience).

Backwards compatible is preferable, if it becomes too difficult/time consuming we can certainly drop it, but I think it is worth trying to do initially.

My inclination has always been for the first step to be splitting out fog-core itself. That would be stuff in lib/core (collections/models/services) and probably some of the more general test setup stuff. I can tackle this and I think it should facilitate then moving other services out more one-by-one.

[tokengeek]

@geemus Do you think #929 (Circular references found) needs to be addressed to break the code out?

[geemus]

@tokengeek quite possibly (I'm not sure off hand). I sort of assumed that step one is to split out fog-core and figure out all the nitty gritty of that. Then pick a willing provider (maybe brightbox ;) and make it happen. Once we do those two the rest should at least have a clear path, albeit maybe still a decent chunk of work to accomplish. Hopefully if there are any issues such as the circular references they'll become clear as we work through things and can then be addressed.

[rubiojr]

@tokengeek, @geemus, folks:

Motivated by #1712 I decided to give a try create a standalone Fog provider. The goal was to implement it the way I would have done that if the provider was going to be merged into Fog:

• No Fog modifications allowed.
• Same testing infrastructure/workflow (shindo, mocks, etc).
• Same directory structure (lib/fog/, tests//{requests/models}/service, etc).

I've pushed the results of my work to https://github.com/rubiojr/fog-wunderlist.

Some notes I've collected during the process:

• I had to copy Shindo helpers from Fog, so perhaps it would be great to have some standard helpers (tests/helpers/*) built into Fog (Fog::Test::Helpers or something like that perhaps) to DRY things up a bit?
• I found no problems whatsoever implementing the provider and adding a new service type (Tasks).
• The only requirement for this provider right now is Fog itself (fog-core in the future?).
• Looks like it would be great to create a Fog provider bootstrapper a la 'bundle gem ' to automatically bootstrap provider structure and standard files and eliminate some of the manual boilerplate code creation.
• fog/bin discovery for this new provider is currently missing. I guess we could implement that on top of rubygems in 'fog-ng'.

I'd love to see this new Fog provider forked and added to the Fog organization and continue the work there, and perhaps experiment with a fog-ng branch cleaning all the providers and leaving core and trying to isolate some existing provider.

What do you guys think?

[tokengeek]

@rubiojr - Looks good.

The thing that immediately jumps out at me browsing the directory is that the new Task service in lib/fog/tasks.rb seems likely for a namespace clash across several providers.

I can imagine something like Fog::Firewall being added independently by people and then having two incompatible Firewall services. It's also something you don't want fog-core to be cluttered with lots of possible services.

Not sure if that can be solved by moving those services like lib/fog/compute.rb and lib/fog/image.rb to Fog::Core::* so we reduce the namespacing risk.

There's certainly something sensible about any new provider code being properly namespaced to begin with and providing a new service needs to be considered.

Service registration may need some work because we have a case statement in https://github.com/fog/fog/blob/master/lib/fog/compute.rb#L13 that we need to eliminate and ideally move a hook into lib/fog/#{provider}.rb that adds the entry.

So I think we might need a way that in lib/fog/wunderlist.rb it registers the presence of the provider and the service, supporting new services and adding providers to existing services.

Part of me is wondering if we ignore the new service aspect for now since the primary use case is cloud computing and Compute with multiple providers but your example hints that could clash easily.

Perhaps if we eliminate the case statements by registering a provider only when they are required then the code currently in all the lib/fog/#{service} files can go away.

I agree with your other points as well.

[rubiojr]

Awesome @tokengeek, thanks. Definitely some valid concerns IMHO.

I think that cluttering/overriding/monkey-patching Fog services and/or the Fog namespace is always going to be possible I guess and having an official, peer reviewed set of providers under the Fog organization will mitigate that perhaps.

Dynamic (new) service registration and/or better service registration code sounds great definitely. My main concern was to maintain backwards compatibility so perhaps the best way to kickstart this is to experiment with it in a new Fog branch. I created fog-wunderlist exactly for that reason: to have a very lightweight provider exposing some new services that we can use as a playground and everybody can easily test (since Wunderlist accounts are free). Being it so small, we could also use this to document the whole process of creating a new, 'compliant' provider I think.

[geemus]

@rubiojr -

• centralized test helpers makes good sense.
• yeah, ideally eventually it would be fog-core
• a provider-generator makes good sense, would certainly help with the initial bits that are otherwise copy-paste tedium
• bin/fog is tricky, not sure how best to address that really.

@tokengeek -

• yeah, maybe Fog::Core::TYPE is worth doing to avoid clashes...
• registration at require might actually be more sensible than current

@rubiojr - in terms of fog org, should we create a new thing as the de facto here or just fork and then you can mark yours as non-canonical? Certainly starting to get at least one (perhaps small/simple and not heavily used yet) provider moved to this new model seems like a great way to get the ball moving in the right direction.

[rubiojr]

@geemus yeah, let's create a new repo and start clean if it feels right. There's not much repo history to save.

[elight]

Ok so I've seen #1250 and #1253. But they're over a year old with nearly a year of inactivity in the best of cases. Is anyone else still interested in these issues? I'm 3 months new to working inside of fog (I joined @krames at Rackspace) but I strongly concur with @tokengeek's concerns.

I've chatted with @geemus about some of these concerns in other issues (see #1266).

Is anyone pushing this ball upfield? I'd like to help. But I don't want to step on anyone's toes...

[geemus]

@elight there is not (yet) ongoing effort here, at least that I'm aware of. I think perhaps that splitting out fog-core would be a sensible first step, but it seems to fall just outside of my radar/scope.

[tokengeek]

Thanks to @elight nudging we've pushed on with this and fog-core is now separate.

Since this is a big discussion I'm going to close it and we can break off smaller chunks based on the original list.

[geemus]

Sounds good, thanks!

On Wed, Feb 12, 2014 at 11:16 AM, Paul Thornthwaite < notifications@github.com> wrote:

Thanks to @elight https://github.com/elight nudging we've pushed on with this and fog-core is now separate.

Since this is a big discussion I'm going to close it and we can break off smaller chunks based on the original list.

Reply to this email directly or view it on GitHubhttps://github.com/fog/fog/issues/1253#issuecomment-34891810 .

[rubiojr]

:metal:

[ringods]

Hello! I've just read up on a number of tickets regarding the future of fog and it all seems nice. The modularization is what I'm needing right at this moment!

Working as a Release Engineer in a freelance position, I'm at the point where I have to support a custom compute API. With a number of tools already integrating with fog, this is my current plan on reducing a fair amount of work for my client. I am hoping that I can now implement a custom compute provider as a separate gem and get it registered for fog to pickup. Can some of you shed some light on this matter?

[krames]

@ringods Creating your own custom gem using fog should just be a matter of including fog-core and fog-json in your gemspec file. You can learn more about the collection and model patterns fog uses here.

Just out of curiosity, can you tell me any more about the custom compute provider?

[ringods]

@krames this is an internal API of a hosting company, that abstracts the different hypervisor technologies in use today. This compute API has the IP allocation, network configuration, software installation integrated on the technical side, and the customer billing on the business side. As can be expected, this is not something that will be OpenSourced. :-)

[krames]

@ringods Gotcha! Glad to see you are using open source though!!

[ringods]

@krames My intent is to link Vagrant to our internal API via Fog with a custom Vagrant plugin, taking vagrant-aws and vagrant-rackspace as examples. Both use Fog. :-)

[tokengeek]

Ultimately one of my use cases for standardising fog's compute service is to have us create vagrant-fog so that vagrant can manage anything supported by fog (when suitably configured).