search

NREL / api-umbrella

Open source API management platform
http://apiumbrella.io
MIT License
2k stars 324 forks source link

Create strategic partnership with other open-source API proxies #159

Open brylie opened 9 years ago

brylie commented 9 years ago

There are several emerging, open-source API proxy solutions. It may prove fruitful to share coding efforts, ideas, and lessons between projects.

Clyde IO

Clyde IO is just getting started. It is being built with Node.js and uses Connect middleware for filters.

Kong

Kong seems to be well conceived and is well documented. It is primarily implemented in Lua with some Python. It has a plugin architecture to provide common features such as authentication, logging, etc.

Tyk

The Tyk API Gateway has a similar ambition to API Umbrella. Tyk is primarily written in Go, with sprinkles of JavaScript and Shell. Tyk implements a middleware model, that may offer a common path to share code between projects.

Task

brylie commented 9 years ago

@lonelycode, I am looping you in on this issue in order to facilitate the connection :-)

lonelycode commented 9 years ago

@brylie Hi :-)

Big fan of what you guys are doing - need more OSS gateways in the ecosystem.

I think there's quite a lot of intersect, especially around the gatekeeper features. We pretty much cover most of those except the URL rewriting (though it's under consideration).

Our stack is pretty small, dependencies for a core Tyk install are just tyk and redis. If you want analytics and the dashboard you would need the tyk dashboard and MongoDB too.

Caching is handled by our gateway, using Redis as a storage backend. We thought about adding a layer into the stack using something else, but for the sake of minimalism it's embedded in the gateway.

We have a mini-service called the host manager that manages NginX configurations for multi-tenant, multi-api services (i.e. instead of just routing on endpoints, you can route on subdomain + api slug).

The dashboard also exposes a developer portal on a per-organisation basis, so multiple tenants can have their own portals on the same box. Portals are also taken into consideration by the host manager and are auto-registered accoridng to a template whenever a new organisation publishes an API.

We put together a SaaS verison of Tyk on https://cloud.tyk.io/, which is basically an orchestrated version of the Tyk stack.

I like the idea of trying to find some kind of coordination, it seems we have very very similar projects...

Keen to hear more! :-D

brylie commented 8 years ago

I have started a collaboration request thread in the Clyde IO issue queue. Antonio (@acanimal), I am looping you in on this thread to facilitate the conversation.

brylie commented 8 years ago

Nick (@GUI) would you mind responding here also?

acanimal commented 8 years ago

Hi everybody !!!

GUI commented 8 years ago

Thanks for making these connections, @brylie! And sorry for my belated response. It's definitely fantastic to see the growing open source options in this space.

@lonelycode: Likewise, I'm also a big fan of what you all are doing. The small, self-contained nature of having everything in a Go binary is certainly nice. I think it's also been a while since I've looked at Tyk, since the feature-set certainly seems to have grown a lot since I last took a peek (or I might have also missed various things the first time). A few random thoughts:

In any case, I obviously owe Tyk another spin sometime soon. But nice job with everything!

@acanimal: Very cool, I hadn't heard about ClydeIO before, but I'll definitely have to check it out.

On API Umbrella's end, one of the biggish changes on our horizon is a possible revamp of the underlying architecture into Lua/OpenResty (along with some other architecture simplification). Those details are mostly outlined here: https://github.com/NREL/api-umbrella/issues/86 (I definitely owe that issue a status update, but the short version is most of the work is complete, it's mainly a matter of testing and tweaking). Interestingly, the Lua fork of our codebase predated Kong's release, so I was pretty interested to hear about Kong being written in Lua when that came out.

There's definitely a lot of interesting stuff happening in the API management space these days! And while the architectures and approaches of these various open source projects seem to differ quite a bit, I'm certainly game to try and figure out how to collaborate more.

brylie commented 8 years ago

I started a collaboration request on the Kong issue queue.

njyx commented 8 years ago

Hi there - there's also APItools (http://www.apitools.com/) and the 3scale API Gateway - I'll flag this to the teams there.

harlow commented 8 years ago

Might be worth including the team from Mailgun (creators of Vulcand) in the discussion too?

acanimal commented 8 years ago

Kin Lane is an API evangelist and maintains a really interesting blog: http://apievangelist.com. I have contacted with him so he knows about this group and contribute with his experience.

ahmadnassri commented 8 years ago

hey guys, (representing Mashape / Kong)

happy to chat & collaborate.

ahmadnassri commented 8 years ago

(following up as I was on the tarin / mobile)

in regards to Kong and the rest of the Mashape family of products, our vision is simply this:

Build the best in class products, with a single focus, on solving a specific problem domain, operate light and efficient, with simple to use interfaces and APIs, while allowing for highly customizable output.

to that end, Kong is focused entirely and ONLY on the HTTP layer and API Management.

We've seen how monolithic products (open-source or otherwise) are built with an attempt to satisfy every itch, and end up in the scenario of "Jack of all trades, master of none".

not only is the code base and the technology and monstrosity to manage, but the Developer Experience suffers as a result.

our family of open-source products currently include:

with more to come... we see each one of these tools as playing key part in solving the API Management needs.

beyond Kong & Open Source, we also offer a standalone Analytics solution that can operate independantly from any API management gateway as well as integration with Kong (read about the API Log Format).

and we're partnered with Gelato for providing a developer portal solution for enterprise clients.

@GUI

Interestingly, the Lua fork of our codebase predated Kong's release, so I was pretty interested to hear about Kong being written in Lua when that came out.

funny enough, Kong is actually what been powering the Mashape API Marketplace for years (circa 2012), the release of Kong, is actually simply open-sourcing the core technology behind the marketplace (having de-coupled the proprietary parts).

In-fact, there was an open-source proxy with Lua & OpenResty released by us since the start! with the same core as what is now Kong, but was tightly coupled with the Marketplace usage of course, we've since removed that project with the launch of Kong ;)

we're also fans of Tyk and @lonelycode and have been in touch for a while :)

@acanimal we're flattered Kong inspired Clyde :) was the choice to build something new purely a language choice?

hello :wave: to everybody else whom I have not connected with personally just yet!

this is a good thread, and we're certainly happy to help make the entire community move forward and to the benefit of developers everywhere.

There might be a market fit for everybody, and certainly different products serve to solve different solutions, and while there might be an overlap in some cases, those would be certainly interesting to even turn into standards / common practices at least.

@GUI be interested to chat more about your Lua efforts, certainly there's bound to be an overlap, and as you probably already noted, the Lua community can be a bit slow :)

at the very least there's probably some libraries we can collaborate on.

kinlane commented 8 years ago

Thanks @acanimal -- definitely would like to see this move forward. Need to wrap my head around concept, before I contribute thoughts, however this is another major milestone in the maturity of the API management space - can't wait until API design catches up. ;-)

brylie commented 8 years ago

It seems like many of the API proxy projects have similar modules, but are written in different languages. E.g. rate limiting, request re-writing, load balancing, etc

What are some tools or formats that we can use to share common patterns for our API middleware?

kinlane commented 8 years ago

Couple of suggestions:

Common API interface definition - Take a play out of 3Scales book (https://support.3scale.net/reference/active-docs), and craft a common API definition for all features, and like Docker does (http://apievangelist.com/2014/12/20/swagger-20-for-the-docker-api/), make it a consistent layer of all implementations.

Import / Export In Common Formats - Beyond the interoperability of features and functionality, speak common API definition formats like Swagger and Blueprint when allowing users to import or export their configurations and setup, so that it can be shared between disparate systems.

Gathered some of my thoughts - http://apievangelist.com/2015/09/24/how-open-source-api-proxies-and-other-api-services-and-tooling-can-strategically-partner/

lonelycode commented 8 years ago

Can't agree more, we're working on a definition export in Swagger, and already support imports in both swagger and blueprint simply because that's what our clients speak. To be honest, what I'd love to see is an LLVM-like transpiler that will take format a and target formats b,c, and d.  Would make for a nice and efficient single tooling to get API definitions across services (had a chat with Runscope recently, third party API ecosystem tools like that need standards in order to inter operate with gateways).  It could even be exposed as a service! ;-) So far we've only seen wide swagger adoption, though RAML also comes out of the woodwork every so often. But we haven't had specific requests to support it yet. Would be up for setting down some standard file formats, the idea of a standard logging / analytics format also appeals.

ahmadnassri commented 8 years ago

I'm not convinced Swagger, RAML, others are suited for defining API Management level of abstraction.

they are by definition meant for documenting and describing APIs, not API Management rules, and although some parts might fit nicely, there are many more higher level operations to be considered that would be missed from such definitions and would be a "hack" to glue into Swagger / others.

it's true that Swagger has the biggest adoption, but that is not good enough of a reason to shoehorn / adapt it to every use case.

lonelycode commented 8 years ago

Ah, if we're talking about management then that's a different story - agree no definition format encapsulates the rules surrounding managing an API effectively. Ultimately that's got to be something that can be broken down into constituent micro formats so that it's possible to encapsulate the diversity of rules, modules, functions and controls users need when setting up a management interface. There's no way a single format would capture it, there's too many different things that need documenting. A suite of interlocking microformats though, that work within modules or on a resource-group and module/function basis might be worth investigating? Especially if their format can be agreed on so that they can be generated by new implementations for non existing features. Ideally whatever this is could also complement an existing widely used format as a superset, that way the existing ones can be annotated or upgraded easily without falling into the just-another-standard trap. Relevant xkcd: https://xkcd.com/927/ :-)

ahmadnassri commented 8 years ago

since you brought up xkcd, worth noting OASIS WS-DeathStar here ... lets not repeat the cycle ;)

kinlane commented 8 years ago

@ahmadnassri "I'm not convinced Swagger, RAML, others are suited for defining API Management level of abstraction." You feel that JSON, YAML, and Markdown can't define API Management? not sure I understand where limitations are? what makes API management so unique.

I used Swagger representations of API Account Management (http://theapistack.com/data/3scale/3scale-account-management-swagger.json), Analytics (http://theapistack.com/data/3scale/3scale-analytics-swagger.json), Billing (http://theapistack.com/data/3scale/3scale-billing-swagger.json), Service (http://theapistack.com/data/3scale/3scale-service-management-swagger.json) to help me orchestrate the deployment and management of APIs regularly. I deploy various versions of APIs into different virtualized scenarios using AWS or Docker, and while I use their API definitions to manage the lower layer, I use 3Scale to manage the service composition layer.

While I agree Swagger may not have everything for every scenario, I see people doing some pretty interesting things with Swagger extensions (http://swagger.apievangelist.com/extensions/index.html) when it comes to codegen, proxying, and virtually every part of API lifecycle. I see UCSB using for campus wide governance (http://apievangelist.com/2015/05/14/usage-of-swagger-for-the-apis-at-the-uc-santa-barbara-lab-for-research-on-adaptive-computing-environments/). Nomos developing a rules engine (http://nomos-software.com/blog/rulex-swagger-editor). Just to point out a few quick wins.

Sharing of common patterns for interfacing will be key too any API service provider, as well as API providers themselves. Swagger and the 14 other API definition formats I am tracking on (http://definitions.apievangelist.com/tools.html) are a bridge towards the world we should have. I'd love to see more media type and microformat standardization. It is happening, but these API definitions provide us with a important bridge to this better place. Something we are trying to connect using API definition format APIs.json, provide us a temporary index of your whole API operations whether you describe your APIs using Swagger, Blueprint, provide execution with Postman, or describe link relations using MSON.

I already track on the common building blocks for API definitions, design, deployment, management, monitoring, testing, performance, virtualization, monetization, evangelism, and discovery(http://apievangelist.com). I'm happy to help refine / translate them into a common list of features across the open source API management tools l track on (http://apievangelist.com/2014/10/05/taking-a-fresh-look-at-what-open-source-api-management-architecture-is-available/).

Then maintain a neutral list of features in HTML, JSON, YAML, or any other machine readable format makes this group happy. I suggest we do it on Github, so everyone can contribute, edit, and evolve as they see fit. Helping API proxy and / or management providers avoid the same mistakes being made by almost every other provider out there--the not sharing of common API patterns.

While this sharing won't ensure every platform is interoperable, it goes a lot further than just saying that won't work, and let's not repeat the mistakes of our past.

njyx commented 8 years ago

Responding to a point made further up about common API management interfaces, we (3scale) would be happy to pitch in our current API definitions as API Commons specs. That we haven't put them out there yet is a question of getting to it rather than protectionism.

They likely aren't prefect, but they are pretty comprehensive and have been tested by a lot of customers.

njyx commented 8 years ago

Speaking to the Swagger/RAML/etc. suitability to capture API management interfaces - we use Swagger but we did indeed have to make a few extensions. (examples: hooks for things like inserting keys on a per-user basis, or arrays/lists as return types). They were minor though and hopefully they'll end up in the Swagger specification on day.

ahmadnassri commented 8 years ago

@kinlane you're using swagger to describe the tool's own API, but I was referring to having a common format that can orchestrate the the API Management rules & setup, for example:

not every API Management solution listed here has an API, and even if they do, it doesn't have to all match up.

would rather having a common format to facilitate exporting / importing the APIs Management Configuration, security, consumers etc ...

ahmadnassri commented 8 years ago

as a side: its probably worth to have a separate discussion thread (with folks here) to discuss how we can improve on the API Documentation Formats we already have (Swagger and friends) taking some of the points @kinlane is referring to (extensions, completeness, etc ...) to the benefit of the API developer community as a whole, not just API Management providers.

kinlane commented 8 years ago

Great conversation folks. I guess I'm coming at it from a focus that API definitions can be used for d) all the above. I'm not solely focusing on Swagger for this, just to be clear, but JSON, YAML, and Markdown, and encouraging folks to define their own schema within, while also using what already exists.

Example of what I mean. I setup my link API in a docker container. My custom designed link API has a swagger describing its surface area. That link API lives in a docker container, which has a Swagger API describing its interface. My link API uses 3Scale for its API management, which has a Swagger specification describing it. My Link API uses API Science for monitoring, which as a Swagger specification.

I bind all four layers defined by Swagger together with a single APIs.json file, put in the root of my link API. I have four separate links to four separate Swagger specs for my one API. Giving me an interface to control all of it.

I also have a JSON file (which I call api-config), which gives me settings for /docker /3scale, /apiscience, and my own link API. Upon firing up of the container, the API knows to look for my APIs.json, learns what interface it has for its base container, management, operations and monitoring, then grabs a JSON to config each of them.

While Docker, 3Scale, and API Science all have a Swagger that describes their entire API, the Swagger I include for use as part of my custom link API only has the endpoints I need, and my api-config has an settings and pre-existing settings for these. ie. I only need user and app signup, and anlaytics for this API, I'm not doing any billing features--ok, I enjoy provide the existence of these endpoints, and configuration for these endpoints.

To add to the concept, I'm including an api-commons for my licensing of my API, and api-pricing to describe the service tiers, pricing, and rate limits imposed by this API instance.

The nuances between Postman, Blueprint, Swagger, APIs.json, RAML, WADL, RADL, MSON, and others are all very interesting, and the roll they play at different places in the lifecycle.

There is no reason we can't import one or a series of API definitions that setup an API, while also configuring the underlying operations of an API from server to proxy to firewall.

Direct responses (@ahmadnassri)

"not every API Management solution listed here has an API, and even if they do, it doesn't have to all match up." - I just can't imagine a tool or service in 2015 that services the API community, not having an API. Sorry. I just can't. Agreed 100%, they don't have to match, but what if they did? Or at least there was a JSON / YAML map of the translations between? AWS API Gateway is brining in mapping capabilities (http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html)

"would rather having a common format to facilitate exporting / importing the APIs Management Configuration, security, consumers etc ..." - I feel people don't seeing the nuances between existing definitions formats, how they work together, and the potential for defining more formats, that work together and do special things very well. The perception that there should be one format to rule them all like WSDL or WADL, or even a single media type is fantasy, there should be many formats, all working together, with more literacy about the nuances of each.

"as a side: its probably worth to have a separate discussion thread (with folks here) to discuss how we can improve on the API Documentation Formats we already have (Swagger and friends) taking some of the points @kinlane is referring to (extensions, completeness, etc ...) to the benefit of the API developer community as a whole, not just API Management providers". - YES! YES!

Direct responses (@njyx )

I'm sorry, I'm already using your API management definition (https://support.3scale.net/reference/active-docs) as a blueprint. You guys lead through example. ;-)

ahmadnassri commented 8 years ago

@kinlane

"not every API Management solution listed here has an API, and even if they do, it doesn't have to all match up."

I just can't imagine a tool or service in 2015 that services the API community, not having an API. Sorry. I just can't.

Yep, agreed, just highlighting that its possible that some currently don't, but shouldn't be excluded from a mutually beneficial standardization.

Agreed 100%, they don't have to match, but what if they did?

I think that's the whole point of the thread, to see what commonalities might exist and how to leverage them

Or at least there was a JSON / YAML map of the translations between?

is also what i'm referring to as a unified configuration format ... (beyond Swagger, just configs)

again, I'm specifically talking about the configurations of the API Management layers, a.k.a the values of what would be stored through APIs defined by Swagger ...

the most basic of use-cases: is export all your defined APIs / Users / Settings from one platform/tool and import into another seamlessly.

ahmadnassri commented 8 years ago

should we perhaps start a github org where we can open-up separate tickets for different topics and draft examples in separate repos? all in favor? :raised_hands:

kinlane commented 8 years ago

ok. didn't mean to muddy thread. Was just trying to help share that the the current API definitions movement is working on both the interface definitions as well as the settings (ie. my postman collection can contain the interface map, and what the keys are for specific parameters).

Suggesting a single map of common features across all existing API proxy / gateway / management, then a common API definition (maybe Swagger like 3Scale has), then a new specification for schema and passing values to this schema--something that could be imported / exported etc.

Will keep working on via my existing front, happy to share work anytime.

lonelycode commented 8 years ago

@njyx Are your API Definitions formats available anywhere? Would be interesting to see how you've encapsulated things :-)

@ahmadnassri Might be worth setting something up that's impartial - so :+1: on the idea of setting up a group for a common API Definitions format.

brylie commented 8 years ago

:raised_hand: aye.

acanimal commented 8 years ago

@ahmadnassri Thanks. Hope Clyde can see the light some day, although to be a "pocket" API gateway.

@kinlane on the line of a common interface for configurations (and probably following the joke pointed by @lonelycode :) ) I'm adding an API interface to Clyde so that its configuration can be more simple.

I working "actively" (when I have time) on the "dev-butler" and there is a document tries to summarise the main concepts: https://github.com/clydeio/clydeio/blob/dev-butler/BUTLER.md Please, it is not finished neither stable It is a bunch of ideas I'm evolving. Take a look it can serve as a new standard base hahahaha :smiling_imp:

From what I saw, kong has similar concepts (but not the same). I don't know the rest of systems. 3scale a much more complex management API.

+1 For a new group talking about all this.

ahmadnassri commented 8 years ago

started a group: https://github.com/api-standards & sent invites to everybody here.

lets continue the discussion there, gonna start with 2 repos (for issues -discussion- and prototyping):

(can rename repos of course and make more with better suggestion, just something to get the ball rolling)

ccsr commented 8 years ago

Hi @ahmadnassri please invite @ccsr and @kyyberi

ahmadnassri commented 8 years ago

@ccsr done.

meabed commented 8 years ago

the group is empty for almost a year now, so whats up?!

ahmadnassri commented 8 years ago

right around the same time this discussion kicked off, https://openapis.org came to existance some of the folks here who expressed interested ended up in that organization.

though in terms of gateway tooling / collaboration, that's still un-covered by Open APIs.

brylie commented 8 years ago

Right. Projects like API Umbrella, Clyde, Kong, and Tyk still seem to be working independently towards similar ends. It may simply be a matter of diversity and choice, but there does seem to be a large duplication of effort. One goal of a collaboration would be to standardize the gateway/proxy architecture and nomenclature, allowing independent implementations to follow similar guidelines.

ahmadnassri commented 8 years ago

@brylie agreed, and on behalf of the Mashape/Kong team, i'm certainly interested in fostering and collaborating on a standardization effort.

one hand can't clap though!

GUI commented 8 years ago

👏 Likewise, I'd also like to figure out better ways to collaborate from API Umbrella's end. For such a broad topic, though, I'm not exactly sure the best way to get started. Are there any specific and tangible outcomes we'd like to achieve? Or would some informal chats be the best way to get started? (I think there was a Gitter someone had created for this a while back, or we could do Google Hangouts, or there's also the api-standards org if there's more structured issues to discuss.)

ahmadnassri commented 8 years ago

@GUI indeed, I started both the gitter and the api-standards org, and @brylie takes all the credit for actually starting to summarize some ideas / topics there ...

perhaps an informal chat is needed to re-kickstart and setup some common goals/objectives to address.

if people here are interested, we're happy to physically host in our SF offices, and virtually for remote folks.

brylie commented 8 years ago

In honesty, I am better with async discussions. This is in part related to my location (timezone difference from the U.S.).

acanimal commented 8 years ago

Hi, I'm following the thread although, unfortunately, I can't participate much actively. My current job position is not direct related with development of api gateways so I will not be of much profit. Cheers.

brylie commented 8 years ago

I have created a repository in the API Standards organization called proxy. We can use that repository to sketch out standard patterns for proxy platforms.

brylie commented 8 years ago

@ahmadnassri may we use any of the design/outreach documents from the Kong project, to give an overview of common API proxy components? I want to make sure it is alright to redistribute those assets. I added the ISC License to the proxy repository, but we can change it to CC Zero or some other license.

ahmadnassri commented 8 years ago

@brylie any of the content on getkong.org is distributed under MIT license: https://github.com/mashape/getkong.org