Define APInf platform API architecture

[kyyberi]

We have now defined MVP design for Catalog REST API. We need to define the larger picture of platform APIs. Are we adding API management methods to Catalog API? Are we adding statistics methods to Catalog API? Or do we have for example Catalog API, Management API and Statistics API?

Added 2017-04-18:

User need

• Main user need: integrating with APInf through the APIs instead of interacting manually with Ui
• Catalog API user needs: e.g. harvesting APIs, adding any new customer APIs automatically to APInf catalog
• Data flow: Use a REST API to follow the traffic, state, changes about the customer APIs (any changing data related to APIs), it will be used for customers' tracking tools, own dashboards. (Open issue: Should this be also MQTT in addition to REST?)
• Organizations: adding / modifying / deleting organizations
• Users: adding / modifying / deleting users and their privileges
• Proxies: adding / modifying / deleting proxies, configurations
• Connecting APIs to proxies

Front-end should also use the APIs and it should be possible to replace any platform technology without the API users noticing a difference.

[kyyberi]

Then we already have API for managing MQTT users afaik.

[bajiat]

The API for MQTT users is not an APInf API. You could compare it with the API Umbrella admin API. We would like to integrate with proxies through (admin) APIs.

[kyyberi]

Ok. Still we should have api architecture defined. Otherwise we build spagetti apis.

[brylie]

I have taken this task. For the first iteration/goal, lets aim for 100% coverage of our data model with REST creat, read, update, delete equivalent endpoints.

We can build on that experience to make more 'custom fitting' endpoints in a second iteration.

[brylie]

After a brief meeting with @ashakunt, @matleppa, and @mauriciovieira we have reached the following consensus:

Recommendation

Itt is recommended that we build out our 'v1' API using Simple REST. This is for reasons, including:

• low maintenance overhead
• it is based on existing code (methods, publications, and collection definitions)
• REST endpoints are defined nearly automatically, sharing functionality such as permissions by default
• giving us relatively quick coverage of entire data model

Caveat

We want to maintain the 'self-documenting' aspect of our API, by providing an OpenAPI Specification compliant API description file. For this, we may need to port our existing Restivus Swagger module to support Simple REST.

Scope of work

Data flow: Use a REST API to follow the traffic, state, changes about the customer APIs (any changing data related to APIs), it will be used for customers' tracking tools, own dashboards. (Open issue: Should this be also MQTT in addition to REST?)

Regarding point two in the task description, there is a lot of clarification needed about the scope of work. Intuitively, the work seems 'epic' level, so might need to be refined to sprint-sized tasks. We made the following observations:

• some of the wording in description implies functionality that doesn't yet exist (e.g. 'changes about the customer APIs')
• some of the items in the description are provided by integrated systems (e.g. traffic analytics are recorded by both MQTT broker and API proxy)
  • in the case of providing wrapper APIs for integrated systems, we need to clarify the business case(s)
  • integrated systems (such as API Umbrella) already provide APIs, so we may be duplicating existing work

@ashakunt, @matleppa, and @mauriciovieira, please provide any further thoughts and clarifications that I may have omitted.

[bajiat]

A few questions from my point of view:

• How would this proposal go with the already designed Catalog API, which is the first priority to be implemented?
• I can understand the point of needing more clarification for customer needs outside the Catalog API, but is that a reason for going with our-system-first approach? Doing what is easy for us and based on our technology?

[brylie]

How would this proposal go with the already designed Catalog API, which is the first priority to be implemented?

The proposal is in alignment with the current APInf Catalog API design, and would likely support the following needs:

• APIs
  • Get all APIs
  • Create API
  • Get API by ID (authenticated)
  • Update API by ID (authenticated)
  • Delete API by ID (authenticated)
• Organizations
  • Get all organizations
  • Search organization by field(s)
  • Add organization (authenticated)
  • Update organization (authenticated)
  • Delete organization (authenticated)
• Users - We need to double check how the proposal for Simple REST would support Meteor Accounts
  • List and search users
  • Add user (authenticated)
  • Update user by ID (authenticated)
  • Delete user by ID (authenticated)

I can understand the point of needing more clarification for customer needs outside the Catalog API, but is that a reason for going with our-system-first approach?

Clarifying customer and business needs is central to a system-first-approach. In principle, customer and business needs are the system - software is a means to an end.

It is also important to identify the boundaries of our software system, for this task and in general. Of specific concern is that our software system encompasses multiple third party systems, with their own development lifecycles and APIs. We determined that it would likely require extensive work and maintenance (beyond one or two sprints) to develop and maintain wrapper APIs for the third party systems. Hence, our initial recommendation is to build/maintain a full REST API based on our internal data structures.

Doing what is easy for us and based on our technology?

Any proposal will, by necessity, be based in our current and forthcoming technology. To that end, the Architecture Team looked at our current system model, data model, and the options available (namely Swagger Codegen, Restivus, and Simple REST) for building the Apinf REST API. Our proposal is about more than just ease and considers maintenance as a cost while trying to avoid waste.

Again, the current proposal is in alignment with this task, the provided design file, and the currently identified customer needs. Of course, we will revise and refine, and possibly even reject, the proposal through active and open dialogue.

cc: @ashakunt @kyyberi @matleppa @mauriciovieira

[brylie]

It is worth mentioning/reiterating two likely challenges related to the Simple REST proposal:

• we may need to find a strategy to provide version stability for the APInf API
  • there is a related discussion in the Simple REST project
  • providing version stability in our API endpoints is a common challenge, not necessarily unique to the Simple REST proposal
• we may need to extend Simple REST to provide a valid OpenAPI Specification description of the APInf API

[kyyberi]

Sounds good to me as long the approach is customer needs first. I already gave slack in design not requiring API economy standard authentication (OAuth2).

I assume that the selected path enables implementation to have designed parameters in the methods as well? Those are fundamental part of the agility of the APIs.

Regarding duplicate work considering Umbrella. We have to keep in mind that Umbrella might not be there for long.

For the record I can state based on the above comments that our own team does not seem to believe or see any value in design first approach. We might as well stop talking about it.

Furthermore, the topic is "Define APInf platform API architecture". I haven't seen any discussion yet about platform API architecture. I have seen good discussion about one API and how to implement it.

[brylie]

I haven't seen any discussion yet about platform API architecture.

Good point. I can understand that it might not seem obvious how the discussion Tuesday reflects on the API Architecture. I am currently drafting a diagram, based on the proposal, that might help clarify how implementation choices might relate to the emergent architecture.

[brylie]

I assume that the selected path enables implementation to have designed parameters in the methods as well?

Yep, that is important. I just double checked, and we can pass in parameters (such as Organization ID) via the URL or even in the request body.

[brylie]

I can state based on the above comments that our own team does not seem to believe or see any value in design first approach.

Is there a specific comment that indicates anyone on our team is rejecting a design-first approach? The only thing we have recommended relates to how we would build the API (which affects architecture), regardless of how it is designed.

[brylie]

We have to keep in mind that Umbrella might not be there for long.

Great point. We will need to consider how to abstract our proxy/broker layer, for example, so that users have a consistent API regardless of the technology.

One difficulty we may face is that, even in the 'API Proxy' realm, there are different solutions that might not have feature parity or share scope with one another. That would likely affect any abstraction we provide through the APInf API.

[brylie]

I already gave slack in design not requiring API economy standard authentication (OAuth2).

This is an important request that has been raised by several sources.

There is a Simple REST BearerToken plugin that, by integrating with our APInf Accounts system, might provide an OAuth2 authentication path. I will have to check with the other @apinf/developers to make sure I understand this correctly.

[brylie]

Here is a depiction of the proposed APInf API Architecture, from a high level:

[bajiat]

Comment about the architecture diagram: should we have one API or actually multiple APIs? One of these APIs would be the Catalog API. Or does adopting Simple REST mandate that there is one single API?

[brylie]

Basically, since we have one APInf Platform I assume we have 'one API'. The APInf Platform API (under consideration) can encompass multiple sub-sections, such as /organizations, /users,and /apis. The code can also be split into multiple modules, since it would use the same aspects (collections, publications, and methods) that are in use throughout the platform.

Would you clarify what you mean by 'multiple APIs'? Do you mean we should be maintaining separate codebases and designs for each API section?

[frenchbread]

In addition to the endpoints above, I'd suggest to extend it to by adding either Get all PIs with URL params for flexible filtering or sorting, or Search APIs. The first option is a general purpose and can be used for search as well.

[brylie]

What parameters might the Search APIs endpoint take? E.g. what fields would be searchable and how would they be specified. Lets do a 'design first' approach with the search endpoint(s), in a similar way to what we are doing with the Catalog endpoints.

[kyyberi]

To me it looks like you guys haven't even looked at the proposed design based on the comments above. https://apinf.io/apis/TYf33t5B6D6fwamKt

There is first GET method called /apis with params. The same method is "search", not just listing APIs

Second thing. Putting all methods in one API makes it more difficult for external developer to onboard. The learning curve will be steeper because of share amount of methods and options in the API. Thus Taija and I suggested to have separate clearly defined and easy to understand swissknife APIs instead of one bloated generic API.

[brylie]

it looks like you guys haven't even looked at the proposed design based on the comments above

Which comment(s)?

Putting all methods in one API makes it more difficult for external developer. Taija and I suggested to have separate clearly defined and easy to understand swissknife APIs instead of one bloated generic API.

What do you mean by 'one API'?

We have simply proposed a way to build any number of API endpoints, or sub-sections, giving us a lot of flexibility with the design.

[brylie]

To reiterate, the proposal is in alignment with the current APInf Catalog API design, and would likely support the following needs:

• APIs
  • Get all APIs
  • Create API
  • Get API by ID (authenticated)
  • Update API by ID (authenticated)
  • Delete API by ID (authenticated)
• Organizations
  • Get all organizations
  • Search organization by field(s)
  • Add organization (authenticated)
  • Update organization (authenticated)
  • Delete organization (authenticated)
• Users - We need to double check how the proposal for Simple REST would support Meteor Accounts
  • List and search users
  • Add user (authenticated)
  • Update user by ID (authenticated)
  • Delete user by ID (authenticated)

We can do a 'design-first' process for other specific APIs (meaning 'endpoints'), to suit other customer and developer needs. We are not standing in the way of any of those proposals.

[brylie]

To me it looks like you guys haven't even looked at the proposed design There is first GET method called /apis with params. The same method is "search", not just listing APIs

The current architecture proposal can support that design.

I was just asking @frenchbread for his thoughts on the API design, since he is in the office this week. He has not been part of previous discussions or the review process up to this point.

Please stop making broad and baseless accusations, we are trying to be constructive in how we solve this issue.

cc: @ccsr @bajiat

[brylie]

I am un-assigning myself from this issue, until we can have some sort of mediation and shared understanding.

[kyyberi]

@brylie So as an architect responsible for system design you will not be able to define API architecture? I believe the idea is to get things done. Or has this issue been forgotten?

@ccsr

[brylie]

I unassigned myself from the issue because it was becoming too stressful.

In alignment with the user needs described in this issue, we are actively developing the three APIs in separate issues (using the Restivus framework).

• Users API
• Organizations API
• APIs API

cc: @bajiat @ccsr

[kyyberi]

Excellent that things are progressing. What about the "flow API"? What is the plan for it?

Defining API architecture became "too stressful"? Who then defines the aimed API architecture?

[brylie]

What about the "flow API"? What is the plan for it?

The Flow API is somewhat broad and vaguely defined. It also implies features that don't currently, and may not ever, exist in APInf platform.

Data flow: Use a REST API to follow the traffic, state, changes about the customer APIs (any changing data related to APIs), it will be used for customers' tracking tools, own dashboards. (Open issue: Should this be also MQTT in addition to REST?)

Some implied features there include:

• traffic (may overlap with Analytics/Dashboard API)
• state (may overlap with Monitoring API, implies Monitoring feature which isn't fully baked)
• changes (implies Catalog API versioning/changelog feature, which does not exist)
• MQTT support (implies features/capabilities that do not currently exist in APInf platform)

[kyyberi]

What is obvious is by this discussion in this issue alone, is that there's a lot of lurking APIs. Defining initial architecture (and keeping it updated) would make communication about the future easier.

" broad and vaguely defined". That is obvious. What is the plan to make it defined? Not knowing about other visible/plausible APIs makes it hard for us in C&C team to provide what is missing as information. Flow API naturally uses same data as for example Dashboard API. You do understand that there are internal and external APIs in systems.

[brylie]

Here is a work in progress diagram showing the REST API architecture.

" broad and vaguely defined". That is obvious. What is the plan to make it defined?

I suppose we can ask for more clarification from the customers and communities team about what the user needs are regarding the *Flow API

Flow API naturally uses same data as for example Dashboard API. You do understand that there are internal and external APIs in systems.

I understand on a basic level that APIs might have some overlap, and that APInf moves data around internally in different ways than the public REST API provides. What I do not understand is the justification or need for a 'Flow API'.

[kyyberi]

Flow API provides for customers data about API activities, performance etc for the needs of their own process automation/monitoring. Not all our customers want to use our dashboard in daily activities. I'm not going to put customer names in public. They have their own larger scope (all development and monitoring of API driven services) dashboards. Is that enough reasoning?

If someone can reason to me why we would have separate API serving data to our dashboard and another API facing outside, I would appreciate it a lot. Those should at least to some extend overlap.

[brylie]

Not all our customer want to use our dashboard in daily activities. I'm not going to put customer names in public. They have their own larger scope (all development of API driven services) dashboards. Is that enough reasoning?

Might customers be able to use our Dashboard (analytics) API and other APIs, without the need for a Flow API? I just don't understand why we (or our customers) need an additional API, when we are working to provide REST access to all of our data anyway.

[kyyberi]

I dont care how it is internally organized. Customers need the dashboard feed/data via API. To me dashboard API is basicly the same as "Flow API". Only th scope changes; one use case is internal, the other is external. Nevertheless, the external facing API structure is something that is designed together with customers, not defined by underlying stack or our team alone.

"I just don't understand why we need an additional API, when we are working to provide REST access to all of our data anyway."

How was I to know that since there has not been any kind of API architecture to look at? What information? How? What kind of API design? How many APIs?

[brylie]

Customers need to dashboard data via API. To me dashboard API is basicly the same as "Flow API". Only th scope changes; one use case is internal, the other is external.

What do you mean by 'internal' and 'external' use cases?

What are the distinguishing customer needs that cannot be met by the Organizations API, Analytics (i.e. Dashboard) API, and Catalog API, that justify the idea of a Flow API?

What are some ways that our current API designs might support the customer needs?

[kyyberi]

Ok. Now it's my turn to resign from this endless discussion. Just provide the API architecture, data models used, what end points it is planned to have.

[frenchbread]

I dont care how it is internally organized.

Let's not bring things not related to software development to GitHub, please.

---

If there are any difficulties related to API design, I'd suggest we go simple and create something similar to RESTful SailsJS blueprint-api pattern. API-wise this concept has all needed features that APInf might need.