Mark endpoints as private/hidden

[mohsen1]

With Swagger inflector and Swagger-node it's now more common to have endpoints that we don't want to document but we want to have them in the spec for other uses. There should be a way to mark endpoints as private or hidden so they don't show up in the UIs but can be used with other tools.

[webron]

Need more that. What are some use cases where you don't want to document them, you do want them in the spec, but don't want them in the UI (which is sort of a contradiction on its own)?

Just trying to understand here.

[mohsen1]

Let's say you have an API that has some endpoints that you want only internal users know about it. In case swagger.{json,yaml} is exposed externally the private: true endpoints are stripped down by the server if user doesn't have "internal user security credentials" they won't see those endpoints .

This also can help swagger-node and swagger-inflector return 404 errors for secure endpoints when a request with invalid credential is coming in. GitHub API does this, if you don't have valid credentials you won't know about some secure endpoints at all.

[webron]

That's ACL on the Swagger definition which if we choose to add should be handled in a much different way than just adding a private/hidden property to an operation (there are other considerations as well, such as hiding tags and possibly others).

I believe we discussed it for 2.0, and not saying it should be dealt with in future releases, but the filtering may actually be the responsibility of the tools and not really part of the spec itself. I don't know the 'right' answer here.

[mohsen1]

Right, I think a simple boolean property will not be enough and there could be other scenarios more complex than internal user vs. public.

Swagger can use it's own security system to define what operation is available to view to which credential owner. By default all operations are available to public to see unless the showOnlyTo has the security that you have the credentials of.

For example:

swagger: '2.0'
info:
  version: 1.0.0
  title: Example
securityDefinitions:
  basicAuth:
    type: basic
paths:
  /:
    get:
      security:
        - basicAuth: []
      responses:
        '200':
          description: 'OK'
  /foo:
    get:
      x-showOnlyTo:
        - basicAuth
      security:
        - basicAuth: []
      responses:
        200:
          description: OK

In this Swagger, if I don't have basicAuth credentials I won't see /foo in the documents or swagger.json.

[webron]

I'd probably go with a different name, and possibly with a different kind of granularity, but yeah. At least now it can be solved with vendor extensions. We'll need to decide whether it should be part of the actual spec, but it's a legitimate overall use case.

[theganyo]

Something like x-visibility or x-doc-visibility, maybe?

[webron]

Visibility is a result, but it's not necessarily the only issue. You may want to do something further like explicitly disallow execution based on specific credentials and not just hide it from the documentation. In this situation, the Swagger definition is not necessarily used for documentation purpose only anymore.

x-access, x-acl and so on may be more fitting. The value may not necessarily be a simple string or array either but a complex object (for different access levels and so on, or access groups and so on).

[theganyo]

Isn't the "security" section for execution access control?

[webron]

The security refers to the requirements to execute the API, but it doesn't mean it should be hidden from the user. It just means the user needs to be aware of what security procedure is required before they an execute a request.

[theganyo]

Right. What I'm saying is: The security section is already there for execution. We were talking about visibility. Why are you saying that we need an x-access to also control execution?

[webron]

And again the quote "There are only two hard things in Computer Science: cache invalidation and naming things." (Phil Karlton) comes to mind. I get what you're saying regarding the security section and the execution, but I see the restriction as somewhat different in the two cases.

The security is always exposed to the user, but the new property isn't (or is it?). The first is documenting the API, the second is meta documentation. I guess visibility could work, though to me it would mean something slightly different.

[theganyo]

Well, I see several domains here:

1. Expression of an API as documentation
2. Authorization of access to sections of the API documentation
3. Expression of an API as executable
4. Authorization to execute an operation

My confusion is that I'm not sure which tags apply to which domains and how...

[webron]

I'm not sure I see the difference between 1 and 3, so would appreciate further explanation.

For 4. - security is meant to cover it. For 2. - this is what I believe we're discussing right now. To be honest, I'm not sure this even belongs in the scope of the spec.

[mohsen1]

If ACL is not defined in the spec so the server can determine which endpoints to show programmatically the only alternative is to have multiple subsets of your spec served to different credential holders. Having multiple swagger specs for an API is not optimal IMO.

[webron]

It's not the only way. Nothing prohibits you from adding your own vendor extensions to perform it. In swagger-core we allow definition filtering based on information available from the code. There are a few ways to solve it. Not every vendor extension should make it into the spec eventually. It could also be a separate file describing it, as it is meta documentation.

For now I'm just throwing ideas around, no concrete opinion yet.

[fehguy]

See #569

[tomchristie]

Alternative: Serve different variations of the schema depending on the authentication context. That'll allow you to present differing sets of information to different clients.

[petermorlion]

Came here looking for this possibility. I'm totally new to Swagger, but here's my two cents: I believe @webron 's idea on vendor extensions allows for the most flexibility. It would allow the server to decide not to show something based on who is looking (anonmyous, paying customer, premium customer,...), but also other options we can't think of, e.g. only/don't show it after a certain date, etc.

[fehguy]

Yes, this is done in several OSS tools, such as swagger-inflector and swagger-core. Probably best not in the spec, though.

[dkoston]

@fehguy without supporting this in the spec, it won't make it upstream into all the consumers of OpenAPI. For example, I use google cloud endpoints and they don't support private endpoints as it's not really a thing. This means I don't have a way to publish endpoints with them and keep them out of documentation. Private endpoints are going to happen in almost any large API deployment. It seems like supporting this at the spec level would help define private and allow it to push out to the ecosystem. Not supporting it means that you're going to get many different implementations of private which are likely incompatible which is kind of anti-spec.

[jgomes94]

Hey everyone,

what i did was define a tag called 'hidden' and then used custom css to hide it:

app.use('/docs', swaggerUi.serve, swaggerUi.setup(swaggerDocument, { //hide the internal api endpoints customCss: 'div[id^="operations-hidden"]{display:none} #operations-tag-hidden{display:none}' }));

[petermorlion]

But that only hides it visually. It's still there in the source, right? That wouldn't be secure enough for a lot of my clients.

[jgomes94]

Yes you are right @petermorlion, it can still be visible with some DOM manipulation

[MikeRalphson]

@dkoston @petermorlion my personal take is that this is functionality which belongs in a pre-processing tool, not within the OAS itself. That way you can guarantee no leaks of internal endpoints etc between your varying audiences. One such tool is openapi-filter. I understand that if your API management tool generates documentation from the same OAS document as the implementation then this won't help.

It is, perhaps, another pointer that properly secured endpoints with appropriate access-controls are preferable to security-by-obscurity.

[dkoston]

@MikeRalphson the attempt here is not security by obscurity. If people aren't defining security for endpoints they want to have hidden from documentation/tools, that's a huge mistake and could cause major headaches for them.

The thought behind this request is that there are common patterns that will be relegated to custom code/solutions if they are not supported by the spec.

Sure, tens of thousands of people can use something like openapi-filter to achieve the same results, or possibly not hear of it and create their own solution.

Thinking about this again, I do believe it would be a mistake to put something that only supports "hidden/private" in the spec. The real issue is that there are properties of some routes that make it so they should be treated differently. For example: I want to have different versions of my API docs depending on what type of user you are (why send you a bunch of endpoints you don't have access to?).

What's likely a good solution at the moment is to use "tags" on your Operation Object. However, without some formalization of what tag to use, I may choose "private", you may choose "hidden", others may choose to use "x-internal" at the top level of the Operation.

I believe this may just be a philosophy issue.

I personally believe that setting a standard without ambiguity leads to a small set of well maintained tools which allow newcomers to an ecosystem to use and have success with without spending hours/days researching and customizing to get to a common solution.

Alternatively, one can have a very open ecosystem with tons of tools and extensions where some are maintained, some are high quality, and others are not.

The fact that this thread has gone on for years without someone being like "Hey, this pattern can easily be solved by X", means that the OpenAPI ecosystem has some issue (documentation, lack of tooling for this pattern, or something else).

Perhaps a short term solution would be for the team to list well maintained extensions here: https://swagger.io/docs/specification/openapi-extensions/

I'm not sure what the communities' policies are around the concept of "when does something become so common that its part of the spec" and that's the main goal of this request.

The idea is that a large portion of the OpenAPI community deals with hidden/internal/private/segemented endpoints. Adding something formal to the spec would allow tooling to be built that conforms to the spec rather than introducing a new "meta-spec" for each tool.

[MikeRalphson]

The real issue is that there are properties of some routes that make it so they should be treated differently

Unless there is some consensus on what "treated differently" means, by the tooling providers, I think this idea is so vaguely-defined as to be unlikely to be adopted by the spec itself, over other existing mechanisms like specification-extensions.

Though this issue has been open a while, it hasn't garnered that many comments or up-votes compared to others being considered.

There is a plan to create a registry of common / standardised specification-extensions, see #1351

[dkoston]

Unless there is some consensus on what "treated differently" means, by the tooling providers, I think this idea is so vaguely-defined as to be unlikely to be adopted by the spec itself, over other existing mechanisms like specification-extensions. I was purposefully being in-specific as "hidden/internal" is not the only property I think could be valid here.

I didn't want to dictate to the community what properties I think are important but just to note that changing the spec to add "hidden: true" seems like a lot of work to capture a single use case and something that captures multiple use cases may be a better solution. Maybe there are too many use cases and it's best to just add a single property as suggested above.

I'm totally fine if the project says "we aren't going to do this", that's up to the project.

However, we're getting mixed feedback. @webron thought this idea may have merit and @fehguy opened an issue relating to a solution. A third member (yourself) said it can be handled via extensions.

I think we're just looking for some clarity on what to do when we come across endpoints that should be treated differently. It seems like the 2018 solution is to use extensions but we haven't gotten a definitive: "the project suggests that you handle that use case this way"

People look to the maintainers and contributors of a project for solutions as they have the most experience with the ecosystem. If we're left to go hunt down solutions on our own, we only have so much time and energy to invest as a single person with the problem. Whereas, it's likely that the community has come across this scenario many times and can provide at least a good starting point.

There is a plan to create a registry of common / standardised specification-extensions, see #1351

Great, that's a good step, glad to hear it.

Having some documentation that talks about use cases for the project and how they can be accomplished could really help out those trying to use the project as well.

A good example of this is the design recipes from FoundationDB. FoundationDB chooses not to maintain all those features in its core but they know that there are common use cases their customers have and by pointing users in the right direction, adoption will likely increase in velocity.

[webron]

To make things easier (not really), I'm not really sure it belongs in the spec. It's my (personal) belief that we should encourage API definitions to be public and not just used as a way to automate processes. If that's the case, then having a way to 'hide' something from the documentation won't really help.

@dkoston wrote:

For example: I want to have different versions of my API docs depending on what type of user you are (why send you a bunch of endpoints you don't have access to?).

That's really a classic use case for pre-processing as @MikeRalphson mentioned.

I understand the desire to have us give an official statement as to how it should or shouldn't work, but at the end of the day, we can't address everything. The fact that since the ticket was created there hasn't formed a commonly-used solution also suggests that it doesn't hurt people enough to push this through.

With all that said, soon (🤞) we'll have the Overlays feature implemented, and that would allow you to separate the public and private APIs into different layers and expose the user only with the intended information. This will not cover dynamic cases as much, but should still address many issues. To make this extra sweet, by adding overlays, pre-processing will end up being a must.

[dkoston]

@webron Thanks for the clarity. Seems like pre-processing and/or Overlays is the approach to take when features aren't supported by the core spec.

[wizofaus]

I was looking at the HTML generated by swaggerui, and in principle you might be able to hide entries via CSS styling (which is actually what I want - just to make the documentation more readable for third parties. Internally I only use the raw swagger output for generating/updating clients etc.). However, for instance I'm seeing:

<div class="opblock-tag-section is-open">
    <h4 class="opblock-tag no-desc" id="operations-tag-MyPrivateOps">
    <a class="nostyle"><span>My Private Operations</span></a>                       
...
</div>

And as of late 2019 there's still no way of styling an element based on some property of its children (or conversely, having a selector control the styling of the parents of matching elements), so I can't readily hide the whole opblock-tag-section div (actually I probably want to hide the span above that too).

For the time being I can cope with something that just hides the key elements ([id*=\"MyPrivateOps\"]{display:none !important}) but it's definitely not ideal. And yes, it would be better if the HTML weren't generated at all.

[jondeandres]

any update on this? what's the way to mark endpoints as disabled? or change their visibility?

[tbarn]

@jondeandres depending on what tooling you are using, there might be an extension for this.

[jondeandres]

mmm thanks @tbarn, we are using redoc, but I don't see anything related to thisb

[thernstig]

Wouldn't a simple private: true or public: true (mutually exclusive) suffice? Or scope: "private/public"? I am sure there are more advanced scenarios to cover, but I think the main gist here is to cover the public vs. private APIs similar to how functions in code can be marked with @private or @public for documentation purposes (these tag examples is from JavaScript and JSDoc/Typedoc, but also exists in other languages).

Using the private/public terminology would help users understand it better as they can relate it to function scopes, which is a big win.

The advantages of supporting this are clear to many users. We have APIs we wish any user to be able to call, but we also have APIs that can be called, but they should not be called as we don't support public API operations on them.

For me it is just an annotation for tools to use (e.g. genering docs which can hide @private if wanted), and tools can decide various way to handle those tags to accomodate users needs.

[negarnegma]

Can we show(hide) some APIs to(from) some users? or hide some APIs that ui users can access via a password/token?

[slogic]

I need to make hidden endpoint to overcome issue with validation of callback enpoints which are not supported by latest atlassian validator.

[adamaltman]

It doesn't look like this is going to make it into the spec (at least anytime soon).

This use case can be accomplished using specification extensions. Hiding from the UI is one thing. Removing from the OpenAPI definition before rendering the UI is better.

I made a tutorial with a sample implementation showing how you can use specification extensions to hide an internal API (I hide an operation, and a schema property).

[kscheirer]

I will take a look at if/how overlays can solve this problem. This is part of determining whether overlay spec is sufficient. I think it is more or less the same as @adamaltman's, but there is no tooling support for overlays yet. So his solution definitely wins for being something you could implement today.

Using an overlay with traits (x-oai-traits) would look like this

{
  "overlay": "1.0.0",
  "info": {
    "title": "Private Endpoints Overlay",
    "version": "1.0.0"
  },
  "actions": [
    {
      "target": "info",
      "update": {
        "x-overlay-applied": "private-endpoints"
      }
    },
    {
      "description": "Remove any operation with private trait'",
      "target": "paths.*.*[?contains(x-oai-traits, 'private')]",
      "remove": true
    }
  ]
}

schema would be something like

{
  "openapi": "3.0.0",
  "info": {
    "version": "1.0.0",
    "title": "Swagger Petstore"
  },
  "paths": {
    "/pets": {
      "get": {
        "x-oai-traits": ["private"],

[kscheirer]

"paths.*.*[?contains(x-oai-traits, 'private')]" is JMESpath, not JSONpath, will update that portion

[adamaltman]

@kscheirer

This is part of determining whether overlay spec is sufficient.

My answer is no. I wrote a blog post about it:

https://redocly.com/blog/openapi-overlays/

[BillyBolton]

I would like to see more granularity than just internal/external, public/private since this assumes there are only two types of visibility. I've had a use-case where I needed to dynamically expose endpoints based on where the specification is being deployed to, so using hidden, or the possibility of internal/external, would not suffice. (There were three separate environments I need to consider, for example).

There's a couple comments I'd like to contribute for this issue.

In SpringBoot, I've implemented a rather verbose way to dynamically hide/expose operations using extensions but the level of pre-processing that is required is quite expensive. I can't share the code but basically I decorate the Operation model using extensions, which I then use to scrub the Operation from the spec's Path/PathItem based on the extension's value vs deployment environment using the OpenApiCustomiser. Once this extension is used for this scrubbing, I then remove the extension from the operation since it is only needed internally.

OpenAPI unfortunately only offers customizers for OpenApiBuilderCustomiser, OpenApiCustomiser, OperationCustomizer, ParameterCustomizer, and PropertyCustomizer. These customizers are great for pre-processing prior to generating the spec. There are cases, however, that other components such as $ref/definitions/Schemas, Tags, etc. need to also be hidden when they are not referenced by other now hidden components. So scrubbing needs to be done recursively after the spec has been generated, in my case. If customizers were available for all components in the OAS, and especially for Paths and Schemas, then this scrubbing could be done more efficiently prior to the spec being generated.

It seems there is a sentiment that a solution to dynamically hiding APIs should be provided by vendors or third-parties. There are implications for this, however, that I don't see being considered. Firstly, if many vendors are having the same use-case, as shown in this ticket from 2015 and others, then there should be a way for the tool itself to provide a flexible solution that answers this call-to-action. I'm not clear why it shouldn't otherwise if so many users over such a long period of time are stating they need this feature. Secondly, relying on third-parties means introducing dependencies into a stack that just are not needed -- potentially vulnerable dependencies at that too, which could also implicate strict security standards users need to abide by due to the stakeholders they develop for. I understand this risk is the expense of open source, but assuming third-parties are as trusted as the tool itself does not seem appropriate. These third-party tools are trying to compensate for something that should be offered directly, after all.

I look forward to when this ticket moves beyond the OpenAPI.Next Proposal label.

[adrienbricchi]

Since the deprecated tag already exists, maybe an experimental tag...
Those keywords often works together, and the first is already there.

That would do the trick for me, to discriminate internal (private) and public (supported) API entrypoints,
without having to create some kind of complex granularity.

[wparad]

The real problem here is that it turns out a public/private flag is looking to be used in many different ways. And that would normally be sufficient, but it becomes clear reading through the issue that each of this situations that's actually not sufficient for most use cases. If it doesn't solve even one use case perfectly, and it also isn't extensible, adding it to the spec is clearly a bad idea.

It sounds like the right way to approach thin problem is rather than just complaining that the spec doesn't have this, is to contribute what problem you are trying to solve. Those problems/user stories help drive new features. Let's not directly talk about "we need this or that flag" that's not as productive.

For instance, one solution suggested in this issue has been use overlays and another is build separate specs. Both of those are great suggestions. But don't complain if they don't work for you, you need to contribute why they don't work. Otherwise they won't be improved. Further, you almost certainly open a new issue.

An example, for instance regarding public/private/experimental is to clearly document your SLAs. Maybe today you believe you are using a flag to do that, but really it's an sla problem. Maybe the solution is have the SLA be a real first class object with extensible properties Also as many pointed out this could be a tool problem. If every tool has a concrete problem with their implementation that isn't something that can be fixed at the spec level.

TL;DR if spec can't do something you have a need for, open a new ticket. Dumping complaints in here to bespoke problems will have it lost in the deluge of others' unique situations. If it's a revelation about the implementation of public/private and how to correctly solve it's extensibility issues, please share.