token based authorization is reccomended/recommended. Why?

https://geonovum.github.io/KP-APIs/API-strategie-extensies/#authorisation states token based authentication is recommended, but without providing info why.

I think there should be made a distinction between external API communication and internal communication.

Internal communication is easy: The systems can trust each other, and probably no (state) changes are to be expected, so tokens created during the initiation of the process are still valid during the calling of underlying systems and so don't need to be revalidated in the meantime. Here tokens are the prefered way of distributing authentication/authorization, since they have to be retrieved only once from a authentication server, and can then be distributed across different systems.

External communication is harder: The systems can't trust each other, and you have no control over possible events between requests, like a logout action. So if you use tokens, which are provided by an authorization server and because of the signing the token itself can verified and trusted, you still have to check if the token hasn't been revoked (because the user has been logged out in the meantime). To solve this you can of course reduce the lifetime of the access-token, and use a refresh-token (with a long lifetime) to get a new access-token from the authentication-server, that in its turn can check if the user hasn't logged out in the meantime (now this sounds like a session to me though). Although it's not uncommon to use a refresh token, it does make things somewhat more complicated. Another problem with the refresh-token is that you have to make sure that it doesn't get stolen, since else the bearer can of course gets its own access-tokens from the AS. So developers have to be aware they should handle/store refresh-tokens differently than access-tokens, although they are both tokens. The only advantage of tokens, compared to sessions, is that tokens can contain basic user info, that can be used by the client, although this information can just as easily be provided to the client by an API call.

So in my opinion it is not that bad to use session IDs with secure http-only cookies. Cookies are applied automatically by the user-agent when communicating with the services domain. In the services domain you place a gateway/proxy that checks the sessionID (via a redis session) with the Authorization Server one time per main request (just like you have to do with tokens), and then inject a token further downstream for further internal service calls, so you don't get a cascade of authentication/authorization checks by all underlying systems. The client implementation gets much simpeler, and there are many well-known practices to secure session-IDs and cookies. This in comparison to tokens, that can be accessible by JavaScript and can than leak through XSS.

In modern web applications, state is managed on the client, where it doesn't matter if it's a browser/mobile/desktop or server app. A session cookie is a browser-specific approach and typically used to maintain state on the server, which is explicitly discouraged in the design rules (https://geonovum.github.io/API-Designrules/#api-02-do-not-maintain-state-information-at-the-server). On the contrary, token-based authorization is meant for stateless invocation of API's, which provides lots of benefits in terms of decoupling, scalability, performance etc. and enables features like delegated authorization and single sign-on (SSO).

Tokens are often stateful (e.g. JWT), which means they can be obtained once and contain all necessary information to verify the token for subsequent requests, without having to reach out to the authorization server every time. In that case, a user sign-out removes the access token from the client and revokes the refresh token on the authorization server. The access token will still remain valid during the remaining lifetime (therefore a short expiry lifetime for access tokens and long lifetime for refresh tokens is a best practice), but the user won't be able to use it anymore since it has been removed from the client. The refresh token cannot be used anymore to obtain new access tokens, since it has been revoked. In the experience of the user, it has been signed out instantly, despite the access token remaining valid for a relatively short time. Usually, this is perfectly acceptable.

If you really want to be able to instantly block access to the API for specific access tokens, you must either validate the token against the authorization server for every request (which increases latency) or otherwise implement some other sort of blocking mechanism in your API gateway, but in most cases this is unnecessary and introduces needless complexity.

Token-based authorization (e.g. OAuth2) really shouldn't be hard to implement or insecure, because there are plenty of client libraries which can help you with that and provide a high level of security. On the server-side I would recommend using standard products or cloud services, like AWS Cognito, Azure AD or OAuth0, which play very well with popular API gateway solutions.

Regarding internal vs external communication, I don't think there should be a large distinction. In a microservices landscape, as the number of services grows, the attack surface increases and security risks will grow. Every service should therefore protect its own boundaries and must therefore consider every client, internal or external, as a possible threat (zero-trust model). Therefore, it's a good practice to protect your internal services by using (micro-)gateways or service mesh.

Thanks for your feedback @joostfarla

I agree with you that tokens have a potential (performance) benefit, under the assumption that it's OK that services can still be accessed for a relative short amount of time after a (forced) logout. That is however exactly my point. the recommendation to use token based authorization has been made under the "security" headline, and the only thing I see is a step backwards security wize regarding tokens, vs sessions unless you check the state every request. Of course you can check the validity of a token every request, just like you have to get the session at every request when you are using session-ids, but in that case session-ids are a lot easier to implement than tokens.

You say that only checking if the session is expired when the access-token has been expired is usually good enough, but I don't think that is a very strong argument, especially under the headline "security", and certainly not without mentioning this assumption. At the dutch government they are working towards DigiD and eiDAS substantial and high, creating a very high level of reliability of identification. The fact that a stolen token can than be used to access all kind of services, without the knowledge at the service if this token has been provided by the logged in user and without the user or some operator being able to revoke the access immediately can't be a good security measure I think.

Regarding internal and external services, I still see a distinction in the fact that request from the outside still have to be validated first (check if access hasn't been revoked at the authorisation server), while internal calls are already validated once and it shouldn't be necessary to recheck this every time (when services call each other directly). An API-gateway can optimize this.

I like the discussion in order to describe/find acceptable session requirements. In the past I've already done some research together with a colleague of mine, and concluded that using tokens introduces some additional security risks compared to "good old" sessions. At OWASP many risks and mitigating solutions have been described in order to secure your (web)applications when using session-ids in cookies, and I cannot see that using JWTs should change anything about how to handle these identifiers, although in practise you see that JWTs are handled completely different (and so much more vulnerable for theft).

When you got the token (session-id or JWT) in your possession you can access services for this identity, so you have to be very careful not to expose this information. That is why your should protect yourself against (among others) man-in-the-middle, XSS and CSRF-attacks. One of the solutions to accomplish this is by putting your token in SECURE HTTP-ONLY cookies, so that JavaScript cannot access this content (and leak it to the outside).

In my search for more best practices I found the following blog post regarding the secure use of sessions: https://hackernoon.com/the-best-way-to-securely-manage-user-sessions-91f27eeef460 (please also read part 1) It contains a good overview of different situations, risks and solutions around the aspects of secure session management. It shows how to prevent and identify misuse (based on server-sided sessions, as well as with tokens), with many considerations on how to deal with different requirements, accompanied with an opensource implementation: https://supertokens.io/

In short, prevent token theft, protect against:

XSS
CSRF
Man-in-the middle

Accomplish this by:

placing vulnerable session identifiers in Secure http-only cookies, outside the reach of JavaScript
use anti CSRF-tokens
use STS
apply CSP
use rotating access and rotating refresh tokens
be able to detect misuse

See see https://supertokens.io/benefits for a more detailed overview

Ideal would be to be able to bind tokens to a user agent, but unfortunately token binding has not been adopted as a feature in browsers, and creating your own binding implementation with the use of JavaScript encryption functionality (creating a private/public keypair, where the private key is securly stored in your browser) is very hard to do right (you have to sign secrets, that can handle replay attacks), and introduces complexity for both the API and developers, while introducing performance issues for both users and systems.

FIDO 2 might introduce some nice opportunities to bind a session to a user session, but it requires users to have a compatible device with possibly additional security hardware, and a user-agent that supports it. Although the possibilities and benefits look very promising, but for now the adoption of these requirements seems to low.

Apps still have to make sure tokens are handled and stored secure, but in contradiction to web, probably don't have to deal with the complications of unsandboxed scripts having access to your session-identifiers. However the API still have to be able to deal with both web and app requests..

Finally I want to add a link to a presentation of netflix, showing that they too show a similar implementation as I mentioned, regarding a gateway that deals with handling the authentication identifiers for outside requests, before internal services are being reached. https://qconsf.com/system/files/presentation-slides/qconsf2019-satyajit-thadeshwar-user-device-identity-for-microservices-netflix-scale.pdf

I completely agree with Leon van der Ree with his critique on this section of the standard:

So in my opinion it is not that bad to use session IDs with secure http-only cookies.

I second this for the same reasons given, notably:

in practice you see that JWTs are handled completely different (and so much more vulnerable for theft).

I do disagree with a remark made IMHO too lightly:

Of course you can check the validity of a token every request,

Practically, in a high traffic API, you can't, as secure token validation has very high costs. I feel that being able to implement a performant API at reasonable cost is important for government APIs. Therefor some form of authentication (and probably authorization) caching will be necessary, practically (re)implementing session IDs.

I do not agree with any reason I've heard so far to advice against session cookies in the standard (as an alternative to session token). I also want to bring another reason to allow session cookies:

Session cookies are better standardized, documented, supported and simple to understand and implement and that makes them (since they have no security downsides, compared to cached session tokens) more secure.

I'm under the impression that some concepts get mixed up in the discussion. Let me try to define the most important ones before proceeding:

Cookies

A cookie is a standard mechanism to store pieces of data (key-value pairs) in a web browser (client-side). Cookies can be used for various purposes, like storing user preferences or a session ID. Cookies are bound to a specific domain and the cookie data will be attached automatically to every request to this domain, which can be convenient in some cases. Cookies offer the ability to make them 'HTTP-only', which prevents the cookie data from being accessible by JavaScript. This is an advantage compared to other local storage options offered by web browsers, because it guards against XSS attacks. It's important to note that cookies are a browser-standard, which is not implemented by other agents like a server environment or mobile/IoT devices. Non-browser agents typically have other (native) ways to store sensitive information in a secure manner.

Sessions

A session is a way of persisting data belonging to a given session ID (server-side). Sessions are typically short-lived and used to save temporary state throughout the user's interaction with the web application. Examples could be login state, a shopping cart or form data in a multi-step registration form. Sessions are useful for traditional web applications, where every user interaction triggers a full page (re-)load. On the contrary, modern web applications (a.k.a. Single Page Applications) do not re-load pages, but invoke API calls on the background to store or retrieve data. Sessions violate one of the most important REST principles: stateless. Quote:

...each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server. Session state is therefore kept entirely on the client.

Given the definitions above, I'll try to respond to the discussion.

So in my opinion it is not that bad to use session IDs with secure http-only cookies.

Sessions are not RESTful, because they keep temporary state on the server. Cookies do offer nice security features, but are only supported by browsers. API's should be client-agnostic.

in practice you see that JWTs are handled completely different (and so much more vulnerable for theft).

Of course, there is a crucial responsibility of keeping tokens secure and protect the application against XSS. For browsers, that's the current reality. Other environments, like mobile devices, do have secure ways to store sensitive information. Cookies can be protected against XSS, but might still be vulnerable to CSRF attacks.

the recommendation to use token based authorization has been made under the "security" headline, and the only thing I see is a step backwards security wize regarding tokens, vs sessions unless you check the state every request.

This depends on the way you look at it. By not storing session information on the server (and thus being stateless), the (resource) server has no notion of a user being "logged in" or not. It only knows how to verify a token. How the user has obtained the token is not important and can be completely decoupled (e.g. OAuth2).

Of course you can check the validity of a token every request, just like you have to get the session at every request when you are using session-ids, but in that case session-ids are a lot easier to implement than tokens.

Verifying a JWT is simple; there are plenty of libraries available. Besides that, JWT's do not even have to be verified by the server application itself, since they can be checked by a gateway sitting in front of your app (e.g. a sidecar container). That's a big advantage when having multiple (micro-)services.

The fact that a stolen token can than be used to access all kind of services, without the knowledge at the service if this token has been provided by the logged in user and without the user or some operator being able to revoke the access immediately can't be a good security measure I think.

If this really is a concern, even with ultra-short-lived tokens, you could implement some kind of revocation list, which can be checked for every request.

As a last note: there is nothing against using sessions for traditional web applications, which perform a page-render on the server-side. I'm just saying they should not be used for API's.

By not storing session information on the server (and thus being stateless), the (resource) server has no notion of a user being "logged in" or not. It only knows how to verify a token.

This is good explanation of REST theory and it would be great if that were implementable in a high traffic API. Unfortunately, it is not (due to high CPU load/latency). We need to cache the validation of the token and by your definition create state on the server. Since we do that, we might as well allow a standardized, explicit, popular and proven (arguably more secure) method to cache the validation (session cookies).

@joostfarla We agree on the "stateless" theory, but can we also agree that the theory is not practically implementable for high traffic usage?

NB: I don't only mean the technical (RSA) verification process of the token signature, I mean the actual authentication (and probably authorization) validity (including revocation lists etc).

(edits: trying to clarify while keeping it short and to the point)

This is good explanation of REST theory and it would be great if that were implementable in a high traffic API. Unfortunately, it is not (due to high CPU load/latency). We need to cache the validation of the token and by your definition create state on the server. Since we do that, we might as well allow a standardized, explicit, popular and proven (arguably more secure) method to cache the validation (session cookies).

I'm not sure what you mean by "caching the validation of the token" or how a session cookie would help with that. Either way, for RESTful API's, access tokens must be verified repeatedly for every invocation and should not rely on a cache. Verifying the signature and validity is not a CPU-intensive task and can easily be performed in a negligible amount of time (<1ms).

We agree on the "stateless" theory, but can we also agree that the theory is not practically implementable for high traffic usage?

Performance and scalability are in fact great benefits of stateless communication and using JWTs. Since the token contains all data attributes (claims) necessary to perform authorization, there's no need to consult a datastore for user details or privileges. And because the token is cryptographically signed, there's no need to consult the token-issuing server to confirm its authenticity. This makes them a perfect fit for low-latency API's in distributed environments.

NB: I don't only mean the technical (RSA) verification process of the token signature, I mean the actual authentication (and probably authorization) validity (including revocation lists etc).

Having a revocation list complicates things a bit (that's why this should be avoided where possible), but this could still be implemented in a performant manner for high-traffic API's (e.g. by synchronising the list with every gateway instance).

I would recommended reading the following article, which might clarify things: https://auth0.com/blog/stateless-auth-for-stateful-minds/

Okay.. I think I get it.. thank you for your thorough explanation and your patience..

@joostfarla Thanks for your feedback and for showing that you also know where you are talking about. I am also well aware of the concepts like sessions, cookies, and tokens.

I also think we might say the same thing (or come the the same conclusion).

In this issue, I begin with mentioning that "the article states token based authentication is recommended, but without providing info why". What I mean with this, and what I often see in practice is, that web-developers are using API's in a same way as other (native) app developers, without being aware of the differences and risks their browser environment are placing them in.

I completely agree that an API of a (micro)service, should not be aware of the state/session of the user. However, when an API is providing access to a service, with only a token, than there aren't many developers that will think of introducing some sort of proxy that can mitigate the risks of their (web)environment. And therefor I think that, in a chapter about security, it would be good to mention the risks there when you are using tokens. That are things like XSS in web enviroments, and stolen tokens (because of device theft, or whatever reason there can be), that you might want to be able to revoke.

Ways you can mitigate these issues are complex: in web you should be aware of XSS, which can be mitigated with HTTP-ONLY cookies (containing tokens, or session-ids, so that web-services can act as a proxy to the API, replacing the session ID with the token that is stored server-sided), and by implementing refresh tokens, with all the means necessary to be able to revoke access as soon as necessary.

So in short, the security chapter currently contains to little detail, and I think that with all the info combined in these comments we can improve it a lot.

Thanks for the interesting discussion, @lvanderree @mevdschee! I fully agree that chapter 4.3 is incomplete and inaccurate on this subject. It should substantiate those principles and it should provide guidance and awareness of the security risks. Hopefully the security working group will improve the contents based on the suggestions in this issue. /cc @fterpstra

It looks like there are two different dimensions: the difference between tokens/sessions vs the difference between different means of storing information (e.g. HttpOnly cookies vs JS accessible storage).

I do fully agree with @joostfarla that sessions violate one of the most important REST principles, statelessness, which is an essential enabler for true scalability. It may be required to store some information about provided tokens on the server to accommodate instantaneous log-out / revocation lists, but apart from that, the client should store the session information.

Regarding the storing of data, it looks to me like there is a gap even in the OAuth and OpenID Connect specifications, as they are both based on communicating session secrets (tokens) via elements in the response body of the token endpoint responses, which is accessible from JavaScript clients and vulnerable to XSS (as rightly mentioned by @lvanderree ).

Hello. I am the co-founder and CTO at SuperTokens. Thank you all for checking out our article mentioned by @lvanderree. Since this discussion is open to the public, I assume that my comments are welcome.

Due to the lack of clarity in the terminology, I will first define the various terms:

Session: A way for a client and server to communicate with each other on behalf of a previously authenticated user. Using "sessions" the server can know which user is making a request, and also know that that the client being used for this request, was also used by the user to previously successfully authenticate themselves.
OAuth: This is a protocol defined only for the purpose of 3rd party delegation. It is not to be used, as is, for the purposes of first party communication. I can get into the details of this if you all want.
Cookie: This is a storage mechanism on browsers.
- It is used to store key-value pairs which are sent to the server on each API call.
- A key and value will be sent only if certain conditions are met (for example if the domain and path match).
- Key and values can also have the httpOnly flag in which case, the client JS cannot set, read or modify that key and value.
- You can store any key and any value for any purpose. The only limitation is the max storage size of 4kb per domain.
Localstorage: This is an alternate storage mechanism on browsers.
- To store key value pairs.
- Is readable and writable by JS running on a page.
- Is per domain. The localstorage for "a.com" is different to that for "b.com"
Tokens: A string literal. Used by the backend to identify the user making the current request. How it's used depends on the type of token:
- Json Web Token (JWT): These are tokens that have a specific structure to them. They contain some "payload" information that can be read by parties that have access to it. To prevent alteration of this payload, this token is signed by the party that issues this token, and must be verified by each party that wants to read it.
- Opaque Token: These are tokens that have no meaning in themselves. They are literally just random in nature and are "difficult to guess". They act as a pointer to information stored on the backend cache / database. Any information can be mapped to them.

Stateful vs Stateless in the context of restful API

My view on this is a little controversial, but with good reason.

It's clear that the main purpose of wanting stateless is to scale easily. In order to understand this statement, we must define what are the bottlenecks that prevent scaling, and what does "stateless" actually mean.

What does session stateless actually mean?

There are multiple definitions for this. Some say that one request should not depend on another. Others say it means no session data is stored on the server side.

In case of Opaque tokens, we will have to store all session data on the server side. However, requests can still be "independent" to each other. What independence means is also not clear since, for example, we would want a "cart checkout" request to succeed only if an "insert item in cart" request was made before. This way, they are not independent... So perhaps, Opaque tokens are not stateless.

In case of JWT, we will have to store the JWT signing key on the server. The same argument about independent API calls holds here too. Does this mean JWTs are not "truly" stateless either?

Finally, do we really want to be stateless when it comes to sessions? This will prevent important business analytics like knowing how many devices are logged in, or allowing users to choose which device they would like to log out of (like apple and google do.)

Hence, personally, I have dropped the notion of worrying about RESTful, stateless APIs. Instead, I focus more on what prevents scalability:

What are the bottleneck for scalability?

In most web apps, the biggest latency is caused by network requests in APIs (database / cache calls for example).

An API with 0 network requests will be orders of magnitude faster than an API with 1 network request.

On the other hand, if we have an API with 100 sequential network requests, removing one of them will not really improve speeds noticeably. However, I think it's safe to assume that most APIs have 1 to 3 sequential network calls.

Given the above, the fewer network calls we can make, the better it is. Hence, in deciding which is better for scalability - Opaque tokens or JWTs, we should consider, not statelessness, but the number of network calls your APIs are likely to have.

Opaque or JWTs

There is no right answer here. This blog post that I have written will better be able to explain the difference.

In short, it's possible to scale with Opaque tokens (Facebook doesn't use JWTs) and it's also possible to minimise all the risks of using a JWT. However, it is harder to scale with Opaque tokens, and harder to secure JWTs. In general, I recommend that if your API is expecting a very high amount of traffic, then stick to JWTs since API speed and uptime is more important than security, given that we can secure the APIs extremely well even with JWTs (As mentioned in the blog link above).

One point I would like to bring about is that we talk about wanting to be able to revoke a session immediately in case some anomaly like theft is detected. But what we fail to discuss is how do we actually detect theft? Because only after proper detection (no false negatives / positives), would we even consider revoking the session.

Detecting token theft

The classic methods for this are to use IP address and device fingerprint pattern matching.

If you user's IP address changes "drastically" you revoke the session. However, this can cause issues when a user uses a VPN. Also, an attacker can bypass this by being "roughly" in the same location as the targeted victim.

If the user's device fingerprints change, then you revoke the session. If an attacker is able to steal the session tokens (Opaque or JWT), they will likely also be able to get the computed device fingerprint, rendering this useless.

That being said, both the above methods can be used, but not on their own. We should instead use the concept of rotating refresh tokens (as recommended by IETF in their RFC here) to really detect refresh token theft in a reliable manner.

Access token theft can be detected by the traditional methods, and then the access token can be revoked if necessary. This will in turn result in the users using their refresh token, in which case we can reliably detect theft if it has occurred. If no theft has occurred, then the users will not be logged out.

I am happy to go more into detail of this section

Where to store the tokens?

For browsers, it's recommended to always use httpOnly, secure cookies to store the access and refresh tokens. This will prevent token theft via XSS attack (which can be done in a variety of ways including social engineering - something you have no control over). If using this, we will have to prevent CSRF attacks, but those are completely solved for.

If the token information is to be read on the frontend for business logic, then you should use another token called Open ID Connect tokens, whose only purpose is to be read by the client. This token cannot be used to get access to APIs - it is not the access token. I'll be happy to elaborate more on this.

Conclusion

I would like to recommend that you use:

Short lived JWT access tokens with long lived, rotating, opaque refresh tokens.
Use httpOnly, secure cookies and also prevent against CSRF attacks.
Do JWT signing key rotation (very, very important)

Some more comments

@joostfarla said: On the contrary, token-based authorization is meant for stateless invocation of API's, which provides lots of benefits in terms of decoupling, scalability, performance etc. and enables features like delegated authorization and single sign-on (SSO).

Delegated authorisation is different to first party authorisation. Delegated authorisation can be done by following the Authorisation code grant flow (which requires the use of JWTs) mentioned in the OAuth spec. However, for the purpose of first party communication, using JWTs or Opauqe is only a matter of scalability vs security.

Single sign on can also be implemented using Opaque tokens.

@joostfarla On the server-side I would recommend using standard products or cloud services, like AWS Cognito, Azure AD or OAuth0, which play very well with popular API gateway solutions.

These products offer user management as a service. They allow you to access and manipulate your user's information via OAuth 2.0 protocol. They are not meant to be used to enable first party communication between your frontend and backend (even though people do that, which is a mistake from a security point of view).

@lvanderree The fact that a stolen token can than be used to access all kind of services, without the knowledge at the service if this token has been provided by the logged in user and without the user or some operator being able to revoke the access immediately can't be a good security measure I think.

As discussed earlier, we must also consider how we will go about detecting such a theft.

@joostfarla Cookies do offer nice security features, but are only supported by browsers. API's should be client-agnostic.

Cookies are transported via headers (Set-Cookie header). If a device does not automatically handle cookies (Android and iOS do), then one can also read that header in the client side and store the tokens wherever possible. This way, the API can use cookies, which is necessary for web clients, without worrying about other client types (APIs are already client-agnostic).

Thank you for taking the time to read the above information. I hope this helped! I'll be happy to answer any questions.

@rishabhpoddar First of all thank you for your extensive and insightful reply.

I have another question: In reality most government API's (I worked with) are provided only (some) authentication claims by some OAuth provider and no authorization grants. Thus leading to a scalability problem if you don't cache the authorization grants (those require expensive look-ups in an authorization service). How would you deal with this situation?

Hi @mevdschee.

authentication claims by some OAuth provider and no authorization grants

I don't quite understand this sentence. The term "authentication claims" is extremely overloaded in general. It would help if you clarify.

if you don't cache the authorization grants

I'm not sure I understand the meaning of "authorization grants"? In the context of OAuth, do you mean access, refresh, ID Connect tokens? Or authorisation code (a token used in Authorisation code grant flow to get the access and refresh tokens)?

Thanks

Also, is this discussion for the purpose of defining sessions between your system and another system (OAuth), or between your system's backend and frontend? Or both?

Okay.. so in most API systems the JWT claims are either "who you are" (authentication) or "what you are allowed" (authorization). I was using the words "authentication claim" vs "authorization grant" to make the difference between those two types of claims in the JWT. We need an LDAP tree with organizational structure and one with grouped permissions and cross connects between them, defining the "authorization grants" that we are obliged to report twice per year to the "application owner" conform the (Dutch) government laws (BIO / BBN2). Most environments thus have an authentication provider and an authorization provider, both supported by a "tree" of either of organization structure or grouped permissions.

...is this discussion for the purpose of defining sessions between your system and another system (OAuth), or between your system's backend and frontend? Or both?

Both, as we are trying to make an API standard for all (Dutch) government APIs, not ruling out any use case.

Thanks for the clarification. I now understand the difference between "authentication claim" and "authorization grant".

Furthermore, I understand that you defining how the Dutch govt's APIs (DGAPIs) are to be accessed, by third party services (any external app that wants to talk to the DGAPIs - for whatever reason) and by first party clients (Dutch govt's own frontend clients, like their website, where users can login and then access DGAPIs).

If the above is correct, then when you say "are provided only (some) authentication claims by some OAuth provider", who are these OAuth providers? Don't DGAPIs store information about its users in its own database?

Regardless, to answer your question, if the authorization grants are not passed in the JWT, then yes, you would have to cache that information. I would recommend having that information as a part of the JWT itself though (if that's possible).

Also to clarify, the OAuth flow that DGAPIs implement can follow the standard OAuth flow - Authorisation code grant flow with PKCE is what the new OAuth 2.1 standard will recommend. On the other hand, for first-party access, OAuth doesn't work as is - you will need to implement something like rotating refresh tokens if you want to maintain long sessions for your first party clients (regardless of the access token type).

I'm not sure if the above answers your query @mevdschee. The system architecture you are talking about is still quite vague in my head. I am happy to use some other method of communication if you'd like.

@rishabhpoddar Thank you again for your extensive reply and especially for explaining the use of OAuth in a complex architecture like that. I guess you are right and we are talking about system architecture and not about API design. Then again we should be careful not to forbid a good system design with too strict API guidelines.

I would recommend having that information as a part of the JWT itself though (if that's possible).

I agree and I believe that that is what @joostfarla was suggesting as well. This means we cannot use authentication providers to directly provide a JWT for our API, but we need to talk to an authorization provider first to enrich the JWT with authorization information.

Sorry for my late reply, and thank you @rishabhpoddar for your insights.

I finally got some time to summarize all provided information and write a recommendation.

I think this discussion showed the following insight:

Tokens are used to provide (among others) authentication and authorization information. This can be done via:

opaque tokens, like session-IDs and other Unique identifiers containing a reference to a (session)store on a server where all information is stored
self contained tokens, like JWTs, that use cryptographic functions to protect the integrity of the content

Both alternatives have there pros and cons:

Opaque tokens are always up to date, very small, but harder to scale.
JWTs are mostly self contained, so they are bigger, and they can possible get revoked (user logged out all sessions, or for example handle single log off calls) or their content can get outdated (user rights are being altered). This still needs to be looked up centrally (which can be done via a (rotating) refresh-token strategy), unless this isn't important for your use case.

API's should be usable for:

apps, that like to exchange tokens via a header
as well as for (JavaScript in) web, that can only transport tokens via (HTTP-Only) Cookies in order to protect against XSS (although then introducing a vulnerability to CSRF, that can luckily be mitigated via for example the SameSite Cookie attribute, as wel as by applying CSRF-tokens). However this also makes the content of any JWT inaccessible to the client, requiring an API to still provide this information when desired.

This difference in transport is something that can be handled via the API, by supporting both mechanisms: Allowing for token transport via cookies, as well as via headers. However it is important that the client is aware of the appropriate solution he should implement (which comes down to: web use http-only cookies).

Said that, it is good to realize that there can be a difference between the outside API that your clients should implement, and the API that your (or even external) services describe, by placing something like an API gateway in between! This gateway is responsible for handling the token of your client. It can talk to a session-store, or check for the validity of a JWT and even exchange the token from the client, for another (possibly even refreshed) token which is used to communicatie with the service performing the actual action.

Security For both JWTs and opaque tokens, you can implement a rotating token strategy to be able to detect token theft. However with JWTs you can separate the communication with the Authentication Provider, form the communication with the Resource Services, by introducing the use of a refresh-token. This makes it possible to be able to coop with parallel/asynchrone requests to (and more importantly responses from) resource services, without getting lost in tracking what the latest token is supposed to be.

Scalability JWTs can be exchanged to all endpoints that trust the authentication server. Opaque tokens are only usable local, but when the client is communicating (with this token) via a gateway that isn't really a limitation: The gateway can look up the session (possibly providing the token(s) used to communicate further downstream), and give access to all kind of resources.

We are potentially looking for solution in which a nation wide Authentication Provider (like DigiD) can provide authentication confirmation about an identity, but I can't imagine this AP can provide all the authorization details for every service in the country. This means additional information has to be added to the user session/token.

Summary An Authorization Code Grant flow, that is implemented in an API gateway that will add additional authorization information next to the authentication claim from the Authentication Provider seems to be able to fit this situation. The gateway can then use its own (up to date) (access) tokens to communicate with the APIs of the Resources it provides access to. The client can communicate with this gateway via both opaque and self contained access tokens, but in order to be able to detect token theft it should Implement some kind of rotating token strategy, which is easiest to implement when separating access token functionality from refresh token functionality (in order to coop with parallel async resource consumption).

Implement some kind of rotating token strategy, which is easiest to implement when separating access token functionality from refresh token functionality (in order to coop with parallel async resource consumption).

Is there any way I (SuperTokens.io) can help you out in building this? We have an open-source project that facilitates exactly that. The project is unique in its ability to handle all the edge case issues when implementing rotating refresh tokens while maintaining scalability.

I'd be happy to chat more about it on a call if that works for you. Else, i'll also be happy to answer questions here - should you be interested in this.

If the above seems spammy, I apologies for it. I would not be suggesting my product unless I genuinely believed it solves the problem you are discussing about (or at least a core part of it). Thanks

Geonovum / KP-APIs