Standardizing serving the API on the unversioned URL

[rartino]

This was brought up at the end of the OPTIMADE 2020 workshop:

Right now the specification does not forbid, but also does not in any way endorse, serving the the API on the unversioned base URL.

Do we want to standardize how the API is served on the unversioned base url URL?

If we cannot agree, then the "default" is that we release as v1.0.0 the specification that neither forbids, nor endorses, serving something at the unversioned URL.

This question also ties into what the proper procedure for version negotiation between a client and server is. Presently the specification does prescribe the following procedure:

For implementers: Clients are recommended to discover the highest version supported by both the client and the API implementation by trying versioned base URLs in order of priority. E.g., if major version 2 and lower are supported by the client, it would try: /v2, /v1, and then /v0.

[rartino]

One suggestion from me in the discussion was to add something along these lines to the specification: "The response to any request to the unversioned base_url beyond the root MUST result in an HTTP 307 redirect to the same URL that was requested but prefixed by /v1, where 1 is to be replaced with the latest major version of the API that the implementation serves."

[sauliusg]

The impossibility of the OPTIMADE API to easily list available versions for a client is a nuisance, if not a serious drawback. It would be nice to have it specified in 1.0.

A suggestion would be the following: "if an OPTIMADE endpoint is queried by a client, an implementation MAY serve the most recent stable version implemented by a provider."

So you could always query a default endpoint, e.g. (https://www.crystallography.net/cod/optimade/info), and get the reasonably machine readable version information. Here "reasonably machine readable" means that either your client can parse it, or it can detect that it can not parse it and report that fact to you as an error message. The only commitment from OPTIMADE spec. developers that is needed for this to work is to make sure that the same JSON key, "api_version", is never attached to anything else except the OPTIMADE API version (a promise which does not seem too difficult to keep...)

Other APIs handle the unversioned endpoints in a similar way:

• CrossRef: "If you need to tie your implementation to a specific major version of the API, you can do so by using version-specific routes. The default route redirects to the most recent version of the API." All examples specify API endpoints without a version.

• Other APIs (DataCite, PDB, Zenodo(?), EuropePMC, ...) all behave in a similar way.

[sauliusg]

One suggestion from me in the discussion was to add something along these lines to the specification: _"The response to any request to the unversioned baseurl beyond the root MUST result in an HTTP 307 redirect to the same URL that was requested but prefixed by /v1, where 1 is to be replaced with the latest major version of the API that the implementation serves."

I have uneasy feeling that this ties implementers to one specific implementation. For example, why can't I just point my server's end-point to the most recent implementation (using RewriteRules or good old symlinks)?

[sauliusg]

Do we want to standardize how the API is served on the unversioned base url URL?

This would be very desirable.

If we cannot agree, then the "default" is that we release as v1.0.0 the specification that neither forbids, nor endorses, serving something at the unversioned URL.

Such arrangement would leave unversioned endpoints "orphaned", not belonging to the OPTIMADE API at all, even if they look like OPTIMADE API and probably everyone in the world will assume they are the part of the spec. Also, there will be a lot of confusion on how and what to implement on these endpoints. We have already talked about several possible policies: a) serve v1 on the unversioned endpoint always (Facebook way?); b) serve the latest stable API version on the unversioned endpoint (nearly everybody elses way). Inclusion of the recommendation on what to do on a MAY level would clarify situation a lot.

[rartino]

There is an important distinction between our case and the examples @sauliusg cites. Those APIs are provided by just by one entity, i.e, there is only one (or very few) implementers of the API.

OPTIMADE is more like an Internet communications protocol where we hope we will end up with many clients communicating with many servers. So, I think we should look more at version negotiation in protocols like, e.g., HTTP, XMPP, SSH:

• In HTTP it is mandatory that the client indicates the highest version number it supports with every request (see e.g., here). If no version number is transmitted, then the oldest version is assumed.
• XMPP outlines a version negotiation scheme that starts with a mandatory requirement for the client to indicate the highest version number it supports.
• SSH describes their protocol for version exchange in section 4.2 of rfc4253.

IMO, if we want to encourage serving the API on unversioned endpoints we need to state some other mandatory way for the client to supply a highest version number it supports, e.g., via a header so a reasonable version negotiation can take place.

So you could always query a default endpoint, e.g. (https://www.crystallography.net/cod/optimade/info), and get the reasonably machine readable version information. Here "reasonably machine readable" means that either your client can parse it, or it can detect that it can not parse it and report that fact to you as an error message. The only commitment from OPTIMADE spec. developers that is needed for this to work is to make sure that the same JSON key, "api_version", is never attached to anything else except the OPTIMADE API version (a promise which does not seem too difficult to keep...)

That commitment is only sufficient if you are ok with occasionally breaking automated clients against your database (and require them to be fixed by human intervention) when you upgrade the major version of the API. Maybe that is ok for you, but I don't want to build that assumption into the OPTIMADE protocol. The day we make a larger backwards-incompatible change, we will find larger providers hesitating to upgrade ("we have many clients querying us and we don't want to break them") and they will then likely get stuck serving the oldest version of the API on the unversioned endpoint.

If we consider a stronger commitment to make sure /info always returns the right information (so we never break older clients), then that means that for all future versions of OPTIMADE:

• The location of /info must remain the same.
• The default response format (at least for /info) must remain json.
• The overall organization of the json response with an outer data member, etc. must remain, following the JSON API v1.0 spec. (So if JSON API v2.0 comes along and reorganizes this, we cannot adopt that for our default response format.)
• The JSON key must be called api_version.
• The format of the value of that JSON key must remain the same.

This is a heavy commitment for the future.

why can't I just point my server's end-point to the most recent implementation (using RewriteRules or good old symlinks)?

A HTTP forward means the client sees which version it is being redirected to, and it can "back out" if it gets sent to a version newer than it knows that it supports. This provides some basic level of version negotiation.

[giovannipizzi]

I see both points, but here I would support @sauliusg position. By having to implement parsers for the PRs I did on the providers repo, I admit that having to try and guess the version is not very nice. It'd be better and more robust to try the unversioned endpoint, and get the version from there (and then possibly continue with the correct versioned URL).

And in the future, if I am a client at v2 and I also know v1, and I try against v3, I probably will not understand it and will just have to ignore it. But then I will have to ignore anything I don't understand. Instead, the probability that we can try to keep the /info endpoint more or less stable is high. And even if we cannot, a new parser (say for v3, where we might change any of the points mentioned by @rartino) will first try with the syntax of v3. If this fails, it will retry to check if it's v2 with the syntax of v2. And to have a more robust version negotiation [at least with the purpose to present a message to the user saying "I understood that the server is at version vXXX, but I don't speak the language of that version"], a client needs just to know how the /info endpoint changed over time, not the whole OPTIMADE specs. Or, for instance, a simple version negotiator could parse just the version, and then call the correct implementation for the client of that version.

In any case, you will have to try and test, but in this approach you only do 1 request to the unversioned one, and if it succeeds you can programmatically check what it is. Instead, with the current approach, you have to perform multiple requests, and ignore if they fail (and they could fail for any reason, as @sauliusg pointed out, and be transient - network error, server down or overloaded, ...). And most of these errors might be considered instead by the client as "this server does not serve this version" instead.

And finally, a "good" client would not really do requests to an unversioned endpoint, but to a versioned one (or, better: start with the unversioned one, check the /info, get the version, and then start asking things to the versioned one).

Finally, this is, to me, also mostly useful for people who want to query the API "by hand" (say, from a browser) and can start getting some results soon, rather than a lot of 404s before they either abandon, or go and read in detail the specs.

So to finalise: I would be supportive for a suggestion like the one of @sauliusg like "if an OPTIMADE endpoint is queried by a client, an implementation MAY serve the most recent stable version implemented by a provider." (possibly with some rewording, or an example to make sure it's clear what we mean).

[rartino]

@giovannipizzi Your discussion seem to focus mostly on the version negotiation protocol. And, I agree that we need to set up some systematic way for the client to get version info without having to iterate over endpoints. But, If you are happy with the version negotiation protocol you describe, wouldn't the HTTP forward also solve that? You would get forwarded to, e.g., /v3 and can try to read the /info there. That experience cannot be worse than trying to read /info from the unversioned endpoint if it serves the latest version (e.g., v3)? Or am I missing something?

Nevertheless, the larger issue to me is the future client-server compatibility problems we set up for ourselves.

You mention what a '"good" client' should do, but is it not obvious that if we endorse serving the present version of the API on the unversioned URL, then more or less every client written from now on will ignore any suggested means of version negotiation and will simply submit queries assuming the v1 protocol straight to that unversioned URL, e.g., /structures?filter=...?

Then, in 1-2 years when we see a need to make some backwards-breaking changes, you are going to look at the number of hits that materials cloud gets from clients that straight-off assumes the v1 protocol on your unversioned URL, and you are going to say: "Oh, wait, we cannot flip a switch that break all these clients overnight." And then we'll have to go through lengthy discussions about how to work around the backwards-breaking changes; and I'm sure someone will argue that we just should keep serving v1 on our unversioned URLs forever. Can we not try to avoid that situation before it happens?

[sauliusg]

You would get forwarded to, e.g., /v3 and can try to read the /info there. That experience cannot be worse than trying to read /info from the unversioned endpoint if it serves the latest version (e.g., v3)? Or am I missing something?

The whole point is that the redirection is no better than reading the unversioned endpoint directly. Ininstead of a redirection you could be served the data stream of whatever the /vx.y.z endpoint you are redirected to returns. You get exactly the same caveats, but need not bother with a (mandatory :/) redirection.

As for the redirection, it occurred to me that the redirection actually does not tell you the version. You should not try to guess the version form the (future!) endpoint name; in patucular, in the Location: /v12/info the string /v12/info is a resource name, and you should not assume that v12 is a protocol version, or that the URI contains version number at all. It can be Location: /latest/info...

[sauliusg]

To make long story short:

I see the only way to maximally (though not 100%) ensuring that no future client breaks in the future is to make the following META-specification:

Future specification of the OPTIMADE, even the higher major versions, are HIGHLY ENCOURAGED to retain the unversioned /info endpoint and a possibility to get a JSON stream with the version indication compatible with v1 of the OPTIMADE. In partucular, in a JSON response, attributes api_version and available_api_versions should be retained with the same names, the same value syntax and sematics, and preferably in the same JSON document position (aka xpath).

and then specify the following:

• OPTIMADE implementations SHOULD (S.G.: I would even opt for MUST, but I appreciate that some people may have done implementation in the other way and now have no time to change) provide an unversioned /info endpoint;

• the /info endpoint SHOULD (again: MUST?) provide means to request a specific response format, either through HTTP content negotiation, or through explicit request of the format, e.g. /info.json, /info.xml or /info?format=ASN.1

This is supposed to work in such way that when my client wakes up 100 years in the future, it sends an HTTP request to /info.json and gets the response in OPTMADE 10.12, which says that it supports v10, v9 and v2 at specific URIs, and the client then happily proceeds to use /v2 even though its clueless what breaking features v10 might have added since then... :)

PS. More detailed answers to the arguments are below but I appreciate that they are TL;DR :) PPS. /info.json might be preferred to /info?format=json since the former can be served from a static file without any further redirections in the server.

[sauliusg]

But, If you are happy with the version negotiation protocol you describe, wouldn't the HTTP forward also solve that? You would get forwarded to, e.g., /v3 and can try to read the /info there.

Yes, but if a client does not understand /v3/info, it will get as badly lost as reading /info abd getting the v3 stream from it...

[sauliusg]

Instead, the probability that we can try to keep the /info endpoint more or less stable is high.

I fully agree. We should try to keep /info stable, and there are few reasons why we should not do that.

@rartino mentioned in the online OPTIMADE Workshop (2020) that we have already changed /info layout several times. But we have at the moment v0.10.0 and not the v1.0 for a reason :). We were developing /info and experimenting; now that this is done we will freeze it for v1.0; and then we can commit ourselves to not making it backwards incompatible (for the version specs only!) without a very, very good reason.

[sauliusg]

If we consider a stronger commitment to make sure /info

I think we should :)

always returns the right information (so we never break older clients), then that means that for all future versions of OPTIMADE:

* The location of `/info` must remain the same.

True. "Cool URIs do not change", right? So we should strive for this anyway.

* The default response format (at least for `/info`) must remain json.

Not necessary. It can be content negotiated. The supplied format, which hopefully will be one of the self-describing ones, can be detected.

We could also think about having format-specific enpoints, e.g. /info.json

• if an implementation supports JSON at all, then supporting /info.json is not a big deal and wold probably work anyway, even of the main "language" of the /info becomes, say, XML;

• if an implementation does not support JSON at all, even to the extent not providing /info.json, then any old client that only can read JSON will not be able to do anything anyway; it must come to a conclusion that it can not interact with the given server and report an error;

• if a client supports several versions and formats, then, even if the default /info changes its format, the client can still work. Imagine (for the sake of the argument) that at some stage /info switches to XML. The old client (that, however, supports both JSON and XML responses) will first try to parse JSON. It will fail with a syntax error. Then it will try to parse the response as XML, will succeed an prceed extracting the supported versions. Now it knows to what extent it is compatible with the server

Or even better, the client will do something like file reposnse.dat, figure out the format and use a relevant parser. A parser factory could take care of this.

* The overall organization of the json response with an outer `data` member, etc. must remain, following the JSON API v1.0 spec. (So if JSON API v2.0 comes along and reorganizes this, we cannot adopt that for our default response format.)

I have mentioned multiple times, and repeat it once more that we should not make any commitments to JSON API unless we want to make our life miserable.

The /info response is not JSON API (JA) 1.0 or 2.0, it is OPTIMADE 1.0 or 2.0, and OPTIMADE has all powers to maintain compatibility even if JA2.0 does not. If we find it useful to switch to JA2.0, and JA2.0 is backwards incompatible with JA1.0, we can serve JA2.0-compatible reponses on other endpoints and keep OPTIMADE-specific, backwards compatible response on /info. I think we are under no obligation to promise any compatibility with any third party future standards; but it would be nice if we promise compatibility with ourselves :)

* The JSON key must be called `api_version`.

Why this should change?

* The format of the value of that JSON key must remain the same.

In the end, its a string. That will not change, most probably. If you want to add a dictionary for example, you can add it in a compatible way and retain the string, e.g.:

api_version: "2.1.13",
api_version_parsed: [2,1,12],
api_version_dict: {major: 2, minor: 1, patch: 12}

The only problem would be if semantics would change. E.g. if v2 of the OPTIMADE would mandate that v1.0.0 means something entirely different than the same string in v1 OPTIMADE. But that would be completely crazy and would compromise all the idea of versioning, let alone semantic versioning. So version number are supposed to stay "compatible" (in their meaning) across the major version changes :)

This is a heavy commitment for the future.

After the above considerations, it does not seem too heavy, does it?

[sauliusg]

In any case, you will have to try and test, but in this approach you only do 1 request to the unversioned one, and if it succeeds you can programmatically check what it is.

Indeed, this would be the main intention.

[sauliusg]

And finally, a "good" client would not really do requests to an unversioned endpoint, but to a versioned one (or, better: start with the unversioned one, check the /info, get the version, and then start asking things to the versioned one).

Absolutely yes!

[sauliusg]

Then, in 1-2 years when we see a need to make some backwards-breaking changes, you are going to look at the number of hits that materials cloud gets from clients that straight-off assumes the v1 protocol on your unversioned URL, and you are going to say: "Oh, wait, we cannot flip a switch that break all these clients overnight." And then we'll have to go through lengthy discussions about how to work around the backwards-breaking changes; and I'm sure someone will argue that we just should keep serving v1 on our unversioned URLs forever. Can we not try to avoid that situation before it happens?

So, let's thing ahead and make version detection features in the unversioned /info as stable as possible.

[sauliusg]

Finally, this is, to me, also mostly useful for people who want to query the API "by hand" (say, from a browser) and can start getting some results soon, rather than a lot of 404s before they either abandon, or go and read in detail the specs.

And even after they read the current spec, they will still need to go through all these 404s... Brrr... :)

How much easier it is to look at the /info and get at once the information on: a) what is the current version; b) what other versions are supported and how to access them.

[sauliusg]

There is an important distinction between our case and the examples @sauliusg cites. Those APIs are provided by just by one entity, i.e, there is only one (or very few) implementers of the API.

I'm afraid I do not see how this makes situation (for a client) any different...

OPTIMADE is more like an Internet communications protocol where we hope we will end up with many clients communicating with many servers. So, I think we should look more at version negotiation in protocols like, e.g., HTTP, XMPP, SSH:

Well, OPTIMADE is a REST API :). The above-suggested /info, I would say, provides what you mean here, doesn't it?

IMO, if we want to encourage serving the API on unversioned endpoints we need to state some other mandatory way for the client to supply a highest version number it supports,

Yes.

e.g., via a header so a reasonable version negotiation can take place.

Headers, as I have argued, do not work (or at least are no better than responses).

We need some meta-commitment to keep the minimal bits of /info stable, and we are done.

[rartino]

@sauliusg It is indeed a very good idea to agree on a forwards-stable way of communicating all major versions available from the unversioned URL. This is already far better than where we started this discussion, where it seemed the proposal was to just let clients deal with breakage.

You seem to propose: we commit to always provide a way to request a v1 formatted /info from the base URL. I'm not very opposed to that idea. I do feel that JSON API v1.0-embedded JSON under '/info` is a bit of an awkward, not-very-future-proof, format to standardize on for all future versions, for something as simple as communicating a list of integers representing version numbers. When our grandchildren implement OPTIMADE v10, they will scratch their heads over this arcane curly-brackety-format called "JAEOSON" that apparently was popular once, much as if someone asked me to support a version negotiation protocol based on the COBOL COMP-3 packed decimal format. But, sure, if this is what others want to commit to, I am OK with that.

But, again, this isn't really what I primarily was opposed to. What I am very opposed to is endorsing access to clients who have provided no versioning information to the present version of the API directly under the unversioned URL. Because that will lead us on the path to having to retain all of the v1 functionality (rather than just the /info response), under the unversioned URL in the future. Does anyone else see my point with this?

Finally, this is, to me, also mostly useful for people who want to query the API "by hand" (say, from a browser) and can start getting some results soon, rather than a lot of 404s before they either abandon, or go and read in detail the specs.

 And even after they read the current spec, they will still need to go through all these 404s... Brrr... :)

If we HTTP-redirect clients who have provided no version information to the latest version, then there will be no 404s.

A few links I was reading investigating our options, which I think shows this discussion is far from trivial:

• https://github.com/json-api/json-api/issues/406 - The issue at JSON API where people discuss how to version their APIs in a way that doesn't collide with the requirements of JSON API. They started in 2015 and have still not reached a consensus.
• https://www.troyhunt.com/your-api-versioning-is-wrong-which-is/ - A discussion of the person implementing the API for the "haveibeenpwned" database about why no option is ideal. Gives a good set of links at the end. He still ends up serving his v1 format "forever" to clients that provide no versioning information.

[rartino]

On the topic of the stability of available_api_versions. I am already plenty annoyed in my implementations that I have to provide full URLs, rather than have some way of indicating relative URLs, for the API versions. Because the full URL is sometimes not trivial to know when setting up the response. I had put off discussing this for a future version, but if you are now going to bind me to forever, for all eternity, providing the v1 format with full URLs, I guess we better have that discussion now :-/

As for the redirection, it occurred to me that the redirection actually does not tell you the version. You should not try to guess the version form the (future!) endpoint name; in patucular, in the Location: /v12/info the string /v12/info is a resource name, and you should not assume that v12 is a protocol version, or that the URI contains version number at all. It can be Location: /latest/info...

We are now discussing other stable ways of communicating version information that can be used alongside this forward. In that case this objection does not apply.

That said, if the only version negotiation we provide would be this forward, then it would have to come with the commitment that we retain versioned OPTIMADE URLs on the format /v<major version>/ so that clients can parse them. (You are probably going to say that this is a worse commitment than committing to a v1 formatted /info. Fine, lets just agree to disagree on that point, and keep the discussion on how we best communicate versioning info to the client from the unversioned endpoint and avoid clients being developed to assume v1 behavior on unversioned endpoints.)

[rartino]

So, given the possible scenario of wanting to alter the available_api_versions format. Would you possibly be ok with that we instead insert one more mandatory key in /info and let that be the first stop in version negotiation? Something like: available_major_api_versions that simply is a straight integer list of major version numbers, rather than URLs, e.g., [10, 5, 2, 1, 0]. A client that recognizes any of those major version numbers will then know how to proceed. E.g., for '1', the client knows to append /v1/ to the URL and continue by, e.g., reading /v1/info to get an available_api_versions field in a format it understand. (But it doesn't have to, it can just submit your query as /v1/structures/?filter=... since that endpoint will adhere to the v1 standard.)

If it was just up to me (which it obviously is not), I'd then rather setup a stable endpoint on the unversioned URL /versions to serve only that list, raw, in a JSON-compatible format, i.e. [10, 5, 2, 1, 0]. But I accept if people don't want the burden of maintaining an extra endpoint (even if it is ridiculously simple and for now just will be the static file with only the contents [1]), and prefer this to be stuffed in the /info endpoint with its (possibly soon to be legacy) JSON format.

[sauliusg]

So, given the possible scenario of wanting to alter the available_api_versions format. Would you possibly be ok with that we instead insert one more mandatory key in /info and let that be the first stop in version negotiation? Something like: available_major_api_versions that simply is a straight integer list of major version numbers, rather than URLs, e.g., [10, 5, 2, 1, 0].

That could be done but why would this be necessary? The current version string contains all information, including the major API version.

A client that recognizes any of those major version numbers will then know how to proceed. E.g., for '1', the client knows to append /v1/ to the URL and continue by, e.g., reading /v1/info

Wouldn't it be more straightforward to tie a particular URI to a particular version? And this is already done in available_api_versions, isn't it?

to get an available_api_versions field in a format it understand. (But it doesn't have to, it can just submit your query as /v1/structures/?filter=... since that endpoint will adhere to the v1 standard.)

Indeed, once you know the URI (from some sort of a bootstrap process) of the API version you understand, you can proceed with any queries and presumably it will work. It's just the first step that is tricky.

[sauliusg]

If it was just up to me (which it obviously is not), I'd then rather setup a stable endpoint on the unversioned URL /versions to serve only that list, raw, in a JSON-compatible format, i.e. [10, 5, 2, 1, 0]. But I accept if people don't want the burden of maintaining an extra endpoint (even if it is ridiculously simple and for now just will be the static file with only the contents [1]), and prefer this to be stuffed in the /info endpoint with its (possibly soon to be legacy) JSON format.

I'm actually not against the /versions endpoint, if others agree. Just in that case, since it must the simplest possible format and should not depend on JSON, I would suggest using CSV (as per RFC4180), in which case the version list would boil down to new-line separated integer list:

10
5
2
1
0

Which is the simplest thing you can ever imagine...

But even then I would tend to go and query '/optimade/structures' without bothering to read the current version... But I agree that here I am getting lazy. Indeed, what @rartino proposes here, seems to be a future-proof stable protocol:

• query optimade/versions, get the version you understand (assume its 2);

• go to optimade/v2/info and other endpoints, and do your queries there (for that day?)

But now: of course we do not want to make 2 queries instead of one each time we access OPTIMADE site (one query to get the version, another – to actually get the data). So the version needs to be cached. For how long? What if the supported version changes between the point I query the version and query the endpoint?

In this sense, querying optimade/structures gives you version and data in one (presumably atomic) query. If you can interpret it, then you are done. If not, then you try optimade/info and figure out the version you are compatible with.

[rartino]

So, given the possible scenario of wanting to alter the available_api_versions format. Would you possibly be ok with that we instead insert one more mandatory key in /info and let that be the first stop in version negotiation? Something like: available_major_api_versions that simply is a straight integer list of major version numbers, rather than URLs, e.g., [10, 5, 2, 1, 0].

That could be done but why would this be necessary? The current version string contains all information, including the major API version.

Your later reply suggests that you understood the point. I want a future-proof stable protocol to "bootstrap" version negotiation, as you outlined. We do not get that from the server version string (if you mean api_version), and I'm arguing that it seems risky to hinge it on available_api_versions as it has a more complex format and I am already thinking about possible changes (e.g., allow relative URLs).

I'm actually not against the /versions endpoint, if others agree. Just in that case, since it must the simplest possible format and should not depend on JSON, I would suggest using CSV (as per RFC4180),

I don't have a problem with that.

But now: of course we do not want to make 2 queries instead of one each time we access OPTIMADE site (one query to get the version, another – to actually get the data). So the version needs to be cached. For how long?

To be clear, this question applies to any scenario that leads a client to send a query to a /v<version> URL.

You can cache it for as long as you are happy with being limited to the feature set of the v2 protocol, and as long as the server supports it. The common way of handling it would probably be to cache it for a session, and in case of unexpected HTTP errors drop the cache and retry. I don't really see a problem.

Furthermore, if we make the commitment of never "overwriting" the meaning of /v<integer> URLs in the future, clients can be allowed to first try to submit their query to /v<integer> for their preferred version, and only if that fails, fall back on version negotiation via /versions.

What if the supported version changes between the point I query the version and query the endpoint?

It would look the same as if you have cache:ed the versions for a longer time and do a query after the supported versions change. There will be some breakage (404?) and the client has to interpret that breakage as possibly a dropped version, re-negotiate version, and retry.

I don't see how any protocol we've discussed avoids this, it will always be the case with these /v<version> endpoints that they could disappear because the server is dropping an older version. The same thing applies to queries directly on the unversioned URL, e.g., if an endpoint disappear in a version upgrade and a client tries to send its query to that endpoint. We could standardize a custom HTTP code for "API version no longer supported" if we are concerned about clients detecting this scenario correctly.

[giovannipizzi]

I think a simple version negotiation with a newline-separated list of integers under /versions is a good idea. Maybe I would make it a bit more future proof by saying it's really a csv, and that it can have multiple columns, and the first one is the version integer, the other ones (if present) are currently reserved for future extensions (imagine you might need to pass further information alongside the version in the future). And maybe we put a header, i.e. with this format

#version,newfield1,newfield2
1,AAA,BBB
2,CCC,DDD

Does it make sense?

Regarding the caching, I don't think it's really a problem. This is just caching of the URL and version under which the server is serving OPTIMADE. Caching for a session is under the vast majority of the circumstances OK, otherwise a length of, say, 1 day is also a good default. Honestly, I think one could even cache for much longer, say weeks - I would be surprised if a server stops serving a version abruptly. But in any case, a few hours is essentially always OK under all circumstances I think.

[rartino]

Great!, any format for a /versions endpoint that can be parsed in 5 lines of code is fine by me.

But, what do we do with the desire to serve queries on the unversioned URLs? I understand your preference to be able to "try out" a query without having to add /v1, and I absolutely see the point of enabling some form of version-agnostic permalinks to resource objects, especially if we consider the possibility to request self-documenting or human-readable response formats (response_format=html, anyone?).

Those two use cases appear to be solved by the HTTP 307 redirect suggestion (which says to repost with GET and POST intact.) The redirect is meant to signal to developers that they need to move to a versioned URL unless they know what they are doing.

[sauliusg]

Great!, any format for a /versions endpoint that can be parsed in 5 lines of code is fine by me.

Why so many? I't should be one line: my @versions = split(",") :)

[sauliusg]

#version,newfield1,newfield2
1,AAA,BBB
2,CCC,DDD

Does it make sense?

This could be done, and in fact the second field could be an URI serving the base if this version, or the URI of the corresponding versioned /info endpoint.

I was thinking in this direction, but just did not put the idea forward since it complicates the things compared to the one-column headerles list. Since the /version endpoint is something which should be so simple that we can be sure it is implemented in the future no matter what happens, a simple list one integer per line has some charm :)

But we could also return the following:

version,URI

1,https://example.com/optimade/v1 2,https://example.com/optimade/v1

[sauliusg]

But, what do we do with the desire to serve queries on the unversioned URLs? I understand your preference to be able to "try out" a query without having to add /v1, and I absolutely see the point of enabling some form of version-agnostic permalinks to resource objects, especially if we consider the possibility to request self-documenting or human-readable response formats (response_format=html, anyone?).

JSON or XML reposnses are also self-documenting and (reasonably) human-readable, so no problem even with the current OPTIMADE specs.

Those two use cases appear to be solved by the HTTP 307 redirect suggestion (which says to repost with GET and POST intact.) The redirect is meant to signal to developers that they need to move to a versioned URL unless they know what they are doing.

Redirects can be used for the solution, but I would insist that they are not mandatory, just one possible implementation. As said, I would prefer to use symlinks if my server environment permits :)

[sauliusg]

But, what do we do with the desire to serve queries on the unversioned URLs? I understand your preference to be able to "try out" a query without having to add /v1

I would be mostly in favour that the provider serves their most mature supported implementation on the unversioned endpoint. In this way we can have permanent links that survive even the major OPTIMADE release changes, which version endpoints will eventually not!

I would be not too much afraid of breaking too many clients. If we clearly state the policy, that is 'unversioned endpoint returns the newest implemented version', people will know and adapt. Clients will evolve together with the servers, wouldn't they? And, let's remember, there is always possibility to address the endpoint with specific version, if someone wants to be 100% sure their (old) client works. Until the server stops supporting the /v1 endpoint, that is :). So, the only way to ensure uninterrupted presence of the OPTIMADE is not a versioned endpoint, but an unversioned endpoint with a clearly specified version and maximally possible backwards compatibility :).

[rartino]

I would be not too much afraid of breaking too many clients. If we clearly state the policy, that is 'unversioned endpoint returns the newest implemented version', people will know and adapt. Clients will evolve together with the servers, wouldn't they? And, let's remember, there is always possibility to address the endpoint with specific version, if someone wants to be 100% sure their (old) client works.

From a non-trivial amount of experience of helping others using APIs, I'd say the vast, vast, number of clients are going to look like this (in the user's favorite language):

data=fetch_url('https://optimade.example.com/structures/?filter=elements HAS "Al"')
data=json_parser(data)
coordinates=data['data']['attributes']['cartesian_site_positions']
...

Look: no consideration of "policy", no version negotiation, no attempt to validate the server version, or that the output format matches any expectation. This user likely isn't even aware which API version they use, nor that there exist versioned endpoints. They've just spent the absolute minimal effort to get what they want, because they have other things to do. However, they will return to this script in 1-2 years, pass it to others, etc., and expect it to still work.

If we endorse hosting the latest version of OPTIMADE on the unversioned endpoint, these are the "clients" we break. And there will be many of them.

So, why do we care? These users don't read documentation, and don't follow best practices, so they get what's coming to them, right? But, there are going to be a lot of those users, and this breakage is going to end up the defining user experience of OPTIMADE. Users are going to say things like: "Yeah, I tried to use OPTIMADE, but I don't like it. It is unreliable. My scripts stopped working."

This is what I am worried about.

The discussion we are having is about committing to one of two possible strategies for future evolvement of OPTIMADE. One is represented by what you say here:

So, the only way to ensure uninterrupted presence of the OPTIMADE is not a versioned endpoint, but an unversioned endpoint with a clearly specified version and maximally possible backwards compatibility :).

In this strategy ("A") all future discussions about new features, even across major versions, have to take into account how they break old clients. E.g., lets not remove endpoint A even though it no longer serves a purpose, because that would break clients. Lets not change the format of property b, because it would be backwards-breaking, etc., etc. We will end up with a lot of deprecated, legacy, "cruft" in the API.

The alternative strategy ("B") is to not allow clients access without a version number. We can then evolve the specification with much less concern about backwards-breaking changes. We can continue to evolve v1 in point releases, while at the same time working on point releases of a completely revamped protocol v2, and there wouldn't have to be much concern at all about how swapping out v1 for v2 breaks old clients.

If OPTIMADE was an already time-tested mature protocol where we did not expect to have to do any backward-breaking changes, I'd go with strategy A. But I don't think that is where we presently are.

[rartino]

I absolutely see the point of enabling some form of version-agnostic permalinks to resource objects, especially if we consider the possibility to request self-documenting or human-readable response formats (response_format=html, anyone?).

JSON or XML reposnses are also self-documenting and (reasonably) human-readable, so no problem even with the current OPTIMADE specs.

I don't think this is very important for the central topic, but I cannot help but comment.

XML can absolutely be self-documenting with a carefully designed schema.

However, I'd argue JSON cannot, because there is no designated place in JSON to define (or link) a schema, nor a mandated format for that schema. Hence, a client has to refer to external documentation to understand where to find the schema, and to know which format it has.

[sauliusg]

From all what was said, it seems that we all agree on the following policy:

An OPTIMADE implementation MUST implement an unversioned http://example.com/optimade/versions; this endpoint MUST return a list of all major OPTIMADE version supported by this implementatio (in a CSV == one-integer-per-line format discussed above);

E.g.:

curl http://example.com/optimade/versions
10
9
3
2

For each major version mentioned in the result of the /version endpoint, a corresponding supported OPTIMADE endpoint set MUST exist, where the version number with the v letter is attached to the base URI, e.g. (assuming the base URI is http://example.com/optimade/):

http://example.com/optimade/v2/info
http://example.com/optimade/v10/info
http://example.com/optimade/v10/structures

In this way, clients can learn which versions are supported by the queried database (implementation), and unambiguously find the endpoint locations. All queries are as future-proof as possible; we depend only on a) a list of integers in the /versions response, as a plain new-line delimited text and b) the structure of URIs and the ability to append /vX to the base URI for each major version X. Each client then queries an endpoint which it for sure can understand.

The next part is more controversial, but hopefully we can agree on it or on some similar setup:

The only (big, IMHO) drawback of the above versioned schema is that, while it is workable and rather simple for "active" clients (scripts), it does not permit permanent URIs in a text. For example, if I publish a http://example.com/optimade/v1/structures/2000000 link in my paper, it will stop working as soon as the database drops support for v1, although the same resource could be perfectly represented by a v2 or v3 or even v10 response.

Thus, I suggest the following feature:

An OPTIMADE implementation MAY provide unversioned endpoints to access its resources:

http://example.com/optimade/info
http://example.com/optimade/structures

If the implementations chooses to provide the unversioned endpoints, it SHOULD provide the highest most mature and stable API version on these endpoints. In that case, the implementation MUST list the major version of this API (i.e. the API provided on the unversioned endpoints) as the first number in the response of the http://example.com/optimade/versions query. For example, if the implementation example.com offers API versions 10,9,3 and 2, and 9 is the preferred (most stable) one served on the unversioned enpoints, then the response of the /versions query MUST be:

curl http://example.com/optimade/versions
9
10
3
2

I.e. 9 is the first (top-most) number on the list (the order of the other numbers is unimportant/decreasing). The unversioned endpoints MAY use HTTP redirections to direct the client to the corresponding versioned API, or MAY return data directly.

In this way clients have an unambiguous and future-proof way to determine which version is served on the unversioned endpoints. Obviously, the same version MUST be server on all unversioned endpoints :)

Since unversioned endpoints are a 'MAY' feature, those implementers who do not benefit of having it can safely skip it and list /versions in whatever order they want. I would still implement unversioned endpoints for the COD, since

a) it allows easy OPTIMADE data citations;

b) it will allow us a smooth transition from the older non-versioned OPTIMADE implementation to the /v1 implementation when v1 version of the spec is out (we do not need to change out working setup, just update the server and add /v1 routes).

[rartino]

From all what was said, it seems that we all agree on the following policy: [...]

The only question I see remaining from upthread is if we should mandate that the response MUST be parsed as a CSV file that may contain multiple columns, or if the one version number per line is allowed to be assumed.

I also like the simplicity of one version number per line, but I also get the point of allowing for extra parameters. I would argue against using it for URIs, because (as shown in @sauliusg outline) the client should know what to do already, and to include them adds to the complexity of handling this endpoint in implementations (since for some it will be dynamic information). However, if we in the future adopt the feature sets of #91, flags for those could be useful in this file. Then a client will be able to make an informed choice already from /versions on whether the highest version it recognizes comes with support for the feature set it needs, or if needs to back down to an older version for that.

(That said, I am slightly worried that if we start with a single column and just one header, we will break clients the day we introduce those even if we tell implementers that they need to be prepared for them.)

Your suggestion about connecting the serving on the unversioned URL to the order of this file seems an improvement, I have to think a bit about it. I will say that your desire for permanent URIs in text would be perfectly handled by the HTTP redirect.

[sauliusg]

I will say that your desire for permanent URIs in text would be perfectly handled by the HTTP redirect.

I am quite unhappy about HTTP redirect mandate because it limits my implementation. Currently we would not use redirects at this point. IMHO redirects add nothing to what is already provided by /versions. Thus I would strongly argue that redirections should be either a 'MAY' feature, or altogether just implementation examples.

[sauliusg]

Your suggestion about connecting the serving on the unversioned URL to the order of this file seems an improvement, I have to think a bit about it.

Please have a look at it. In general, tying things to list order is dangerous, since at some stage a different ordering requirement will come and will be in conflict with the old one. However at the moment there does not seem such situation possible.

An alternative would be to mark the main version somehow, e.g.:

10,
9,*
3,
2,

Bu this would be even more ad-hoc and conflict-prone, probably...

[sauliusg]

I also like the simplicity of one version number per line, but I also get the point of allowing for extra parameters. I would argue against using it for URIs, because (as shown in @sauliusg outline) the client should know what to do already, and to include them adds to the complexity of handling this endpoint in implementations (since for some it will be dynamic information).

I agree. In addition to that, we can face contradicting information in spec and in /versions. Unless we really want the complexity of custom URIs per version, we should not mandate them. Moreover, "strange" URIs can be handled via redirection (here they become a handy implementation feature); e.g. https://example.com/optimade/v1/structures might redirect to https://v1.optimade.powerful.farm.example.com/optimade/structures :)

However, if we in the future adopt the feature sets of #91, flags for those could be useful in this file. Then a client will be able to make an informed choice already from /versions on whether the highest version it recognizes comes with support for the feature set it needs, or if needs to back down to an older version for that.

I agree. This would be a useful application.

(That said, I am slightly worried that if we start with a single column and just one header, we will break clients the day we introduce those even if we tell implementers that they need to be prepared for them.

I would say:

• you can not protect against every sort of stupidity people will do...

• if we mandate that:

-- /versions response is CSV (a-la RFC4180) with headers;

-- the first column is always a version number;

then even in the future when we add columns with different semantics (further columns are identified by their names, not by their positions!), then even the "dumb" clients that take just the first column will not break in the future.

[rartino]

Your suggestion about connecting the serving on the unversioned URL to the order of this file seems an improvement, I have to think a bit about it.

 Please have a look at it. In general, tying things to list order is dangerous, since at some stage a different ordering requirement will come and will be in conflict with the old one.

I actually like the ordering idea even outside the context of serving the unversioned endpoint, because it makes it very easy to get clients to do the thing that the server wants; i.e., go trough the list in order and use the first version the client recognizes, unless it has a specific reason to look for a higher version.

For example, if I publish a http://example.com/optimade/v1/structures/2000000 link in my paper, it will stop working as soon as the database drops support for v1, although the same resource could be perfectly represented by a v2 or v3 or even v10 response.

Thus, I suggest the following feature:

An OPTIMADE implementation MAY provide unversioned endpoints to access its resources: [...] If the implementations chooses to provide the unversioned endpoints, it SHOULD provide the highest most mature and stable API version on these endpoints. In that case, the implementation MUST list the major version of this API (i.e. the API provided on the unversioned endpoints) as the first number in the response of the http://example.com/optimade/versions query.

IMO this is a better design than we started out with. But I have two thoughts:

First, this may not be a showstopper, but we will thus end up with providers serving different versions of the API on their unversioned endpoints. I think this will be a gotcha for new users and give a first experience that goes against the main point of OPTIMADE as a unified API where the same query can be submitted the same way to multiple providers.

Second, isn't the use case for stable URIs not entirely to provide stable links to single resource objects? For example, I agree that it makes a lot of sense to write in a paper:

A permanent OPTIMADE link to this structure is: https://example.com/optimade/structures/2000000

I am far less convinced that it makes sense to write the following:

You can make this query using the OPTIMADE API as: https://example.com/optimade/structures?filter=elements HAS Na AND elements HAS Cl AND nelements=2

because this may soon be outdated and unhelpful ("Why do I get an error? The paper says it should work!"). I'd argue that in this case the following is a much better idea:

You can make this query using version 1 of the OPTIMADE API as: https://example.com/optimade/v1/structures?filter=elements HAS Na AND elements HAS Cl AND nelements=2 . If version 1 is no longer supported by the database, refer to https://optimade.org for up to date information on how to formulate the query in a more recent version.

So maybe what would make sense to standardize on is to only serve single resource objects on the unversioned endpoint to allow for permanent links, but not allow queries (or other "interactions" with the API), and make a soft commitment to preserve the structure of /<type>/<id> for these endpoints, even for types that are discontinued in future versions.

But, to bring the discussion back to be constructive. I am still convinced that it is a really bad idea to serve queries on the unversioned URL. However, I also recognize that as long as this is expressed as a MAY, at least I don't have to do it. Hence, the extent of trouble I foresee this causing me personally is:

1. Those of you who decide to go for this are going to represent a conservative voice when we try to further evolve the API.
2. To the extent we anyway push backward-breaking changes, we risk loosing a bit of goodwill for the OPTIMADE API from users who gets confused / who write the clients I mentioned upthread, which could have been prevented.

I suppose I have to live with that; unless more people show up in this thread to back me up.

[rartino]

@sauliusg, @giovannipizzi I've made a PR that tries to accommodate the ideas in this discussion.

Edit: it is #290

[sauliusg]

First, this may not be a showstopper, but we will thus end up with providers serving different versions of the API on their unversioned endpoints. I think this will be a gotcha for new users and give a first experience that goes against the main point of OPTIMADE as a unified API where the same query can be submitted the same way to multiple providers.

I don't see this as a showstopper. I do not see this even as a slightest problem worth considering. Changing things are inseparable feature of the Internet. By now everyone who uses Internet more for than a month should realise that the things change.

[sauliusg]

Second, isn't the use case for stable URIs not entirely to provide stable links to single resource objects? For example, I agree that it makes a lot of sense to write in a paper:

A permanent OPTIMADE link to this structure is: https://example.com/optimade/structures/2000000

Exactly, that is what I meant! So we agree that unversioned pointers to resources are valuable ans should be a MAY feature?

[sauliusg]

But, to bring the discussion back to be constructive. I am still convinced that it is a really bad idea to serve queries on the unversioned URL. However, I also recognize that as long as this is expressed as a MAY, at least I don't have to do it. Hence, the extent of trouble I foresee this causing me personally is:

Yes indeed, this is a bad idea – if we change the queries in the future so that /v2 is totally incompatible with /v1.

But I am also convinced that it is a really bad idea to cite queries on versioned endpoints, because these will go away when the version is no longer supported.

An it is an even worse idea not to serve queries on unversioned endpoints, let alone to forbid anybody to do, since this will make the system bizarrely unorthogonal for no good reason, and the users, novices and experienced alike, will start asking "what those guys are thinking up there"?

So we have three evils and have to choose among them. For me allowing full service on unversioned endpoins with clear full documentation how this is supposed to work is the least of evils. If the policy is clear the users will find it our, and if they don't then we can't do anything more about this.

[rartino]

So we have three evils and have to choose among them.

So, please take a look at PR #290 where the intent is that implementers get to choose their poison.

[sauliusg]

You can make this query using the OPTIMADE API as: https://example.com/optimade/structures?filter=elements HAS Na AND elements HAS Cl AND nelements=2

because this may soon be outdated and unhelpful ("Why do I get an error? The paper says it should work!"). I'd argue that in this case the following is a much better idea:

You can make this query using version 1 of the OPTIMADE API as: https://example.com/optimade/v1/structures?filter=elements HAS Na AND elements HAS Cl AND nelements=2 . If version 1 is no longer supported by the database, refer to https://optimade.org for up to date information on how to formulate the query in a more recent version.

Important note: the same argument as you put forward for unversioned endpoints applied to the versioned endpoints: as soon as the /v1 endpoint goes away, https://example.com/optimade/v1/structures?filter=elements HAS Na AND elements HAS Cl AND nelements=2 stops working ("Why do I get an error? The paper says it should work!"). And, in contrary to the unversioned endpoint from which you do get, in a worst case, an answer with an appropriate error message which a human (or advanced AI :) can understand, from the versioned endpoint we'll get only the infamous '404 Not found' message, and the user is left clueless (RTFM, of course, but how much more work it is!)

[sauliusg]

TL;DR

The comment below opens a discussion about the future issue (past v1); since the issue is strategically important and closely related to what was said in this Issue, I feel it necessary to provide some links before we possibly move to another one.

I am far less convinced that it makes sense to write the following:

You can make this query using the OPTIMADE API as: https://example.com/optimade/structures?filter=elements HAS Na AND elements HAS Cl AND nelements=2

because this may soon be outdated and unhelpful ("Why do I get an error? The paper says it should work!"). I'd argue that in this case the following is a much better idea:

This is a general problem to provide reproducible queries for reproducible science. The problem turns out slightly more complicated that we would like it to have. RDA has several recommendations how reproducible queries can be achieved:

• Identification of Reproducible Subsets for Data Citation, Sharing and Re-Use

• Data Citation of Evolving Data: Recommendations of the Working Group on Data Citation (WGDC)

My understanding is that a universal, future proof way to identify queries would be to assign unique identifiers to queries. In OPTIMADE, we could have a special endpoint /queries:

https://https://example.com/optimade/queries/fa36ca8e-b06a-11ea-a512-13c835dbd535
https://https://example.com/optimade/v1/queries/fa36ca8e-b06a-11ea-a512-13c835dbd535
https://https://example.com/optimade/v12/queries/fa36ca8e-b06a-11ea-a512-13c835dbd535

That would correspond to exactly the same query that you mentioned, https://example.com/optimade/structures?filter=elements+HAS+"Na"+AND+elements+HAS+"Cl"+AND+nelements=2, performed on a specific revision of a database. If you GET data from the https://https://example.com/optimade/queries/structures/fa36ca8e-b06a-11ea-a512-13c835dbd535, you will always get the same data as in the original request (possibly in a different representation), and if you query https://https://example.com/optimade/v1/queries/fa36ca8e-b06a-11ea-a512-13c835dbd535, you would get the same same data and the same representation, but only for as long as the /v1 endpoint is supported. I.e. the usual policy of versioned/unversioned enpoints would apply.

The query would of course store as query metadata the filter string (or whatever other request that was submitted), the original query URI, a database version, and query date.

Implementation might choose from different implementations:

• if /v2 will have the same query language, it can simply replay the same filter query on a new database, adding the database revision requirement;

• if the filter language changes beyond recognition, the database may choose to translate the original filters into new filters and replay these new filter on the current implementation;

• if translating filters is not feasible, the database can choose to simply cache record identifiers (or ranges of identifiers) selected by the query, and return just these records again;

• finally the database can simply store the returned answer and replay it when the query is addressed later on.

The ID of the query would be returned in the response from the database.

In and interesting self reference twist, we can now make (reproducible!) queries about queries!. E.g. we can ask 'which queries last year requested a band-gap property of the Si-containing material' and the like. These would also get their UUIDs and become reproducible :)

Of course no database is under obligation to keep all queries forever, a retention policy will probably has to be developed for each database (e.g queries that are requested more often are kept for longer). Resources have a right to vanish.

[sauliusg]

am far less convinced that it makes sense to write the following:

You can make this query using the OPTIMADE API as: https://example.com/optimade/structures?filter=elements HAS Na AND elements HAS Cl AND nelements=2

What about: https://example.com/optimade/structures?filter=elements+HAS+"Na"+AND+elements+HAS+"Cl"+AND+nelements=2&requested_api_version=v1

O a shorter version: https://example.com/optimade/structures?filter=elements+HAS+"Na"+AND+elements+HAS+"Cl"+AND+nelements=2&api=v1

BTW: Is actually the QS considered to be the part of the resource name? I have impression that in this example 'https://example.com/optimade/structures' is a resource name and querying it will produce some resource representation; The rest ?filter=elements+HAS+"Na"+AND+elements+HAS+"Cl"+AND+nelements=2&api=v1 is a modifier that influences how the resource is represented (in this case, selects some structures, not all).

The (https://restfulapi.net/resource-naming/) is open to interpretation on this point.

[rartino]

You can make this query using the OPTIMADE API as: https://example.com/optimade/structures?filter=elements HAS Na AND elements HAS Cl AND nelements=2

because this may soon be outdated and unhelpful ("Why do I get an error? The paper says it should work!"). I'd argue that in this case the following is a much better idea:

You can make this query using version 1 of the OPTIMADE API as: https://example.com/optimade/v1/structures?filter=elements HAS Na AND elements HAS Cl AND nelements=2 . If version 1 is no longer supported by the database, refer to https://optimade.org for up to date information on how to formulate the query in a more recent version.

Important note: the same argument as you put forward for unversioned endpoints applied to the versioned endpoints: as soon as the /v1 endpoint goes away, https://example.com/optimade/v1/structures?filter=elements HAS Na AND elements HAS Cl AND nelements=2 stops working ("Why do I get an error? The paper says it should work!").

The difference is that in my version of the text it says right there, in the text, just after the query, why the query may not work perpetually, and what you are supposed to do if it does not. Sure, you can formulate a similar addendum also for the unversioned query, but if you try to cover most eventualities possible for such URLs, it gets tricky. If I had to serve unversioned queries that use more advanced features of the API that may change in a backwards-breaking ways, and I had to cite them in printed papers, this is what I would come up with:

This query is formulated using version 1 of the API. If the server upgrades the version it serves on this URL, the query may break or the behavior may change in some other way. Please refer to https://optimade.org to see what have changed since version 1 in the version that is served. (You may be able to see the version of a served response in its meta -> api_version field, if present.)

My other text that handles the same thing on the versioned endpoint seems much more clear. If you think I exaggerate the wordiness here compared to what I wrote for the versioned endpoint, feel free to show how you intend to formulate this in your own papers.

And, in contrary to the unversioned endpoint from which you do get, in a worst case, an answer with an appropriate error message which a human (or advanced AI :) can understand,

No, the worst case is that you get a slightly altered behavior that leaves you thinking that the query worked as intended, but with e.g., entries missing.

You may interject "but we will never change the behavior like that!". Sure, we can try to avoid it. But it is not always super obvious how interconnected features interact when they are changed in backwards-breaking ways (e.g, the structure of fields), and something like this could slip through the cracks unintentionally. It will be your fight, every time we evolve the API to a new major version, to make sure to avoid any such subtle breakage for the examples published in your papers.

from the versioned endpoint we'll get only the infamous '402 Not found' message, and the user is left clueless (RTFM, of course, but how much more work it is!)

Actually, it is completely trivial for the server to detect that the client is trying to access a versioned URL that is no longer is available. Hence, a very user-friendly error message can be provided that explains that access to version 1 of the API is no longer supported with links that explain to an end-user what to do. We can even define a specific HTTP error code for "version not available" so also automated clients can distinguish this situation from other errors and act accordingly. I'd say this is a major advantage of having static examples point to versioned endpoints over trying to handle old requests on the unversioned endpoint, where you have no way to tell what version of the API a client is trying to interact with.

What about: https://example.com/optimade/structures?filter=elements+HAS+"Na"+AND+elements+HAS+"Cl"+AND+nelements=2&requested_api_version=v1

What are you trying to achieve that isn't covered by the versioned URL?: https://example.com/optimade/v1/structures?filter=elements+HAS+"Na"+AND+elements+HAS+"Cl"+AND+nelements=2 That it looks more like a static resource? You still need to figure out what to do if the server doesn't serve the version requested.

Is actually the QS considered to be the part of the resource name?

I suppose we can say that it technically isn't. But if the QS severely alters the response, it could just as well be. The model of 'resources' provided via a REST API breaks down a bit when you use more sophisticated API functionality, and the situation becomes more similar to a general communications protocol. (Where, typically, some form of version negotiation is mandatory when there is a risk of clients and servers of different versions interacting.)

[sauliusg]

I'd say this is a major advantage of having static examples point to versioned endpoints over trying to handle old requests on the unversioned endpoint, where you have no way to tell what version of the API a client is trying to interact with.

So add &api=v1 to the QS.

[sauliusg]

Actually, it is completely trivial for the server to detect that the client is trying to access a versioned URL that is no longer is available. Hence, a very user-friendly error message can be provided that explains that access to version 1

Yes, but this means that I need to bother about all previously existed versions (what you suggest is even worse – I need to bother about version which I never implemented, if we want to make the behaviour consistent).

When I mean /v1 is no longer supported I mean nobody on the server side eve cares or even is aware that it ever existed :)

[sauliusg]

The difference is that in my version of the text it says right there, in the text, just after the query, why the query may not work perpetually

In the text – where? In the documentation? What prevents you from putting a similar disclaimer next to unversioned endpoint specs, with exactly the same effect?

[sauliusg]

It will be your fight, every time we evolve the API to a new major version, to make sure to avoid any such subtle breakage for the examples published in your papers.

This can only be solved with citable queries that have unique resource IDs, as mentioned above.