026-Support resource management (structure and data)

[dosse]

Please add clear specifications for usage of HTTP verbs POST, PUT, DELETE, PATCH with restful SDMX API to allow managing artefacts and data in a coherent manner. An interesting article (in French) is here: http://blog.xebia.fr/2014/03/17/post-vs-put-la-confusion/

[achronis]

For the record this is what we added to Eurostat's SDMX RI Web Service (currently under development)

Action Behavior

Requested Action	REST Action	Exists	Is Final	Is used	Dependencies exist or provided	Result
Append	POST	No	-	-	Yes	Success - insert artefact
Append	POST	No	-	-	No	No Results - include missing references in response
Append	POST	Yes	*	*	*	Forbidden (Semantic Error)
Replace	PUT	No	*	*	*	No results
Replace	PUT	Yes	No	No	Yes	Success - Delete and Insert artefact
Replace	PUT	Yes	No	Yes	*	Conflict
Replace	PUT	Yes	No	No	No (Corner case)	No Results - include missing references in response
Replace	PUT	Yes	Yes	*	*	Success - Update non final attributes; else Forbidden
Delete	DELETE	No	*	*	*	No Results
Delete	DELETE	Yes	No	No	*	Success
Delete	DELETE	Yes	Yes	No	*	Success
Delete	DELETE	Yes	*	Yes	*	Conflict

Error messages

Currently the SDMX guidelines provide SDMX codes and the corresponding HTTP Status Codes, see the corresponding GitHub page

None of the codes can describe the following:

1. The artefact cannot be deleted because it used by other artefacts or non SDMX application specific entities.
2. The artefact cannot be append/replaced because it already exists.

One proposed solution would be to add a new client side SDMX code for resource conflicts, e.g. 160.The corresponding HTTP status code would be the existing HTTP Status code 409 Conflict.

Error	SDMX Error	HTTP Status Code	Notes
Delete/Replacing artefact that doesn't exist	100 No results found	404 Not found
Delete artefact that is referenced	160 Conflict	409 Conflict	Code not in SDMX standard
Replacing non final artefact that is referenced	160 Conflict	409 Conflict	Code not in SDMX standard
Append existing artefact	150 Semantic Error	403 Forbidden
Authorization errors	110 Unauthorized	401 Unauthorized
Unpredictable errors	500 Internal Server error	500 Internal Server error	Bugs, DB related errors
Request valid but no updates made	150 Semantic Error	403 Forbidden
A combination of success/error results	Included in each message	207 MultiStatus

REST

The methods and resources used in SDMX RI WS are the following. The resource paths are relevant to the SDMX RI REST base path.

HTTP Method	Resource	Description
POST	/ (might change to /structure)	Append only new artefacts
PUT	/ (might change to /structure)	Final artefacts replace non-final information, Non-Final artefacts replace completely
DELETE	/{structureType}/{agencyId}/{maintainableId}/{version}	Delete the specific artefact if it is not used

Below is an example request and error response.

The request to delete a DSD:

DELETE http://nsiwsurl/datastructure/TEST_AGENCY/STS/2.0 HTTP/1.1

HTTP/1.1 409 CONFLICT

Content-type: application/xml

The error response:

<?xml version="1.0" encoding="utf-8"?>
<message:Error xmlns:common="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/common" xmlns:message="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/message">
    <message:ErrorMessage code="160">
        <!--Conflict-->
        <common:Text xml:lang="en">urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=TEST_AGENCY:SSTSCONS_PROD_A(2.0)</common:Text>
    </message:ErrorMessage>
    <message:ErrorMessage code="160">
        <!--Conflict-->
        <common:Text xml:lang="en">Failure:  Action Delete for Data Structure Definition TEST_AGENCY:STS (v2.0) cannot be performed. REASON: Cannot replace artefact. Non-Final artefact is used by other artefacts</common:Text>
    </message:ErrorMessage>
</message:Error>

Multi Status

Since a structure submission may contain multiple artefact it is possible to have different results per artefact. For example when submitting a structure file containing a DSD with it's dependencies and one or more Codelist already exist. When multiple structural metadata is submitted and/or deleted, and only for some artefacts there was a failure while other actions were successful.

The HTTP Code will be 207 Multi-Status. Only the failures will be recorded. If an artefact is not referenced then it didn't fail.

In general multi-status will appear only if one of the following conditions are met:

1. Two or more errors with different error codes.
2. For one request, at least one operation succeeded but at least one other operation failed.

[Tzaphkiel]

I believe that one should refer as much as possible to existing specifications, and in this case, the HTTP/1.1 RFC section 9.5, 9.6, 9.7. Here is a more readable explanation

For the less used PATCH method, one has to refer to the previous versions on this specification: RFC 2068.

[dosse]

@Tzaphkiel

I believe that one should refer as much as possible to existing specifications, and in this case, the HTTP/1.1 RFC section 9.5, 9.6, 9.7. Here is a more readable explanation

For the less used PATCH method, one has to refer to the previous versions on this specification: RFC 2068.

A more recent HTTP specification is here: https://tools.ietf.org/html/rfc7231

The conclusions "Generally, in practice, always use PUT for UPDATE operations." and "Always use POST for CREATE operations." in https://restfulapi.net/rest-put-vs-post/ are incorrect and miss-leading. Rather than using weak interpretations, better would be to refer only to the original standard:

• https://tools.ietf.org/html/rfc7231#section-4.3.4: “The PUT method requests that the state of the target resource be created or replaced with the state defined by the representation enclosed in the request message payload. A successful PUT of a given representation would suggest that a subsequent GET on that same target resource will result in an equivalent representation being sent in a 200 (OK) response.”

• https://tools.ietf.org/html/rfc7231#section-4.3.3: “The POST method requests that the target resource process the representation enclosed in the request according to the resource's own specific semantics. For example, POST is used for the following functions (among others): o Providing a block of data, such as the fields entered into an HTML form, to a data-handling process; o Posting a message to a bulletin board, newsgroup, mailing list, blog, or similar group of articles; o Creating a new resource that has yet to be identified by the origin server; and o Appending data to a resource's existing representation(s).”

  “The fundamental difference between the POST and PUT methods is highlighted by the different intent for the enclosed representation. The target resource in a POST request is intended to handle the enclosed representation according to the resource's own semantics, whereas the enclosed representation in a PUT request is defined as replacing the state of the target resource. Hence, the intent of PUT is idempotent and visible to intermediaries, even though the exact effect is only known by the origin server.”

--> Therefore, POST and PUT allow both for CREATE and UPDATE operations. But the usage is different. With PUT only one single artefact can be uploaded (created or replaced) specifying the appropriate target resource (target URL of the artefact). POST allows us efficiently uploading multi-artefact structure messages to create or update (whatever is most appropriate according to the current server status) the contained artefacts.

E.g. a typical use case is structure synchronisation between two SDMX web services. First you do a GET with "references=all" on the 1st service and then you can POST the multi-artefact structure message to the 2nd service, which will create the artefacts that do not exist yet and replace the ones that exist already (if allowed). I would not like to first split the message into multiple messages and then figure out myself for which message I have to use PUT and for which POST. This would become a nightmare.