Support deep objects for query parameters with deepObject style

[bajtos]

Background

Many applications expect deeply nested objects in input parameters, see the discussion in swagger-ui starting from this comment: https://github.com/swagger-api/swagger-ui/issues/4064#issuecomment-357417513 In LoopBack, we are running into this problem too, see https://github.com/strongloop/loopback-next/pull/1679.

Consider a filter parameter defined as follows:

parameters:
 filterParam:
   in: query
   name: filter
   schema:
     type: object
   style: deepObject
   explode: true
   description: Options for filtering the results
   required: false

Let's say the user wants to provide the following value, for example by entering a JSON into the text area rendered by swagger-ui:

{
  "name": "ivan",
  "birth-date": {
    "gte": "1970-01-01"
  }
}

At the moment, the OpenAPI Specification v3 does not describe how to encode such value in a query string. As a result, OpenAPI clients like swagger-ui don't know how to handle this input - see the discussion in https://github.com/swagger-api/swagger-js/issues/1385 for an example.

Proposal

The following query string should be created by swagger-js client for the input value shown above.

filter[name]=ivan&filter[birth-date][qte]=1970-01-01

The proposed serialization style is supported by https://www.npmjs.com/package/qs, which is used by http://expressjs.com as the default query parser, which means that a vast number of Node.js API servers are already expecting this serialization style.

I am not sure if there is any existing formal specification of this format, I am happy to look into that once my proposal gets considered as acceptable in principle.

Additional information

Existing issues that are touching a similar topic:

• style: form + object looses the parameter name? OAI/OpenAPI-Specification#1006
• Are indexes in the query parameter array representable? OAI/OpenAPI-Specification#1501

Two older comments from swagger-js that may be relevant:

https://github.com/swagger-api/swagger-js/pull/1140

Limitations: deepObject does not handle nested objects. The specification and swagger.io documentation does not provide an example for serializing deep objects. Flat objects will be serialized into the deepObject style just fine.

https://github.com/swagger-api/swagger-js/pull/1140#issuecomment-333017825

As for deepObject and nested objects - that was explicitly left out of the spec, and it's ok to just Not Support It™.

[earth2marsh]

Are there any other examples of these nested deepObjects outside of Express? The more widespread a pattern, the more likely it is to be considered. Myself, I have an aversion to passing such a complicated object in the query string. Any insight into why Express even landed on this pattern?

[darrelmiller]

We were a bit stuck when allowing the deepObject style. We understood that users wanted this capability but there is no standard definition of what that serialization format looks like. We had a few choices, allow it and define our own standard and hope implementations followed it. Don't allow it because there are no standards, or add it in, say nothing about its format and hope that a default serialization format emerges.

If we can get some confidence that the qs package is becoming a defacto standard and we can create an accurate description of the serialization, then I have no issue recommending that we include that description in a future minor release of the spec.

[rmunix]

What about the URL max length limits? I think one of the reasons why a standard for object serialization in the URL is hard to materialize is because of the URL max length problem, the URL just wasn't intended to pass data in this way. Depending on the browser and/or server software being used the URL max length varies but in general it is small when compared to how much data can be transmitted with other methods like POST or PUT. It certainly would work for small objects but people tend to inadvertently abuse these kinds of APIs by passing bigger than allowed payloads.

[handrews]

@rmunix regarding URL max length problems, I was very happy to see the HTTP SEARCH method draft RFC revived last month: https://tools.ietf.org/html/draft-snell-search-method-01

[louisl]

Are there any other examples of these nested deepObjects outside of Express? The more widespread a pattern, the more likely it is to be considered. Myself, I have an aversion to passing such a complicated object in the query string. Any insight into why Express even landed on this pattern?

http://esbenp.github.io/2016/04/15/modern-rest-api-laravel-part-2/

I use a modified version of https://github.com/esbenp/bruno referenced in the article above in a few apis, it's extremely useful for including related data and search filtering. I'm not really sure how to define those in a spec. I do appreciate that the url strings could get silly long to the point of failing if abused, but without this sort of thing searches and such would have to be actioned as POST requests or limited to basic GET request params. It seems to me no ones really come up with a holy grail API solution for complex search queries so it's a bit of a free for all at the moment.

[bajtos]

Ruby on Rails

Seems to use the same approach.

Reference: https://edgeapi.rubyonrails.org/classes/Hash.html#method-i-to_query Source code: https://github.com/rails/rails/blob/b5302d5a820b078b6488104dd695a679e5a49623/activesupport/lib/active_support/core_ext/object/to_query.rb#L61-L86

Example code:

require "activesupport"

data = {
  "name" => "David",
  "nationality" => "Danish",
  "address" => {
    "street" => "12 High Street",
    "city" => "London",
  },
  "location" => [10, 20],
}
print data.to_query("person")

Produces the following query string, I have urldecoded and reformatted it for better readability:

person[address][city]=London&
person[address][street]=12+High+Street&
person[location][]=10&
person[location][]=20&
person[name]=David&
person[nationality]=Danish

Notice that array items are using an empty index, i.e. person[location][]=10, instead of person[location][0]=10.

[bajtos]

Python 2.7

AFAICT, Python does not support nested objects in query parameters.

Example code:

from urllib import urlencode
from urlparse import parse_qs

data = {
  'person': {
    'name': 'David',
    'nationality': 'Danish',
    'address': {
      'street': '12 High Street',
      'city': 'London',
    },
    'location': [10, 20],
  }
}

print urlencode(data)

Produces the following query string, I have urldecoded and reformatted it for better readability:

person={
  'nationality':+'Danish',+
  'location':+[10,+20],+
  'name':+'David',+
  'address':+{
    'city':+'London',+
    'street':+'12+High+Street'
  }
}

Interestingly enough, the roundtrip does not preserve the original data.

print parse_qs(urlencode(data))

Outcome:

{'person': ["{'nationality': 'Danish', 'location': [10, 20], 'name': 'David', 'address': {'city': 'London', 'street': '12 High Street'}}"]}

Another example:

print parse_qs('foo[bar]=1&foo[baz]=2')
# {'foo[baz]': ['2'], 'foo[bar]': ['1']}

[louisl]

Not API specific, but jQuery can generate nested array url params from objects.

http://api.jquery.com/jquery.param/

[Stratus3D]

JSONAPI sparse fieldsets require this: https://jsonapi.org/format/#fetching-sparse-fieldsets

[benhaynes]

Hey @earth2marsh and @darrelmiller ... We (Directus team) have been trying to use OpenAPI 3.0 for a while now, but the lack of support for nested deepObjects has kept us from using this spec. We have a dynamic API that allows for relatively complex filtering, for example: filter[<field-name>][<operator>]=<value>

GET /items/users?filter[category][eq]=vip

Our API Reference for this filtering

Is there any hope for this being supported in the future or should we "move on"?

[ewrayjohnson]

@bajtos: On October 11, 2018 you wrote "I'll try to find some time to fix swagger-js in the next few weeks." What is your status on this?

[bajtos]

Eh, I didn't even started 😢 Feel free to contribute the fix yourself.

[darrelmiller]

@benhaynes Sorry if this comes across as snarky, it's not intended to, it's just I'm in a rush and I don't know how to ask this in a friendly/sincere way.

What would you like us to do? Pick a winner from the many options? Design our own format? If we pick a format that is incompatible with what you currently use, would you switch? Should we find a way of supporting many different formats?

[benhaynes]

Hey @darrelmiller — not snarky at all, I sincerely appreciate the response as it maintains some momentum in the discussion!

We're certainly not trying to force the spec to follow our format, and understand your position of not wanting to blaze ahead without a standard to follow. To answer your question honestly, if the option you choose is incompatible with our method, then we wouldn't be able to use OpenAPI since we can't introduce a breaking change into our platform's filtering. Still, we'd support your team's decision if they think a different direction is a better solution.

I'm not sure how "extensible" your spec/codebase is, but support for multiple (even optional) serialization formats seems the most promising. Perhaps leaving these as unofficial until a "winner" eventually surfaces. In our experience, industry-leading frameworks offering solutions is the most efficient way for a de facto standard to emerge.

Our proposal is to support a deeply nested structure, such as: param[nest1][nest2]=value, where there can be 1,2,3,n levels of nesting. The comments here might be biased, but it seems that most are either already using this pattern or are recommending it.

Thanks again. I'd love to hear your (or anyone else's) thoughts on this approach!

[wellingguzman]

Hey @darrelmiller, I would like to understand the OpenAPI position on this. Is the reason to not support the outcome /items/users?filter[category][eq]=vip because it's not a standard or because the parameter definition format is not part of the standard?

Also, in the technical side, will this bring a complex/breaking change and it requires much more time?

In term of where else this format is supported I would like to add another example.

PHP

Example:

<?php

$data = [
  'person' => [
    'name' => 'David',
    'nationality' => 'Danish',
    'address' => [
      'street' => '12 High Street',
      'city' => 'London',
    ],
    'location' => [10,20],
  ]
];

echo http_build_query($data);

Result:

person[name]=David&
person[nationality]=Danish
&person[address][street]=12+High+Street
&person[address][city]=London
&person[location][0]=10
&person[location][1]=20

The result is urldecoded.

Also PHP automatically parse these parameters into an array. Passing the result above into query string will result in the original array.

[ardalis]

I think I'm running into this same issue. I have a simple C# type that needs two integers. I can define an API endpoint like this just fine: public IActionResult Foo(int x, int y) {}

and it works but if I use my binding model type with those same two integers as properties: public IActionResult Foo(FooModel model) {}

then my /swagger endpoint wants to generate JSON+Patch stuff and has no idea how to generate a URL with the appropriate querystring values to bind to that model. Will complex / deep object support help me in this (very simple) case? If not is there a known good way to support it currently?

[apimon]

Arrrgh, hit that wall, too.... great example of a not so obvious limitation that might kill your whole project dev setup and workflow.

[okirmis]

Ran into the same problem (using Rails), also with a parameter allowing for dynamic filtering.

The funny thing is, that swagger-editor does generate deeper nested objects as placeholder text for the parameters text area when clicking "try it out":

      - in: query
        name: filter
        schema:
          type: object
          properties:
            boolean_param:
              type: boolean
              nullable: true
            some_enum_of_types:
              type: array
              items:
                type: string
              nullable: true
          default: {}
        style: deepObject

results in

{
  "boolean_param": true,
  "some_enum_of_types": [
    "string"
  ]
}

which will be silently ignored. I know that this is actually a bug in swagger-editor, but it shows that it would be more consistent to allow deeper nested objects.

[rijkvanzanten]

What needs to be done to move on with this @earth2marsh? You asked for a couple examples outside of Express, which I think have been provided in the messages above. I can help out writing some of the needed documents for this in a PR if that helps.

[darrelmiller]

Does someone have a proposal of how we can fix the OpenAPI specification to allow it to tell tooling which flavour of deepNesting object serialization to use?

If someone wants to tackle this, here are some things to consider when creating the proposal:

• Which serialization flavors exist? Will they be enumerated in the spec?
• Will it be required that all tooling support all flavours?
• If not, what should tooling do when it hits a flavour it doesn't support?
• Do we have commitment from tooling implementors to implement this feature?

[rijkvanzanten]

Which serialization flavors exist? Will they be enumerated in the spec?

I hope I'm understanding what you're asking correctly, but based on the comments above

input

parameters:
 filterParam:
   in: query
   name: filter
   schema:
     type: object
   style: deepObject
   explode: true
   description: Options for filtering the results
   required: false

{
  "name": "ivan",
  "birthdate": {
    "gte": "1970-01-01"
  }
}

should result in

output

filter[name]=ivan&filter[birthdate][qte]=1970-01-01

The main difference seems to be in how nested arrays are handled. There seem to be slight differences between PHP, Node and Ruby on Rails.

Both PHP and qs* seem to favor the &numbers[0]=5&numbers[1]=10 syntax

$params = array(
  'filter' => array(
      'numbers' => array(5, 10)
  )
);

echo http_build_query($params);
// => filter%5Bnumbers%5D%5B0%5D=5&filter%5Bnumbers%5D%5B1%5D=10

var qs = require('qs');

var result = qs.stringify({
  filter: {
    numbers: [0, 10]
  }
});

console.log(result);
// => filter%5Bnumbers%5D%5B0%5D=0&filter%5Bnumbers%5D%5B1%5D=10

In normal non-nested use the Node built-in querystring and Swagger favor &numbers=5&numbers=10:

var querystring = require("querystring");

var result = querystring.encode({
  numbers: [5, 10]
});

console.log(result);
// => numbers=5&numbers=10

Combined by the comment @bajtos' comment about Ruby's implementation, this leaves us in a scenario where there are three different outputs for this given input for filter:

{
  "name": "ivan",
  "birthdate": {
    "gte": "1970-01-01"
  },
  "numbers": [5, 10]
}

&filter[name]=ivan
&filter[birthdate][gte]=1970-01-01
&filter[numbers]=5
&filter[numbers]=10

&filter[name]=ivan
&filter[birthdate][gte]=1970-01-01
&filter[numbers][0]=5
&filter[numbers][1]=10

&filter[name]=ivan
&filter[birthdate][gte]=1970-01-01
&filter[numbers][]=5
&filter[numbers][]=10

Seeing that different languages and implementations are already out there, I don't think we can pick one over the other. (Eg PHP's parse_str doesn't work with 1, while Ruby's won't parse 2).

qs seems to support all three of these flavors in parsing.

Re: your comment Oct 29 2018

If we can get some confidence that the qs package is becoming a defacto standard and we can create an accurate description of the serialization, then I have no issue recommending that we include that description in a future minor release of the spec.

Seeing the ~22 million weekly downloads, I think it's safe to say that qs has become the defacto query string parser in Node JS.

I don't know enough about the governance of the project nor the extended OpenAPI family of projects to make any comments about the other questions mentioned.

[philleepflorence]

I concur and second, @rijkvanzanten...

&filter[name]=ivan
&filter[birthdate][gte]=1970-01-01
&filter[numbers][]=5
&filter[numbers][]=10

This format is rather important! It would allow for an array of arrays (PHP) or an array of objects (Node).

[br3nt]

@darrelmiller , it seems query params could be defined in the same way as body objects by specifying a schema for a property. I'll add my thoughts about serialization after this example.

So, as an example:

params = {
  some_query_object: {
    hello: "world",
    foo: [
      { value: "blerg" },
      { value: "blah" }
    ],
    bar: [1, 2, 3]
  }
}

jQuery.param(params)

Serializes to this query string:

 some_query_object[hello]=world
&some_query_object[foo][0][value]=blerg
&some_query_object[foo][1][value]=blah
&some_query_object[bar][]=1
&some_query_object[bar][]=2
&some_query_object[bar][]=3

And, I would expect to be able to write something like this:

paths:
  "/api/example":
    post:
      produces:
        - application/json
      parameters:
        - name: some_query_object
          in: query
          description: The object in the query string
          required: false
          schema:
            "$ref": "#/definitions/an_object"
definitions:
  an_object:
    type: object
    properties:
      hello:
        type: string
      foo:
        type: array
        items:
          "$ref": "#/definitions/nested_object"
      bar:
        type: array
        items:
          type: string
  nested_object:
    type: object
    properties:
      value:
        type: string

Using this method, it would be easy to add multiple properties defined by a schema definition.

In regards to query string serialization, the only case OpenAPI really needs to account for is the various array assignment types (The python example above is clearly type: string so can be ignored).

I would suggest adding a single option arrayAssignmentType. It only applies to arrays. Not objects/maps. I'm not sure if it should be a global option or per property. Global should be fine as its generally enforced by frameworks/libraries.

arrayAssignmentType would have the the following options:

direct (or traditional like in jQuery param docs), where the values are assigned directly to the key:

item=a&item=b
OR
obj[item]=a&[item]=b

accessor, where values are assigned via an [] or [int] accessor:

item[]=a&item[]=a
OR
obj[item][1]=a&item[2]=b
OR
obj[item][]=a&obj[item][]=b
OR
obj[item][1]=a&[item][2]=b
OR
obj[][value]=a&obj[][value]=b
OR
obj[1][value]=a&obj[2][value]=b

The naming can be worked out later.

In regards to the accessor option, it would be interesting to see if various frameworks support both formats ([] vs [1]). I would expect the answer is both based on how JQuery serializes params:

$.param({ a: [1, 2, 3] })
"item[]=1&item[]=2&item[]=3 "

$.param({ item: [{ x: 'a' }, { x: 'b'}, { x: 'c'}] })
"item[0][x]=a&item[1][x]=b&item[2][x]=c"

I think the default value should be arrayAssignmentType: accessor just because that format is more explicit in how arrays are defined (...also "to accommodate modern scripting languages and frameworks such as PHP and Ruby on Rails" — jQuery docs lols)

[br3nt]

Ok, I just tested how Rails handle the different array accessors in the query string:

?item=1&item=2&item=3
# { 'item' => '3' }

?item[]=1&item[]=2&item[]=3
# { 'item' => ['1', '2', '3'] }

?item[1]=1&item[2]=2&item[3]=3
# { 'item' => { '1' => '1', '2' => '2', '3' => '3' } }

So, @darrelmiller, following on from my previous comment, perhaps there can be 4 options for arrayAssignmentType... direct (or traditional), emptyAccessor, numericAccessor, dynamicAccessor. Again, the naming is up for debate, it's more about the concept.

Likewise, perhaps arrayAssignmentType should be both a global option and a per-property option, to account for all scenarios.

[bajtos]

Thank you all for pushing this discussion forward ❤️

@br3nt's proposal seems very reasonable to me.

perhaps there can be 4 options for arrayAssignmentType... direct (or traditional), emptyAccessor, numericAccessor, dynamicAccessor. Again, the naming is up for debate, it's more about the concept.

It's not clear to me what is the difference between numericAccessor and dynamicAccessor, but that's a minor detail that can be figured out while working on the actual spec proposal.

perhaps arrayAssignmentType should be both a global option and a per-property option, to account for all scenarios.

+1

Do we want this option to be per-property (i.e. inside a schema definition) or per-parameter (i.e. inside an operation parameter definition)?

I prefer the later.

If we allow arrayAssignmentType on per-property basis, then we can end up with different properties using different array assignment types. That feels too complicated to me.

?q[hello]=world
# arrayAssignmentType: numericAccessor
&q[foo][0]=blerg
&q[foo][1]=blah
# arrayAssignmentType: emptyAccessor
&q[bar][]=1
&q[bar][]=2
&q[bar][]=3

Anyhow, if there is a need for this extra complexity, then perhaps we can allow arrayAssignmentType at three levels?

• global
• specific to an operation parameter
• specific to a property in a schema

The part important for me is that property-level or parameter-level setting should overwrite any globally-defined setting.

[darrelmiller]

Thanks everyone for the input here. This is exactly the type of information we needed, but didn't have when were originally defining the deepObject style.

I am a little concerned about the potential complexity for tooling to support a range of different formats, especially now we are talking about defining it at different levels and allowing overloading. I wonder if this is unnecessary complexity.

There are two major use-cases for being able to define how an object is serialized in the query string. 1) In client tooling where it is necessary to construct a URL so an API can be called. 2) In server side tooling to parse a URL and reconstruct a parameter value as an object.

If we go down the path of creating configuration parameters that reflect how a particular library/language/platform does the serialization then we are leaking implementation details through the OpenAPI description. This is somewhat defeating what OpenAPI is trying to do.

There doesn't seem to be a big a variation in formats as I once believed and I wonder if there is value in OpenAPI just picking the most compatible format and completely specifying it. From what I can tell it would be possible for us to add wording to the specification that would allow us to say deepObject follows these rules: A parameter named q that has a value that looks like this,

{
    hello: "world",
    foo: [
      { value: "blerg" },
      { value: "blah" }
    ],
    bar: [1, 2, 3]
  }

would serialize as

? q[hello]=world&q[foo][0][value]=blerg&q[foo][1][value]=blah&q[bar][0]=1&q[bar][1]=2&q[bar][2]=3

This assumes that all numeric keys are references to elements of an array and all textual keys are references to elements of a map. The keys could be optionally quoted to allow keys for maps to be numbers.

The use of the empty indexer seems problematic to me because URL query parameters usually do not have an order that is semantically significant but array values usually are ordered.

If we assume most frameworks support both the numeric accessor and the empty accessor, then limiting deepObject to just using the numeric accessor should provide reasonable compatibility. It does appear this would be a problem for Rails. Some kind of OpenAPI specific parser would be needed to support the deserialization process. However, whichever format we pick, one or more platforms are going to have to do work to support it.

I do think that having a few platforms need to build libraries to conform to a consistent approach is a better long term outcome than creating a parameterization system that effectively describes the platform. If we did that we might as well create styles for rubyDeepObject, phpDeepObject, nodeDeepObject

I do however recognize there is a need to retroactively describe an API that has selected a non-standard serialization format. There is a solution to that. Instead of using the style parameter which is designed to be quick an easy way to identify a common serialization style, you can use content and a media type.

parameters:
  - name: q
    content:
      application/vnd.ruby.deepObject: {}

The purpose of media types is to describe semantics, syntax and serialization of data going over the wire. Normally, they are used for request and response bodies. However, OpenAPI allows defining a content object for parameters also. This also allows defining a schema if it is necessary to identify if a particular property is an object or an array when the syntax in the URL does not disambiguate.

In the past I was not a fan of OpenAPI picking a winner when it comes to the serialization format. However, considering the impact of the alternative approach, I have to wonder if it is the right thing to do. How would people feel if OpenAPI explicitly described a format for deepObject and required using content and a custom media type to describe parameters that don't match that standard?

[everlast240]

Hey guys, I've been trying different things for some time now, but can't get the standard JSONAPI sparse fieldsets params to work with swagger-ui... JSONAPI specifies a way to fetch only required attributes of the included resources. E.g.:

fields[resource1]=attrib1,attrib2,attrib3&fields[resource2]=attrib1
// example:
fields[person]=first_name,last_name,age&fields[address]=city,full_address,latitude,longitude

This all works for my endpoints, but no matter how I format the payload I enter in SwaggerUI and/or setup the swagger JSON, I can't get it to work... For example, using the endpoint(s) works like that:

curl -X GET "http://localhost:5000/person/39?include=address,address.country&fields[person]=first_name,last_name,age&fields[address]=city,full_address,latitude,longitude"

Tried different swagger JSON setups (e.g.with deepObject etc), and payloads in the swagger-ui text field (e.g.: {person: "first_name,last_name,age", address: "city,full_address,latitude,longitude"}), but no success for now... Part of the complication is that the object / hash keys (model / resource name) are dynamic: they depend on the included resources (in the compound document) and I can't statically describe the schema definition.

Any ideas? Help is appreciated! PS: Using Grape (https://github.com/ruby-grape/grape/) and the Netfix fast_jsonapi (https://github.com/Netflix/fast_jsonapi).

[br3nt]

@darrelmiller, picking one format rather than a more versatile solution will do more harm than good.

It will alienate the frameworks that don't support whatever format is chosen. Using the example of numeric array accessors...

ASP.NET apps would be excluded as they use what I described as direct or traditional style array assignment. ASP.NETs qs format is more similar to:

?q.bar=1&q.bar=2&q.bar=3

JS frameworks which use the qs library, like ExpressJS and LoopBack, will also be excluded. As per the qs documents:

Any array members with an index of greater than 20 will instead be converted to an object with the index as the key. This is needed to handle cases when someone sent, for example, a[999999999] and it will take significant time to iterate over this huge array.

E.g:

?foo[21]=bar
{"foo":{"21":"bar"}}

However, qs supports both the direct or traditional style and the emptyAccessor. E.g:

?foo=bar&foo=baz
{"foo":["bar","baz"]}

?foo[]=bar&foo[]=baz
{"foo":["bar","baz"]}

Would tooling implements be expected to parse foo[21] as an array or a map? Is that fair? Why stop at 20? It's kind of arbitrary. At least the empty array accessor [] is consistent. It's always an array which gets populated based on the order of the query string.

PHP apps would be fine as PHP arrays work basically like hashes/maps.

Picking one option over another tells frameworks that their decisions are right or wrong. I don't think that's the job of OpenAPI Specifications. A better place to decide on standardization would be in a W3C proposal where the community could come to a consensus together.

Your concern about additional complexity is fair enough. However, I think using only content media types sweeps the problem under the rug a little. Small frameworks would be disadvantaged by this decision.

I think its reasonable to expect some minimum support from OpenAPI Specification for the array accessor styles of foo[]=a, foo[0]=a and foo=a&foo=b and it's not unreasonable to expect tooling implementers to support these three styles.

I think it would also be reasonable to assume that most (if not all) frameworks would offer consistent array accessor parsing functionality. As such a global option specifying array accessor styles would be sufficient to tell tooling which to use (rather than on a per-property or per-path basis which I agree is way too complex).

I don't think the above could be considered as leaking framework functionality into the OpenAPI-Specification.

Anything more complex than the above, then I also agree that content media types would be a good solution.

[ewrayjohnson]

My apologies for not responding for while. This may seem to be to simplistic considering the complexity of other comments, but what if content (each qs param value) be subject to JSON.parse/JSON.stringify subject to URL/URI decoding/encoding as needed with whitespace removal.

For example: { "name": "ivan", "birthdate": { "gte": "1970-01-01" }, "numbers": [5, 10] }

can have whitespace removed as

{"name":"ivan","birthdate":{"gte":"1970-01-01"},"numbers":[5,10]}

encoded as

?param1=%7B%22name%22%3A%22ivan%22%2C%22birthdate%22%3A%7B%22gte%22%3A%221970-01-01%22%7D%2C%22numbers%22%3A%5B5%2C10%5D%7D

It may be possible that only certain characters need to be encoded (e.g. ampersand as %26)

[louisl]

param1=%7B%22name%22%3A%22ivan%22%2C%22birthdate%22%3A%7B%22gte%22%3A%221970-01-01%22%7D%2C%22numbers%22%3A%5B5%2C10%5D%7D

That's pretty much the same as defining param1 as a string then decoding and objectifying the string which is how I'm getting around the issue for now.

The problem with that approach is there's no way in the spec to tell what a valid object/array would be for param1 or that param1 should even be an object/array.

[bajtos]

param1=%7B%22name%22%3A%22ivan%22%2C%22birthdate%22%3A%7B%22gte%22%3A%221970-01-01%22%7D%2C%22numbers%22%3A%5B5%2C10%5D%7D

In LoopBack, we support both style. When the parameter is described as array or object with style: deepObject, then:

• First, we let qs to parse [] notation into an object or an array. If this returns an object value then we are done.
• Otherwise if the query provides a string value, then we deserialize the value from JSON.

See https://github.com/strongloop/loopback-next/blob/d1eef59928fe8afb56e8bfce3d04e1e3a656ae2b/packages/rest/src/coercion/coerce-parameter.ts#L169-L200

The problem with that approach is there's no way in the spec to tell what a valid object/array would be for param1 or that param1 should even be an object/array.

Yes, the spec does not provide any means how to express this style. However, I can imagine adding e.g. a new style variant to describe JSON-serialized values.

For example:

   in: query
   name: filter
   schema:
     type: object
   style: json
   description: Options for filtering the results
   required: false

[louisl]

Yes, the spec does not provide any means how to express this style. However, I can imagine adding e.g. a new style variant to describe JSON-serialized values.

For example:

   in: query
   name: filter
   schema:
     type: object
   style: json
   description: Options for filtering the results
   required: false

Something like that would be useful, I wonder if it could be expanded to somehow use a schema for validity of the JSON object?

[hkosova]

@ewrayjohnson

what if content (each qs param value) be subject to JSON.parse/JSON.stringify subject to URL/URI decoding/encoding as needed with whitespace removal. For example:

{
  "name": "ivan",
  "birthdate": {
    "gte": "1970-01-01"
  },
  "numbers": [5, 10]
}

can have whitespace removed as

{"name":"ivan","birthdate":{"gte":"1970-01-01"},"numbers":[5,10]}

encoded as

?param1=%7B%22name%22%3A%22ivan%22%2C%22birthdate%22%3A%7B%22gte%22%3A%221970-01-01%22%7D%2C%22numbers%22%3A%5B5%2C10%5D%7D

I believe this particular use case is already covered by the content keyword instead of schema/style/explode.

parameters:
  - in: query
    name: param1
    content:
      application/json:
        schema:
          type: object
          properties:
            name:
              type: string
              example: ivan
          ...

[chargio]

This would be using this feature extensively. We have many use cases where the user needs this feature

[kurko]

To be able to document things, I've been using the following, which is not ideal:

parameters:
  - schema:
    type: string
  in: query
  name: 'filter[search][eq]'

This is a particular problem with standards like jsonapi.org that rely on nested objects as query strings.

[ahx]

@kurko What problems did you have with this flat-nested-syntax (name : 'filter[search][eq]') ? I am doing the same. The only problem I have had with this so far was that we cannot use anyOf when using this flat syntax compared when using a real object. Other than that, I would prefer using that syntax, because it is shorter. We are following jsonapi.org as well.

[kurko]

@ahx you won't be able to use tools to automatically validate your requests, unless they cast the string into some sort of dictionary/hash/object. For instance, with Ruby (I'm trying out Committee gem for request validations), that spec would be converted to { "filter[search][eq]" => "value" } (note the string as key) whereas in fact it would need to be:

{
  filter: {
    search: {
      eq: "value"
    }
  }
}

I have to check whether none of the tools I'm using isn't trying to hack this and convert to an object. Haven't got to that point yet in development.

[ahx]

@kurko Hey, I am also using Ruby 🚀 This is a known issue for committe, but this was solved in openapi_first (which is an alternative to committee created by myself 👋 )

[PaulWay]

We needed to support filters like these in our API, so I've written this code:

https://github.com/PaulWay/django_filter_deep_object_param

It's kind of a copy and paste, and I'm still working on tests and better documentation. But hopefully that can be a basis of some of the work to support deep object querying. I'm also completely open to any suggestions on how to do this better :smile:

It works for us particularly because PostgreSQL 11 and above support deep querying into JSON objects, but it also supports queries through Django relationships.

[ewrayjohnson]

We needed to support filters like these in our API, so I've written this code:

https://github.com/PaulWay/django_filter_deep_object_param

It's kind of a copy and paste, and I'm still working on tests and better documentation. But hopefully that can be a basis of some of the work to support deep object querying. I'm also completely open to any suggestions on how to do this better :smile:

It works for us particularly because PostgreSQL 11 and above support deep querying into JSON objects, but it also supports queries through Django relationships.

The double underscore delimiter is not foolproof when properties/relations contain leading or trailing underscores. I suggest a character not valid in loopback property/relation names but valid in url query string preferably without encoding (possibly a tilde or believe it or not a period). Caveat: All this is just off the top of my head late at night with no research.

[PaulWay]

Good thinking @ewrayjohnson - good checks to do.

This cannot prevent possible exploration of the database schema using Django's relations, and there are security implications. It all depends on the field you're querying and how you're expressing your relations in Django.

[UkonnRa]

In JSON:API, we prefer design query params like this and https://github.com/json-api/json-api/issues/1445:

GET /authors?page[articles][offset]=10&page[articles][limit]=5 <-- Nested object

GET /comments?filter[post]=1,2&filter[author]=12 <-- Primitive array inside the object

So:

1. I think nested object is useful for more powerful API design
2. About the primitive array inside deep objects, as this comment said, we should fine some way to let users customize the representation

[creckord]

Spring also supports deep objects using dot-notation, i.e. GET /authors?page.articles.offset=10&page.articles.limit=5

When going forward with this proposal, what would be a really great addition would be the option to skip the root parameter name. This would neatly solve (some aspects of) OAI/Overlay-Specification#34:

paths:
  /pageable-endpoint:
     get:
         parameters:
            - name: query
               in: query
               schema:
                   type: string
            - name: page
               in: query
               required: false
               schema:
                   $ref: '#/components/schemas/Page'
               # option to skip 'page' in the effective query parameter name here
               style: deepObject
components:
   schemas:
      Page:
         type: object
         properties:
           page:
              type: integer
           items:
              type: integer

could result in

GET /pageable-endpoint?query=foo&page=5&items=100

instead of

GET /pageable-endpoint?query=foo&page.page=5&page.items=100

[hkosova]

@creckord

When going forward with this proposal, what would be a really great addition would be the option to skip the root parameter name. This would neatly solve (some aspects of) OAI/Overlay-Specification#34: ... could result in

GET /pageable-endpoint?query=foo&page=5&items=100

This particular example is already supported by style: form + explode: true (which is the default style for query params) rather than deepObject.

[creckord]

@hkosova

This particular example is already supported by style: form + explode: true (which is the default style for query params) rather than deepObject.

Ah, good to know. Then it's just the Java+okhttp client generator that does not support this particular style.

What would happen with that style if the Page schema had nested object-type properties?

[hkosova]

@creckord

What would happen with that style if the Page schema had nested object-type properties?

Undefined behavior / not supported.

Nested structures should ideally be handled by deepObject, which is what this discussion is about.

[mrleblanc101]

Hi, On a project I'm working on, we are trying to encore these filters. We tried style: form, explode: true and style: deepObject, explode: true without luck.

{
  persona: [
    'positionner_1er',
    'vendre',
    'selectionner',
    'maximiser',
    'positionner_2e'
  ],
  type: [ 'programme', 'parcours', 'activite', 'capsule' ],
  theme: [
    'ressources-humaines',
    'numerique',
    'marketing-com',
    'vente',
    'finance',
    'gestion',
    'innovation'
  ],
  difficulte: [ 'avance', 'debutant', 'intermediaire' ],
  region: [
    'outaouais',
    'mauricie',
    'bas-saint-laurent',
    'laurentides',
    'province-du-quebec',
    'abitibi-temiscamingue',
    'capitale-nationale',
    'montreal',
    'centre-du-quebec',
    'chaudiere-appalaches',
    'cote-nord',
    'estrie',
    'gaspesie-iles-de-la-madeleine',
    'lanaudiere',
    'laval',
    'monteregie',
    'nord-du-quebec'
  ],
  duree_intervalle: [ '*-3600', '3600-10800', '10800-28800', '28800-230400', '230400-*' ],
  dispo_dans_catalogue: true
}

We need an encoding like this:

filters[persona][]=positionner_1er&filters[persona][]=vendre&filters[persona][]=selectionner&filters[persona][]=maximiser&filters[persona][]=positionner_2e&filters[type][]=programme&filters[type][]=parcours&filters[type][]=activite&filters[type][]=capsule&filters[theme][]=ressources-humaines&filters[theme][]=numerique&filters[theme][]=marketing-com&filters[theme][]=vente&filters[theme][]=finance&filters[theme][]=gestion&filters[theme][]=innovation&filters[difficulte][]=avance&filters[difficulte][]=debutant&filters[difficulte][]=intermediaire&filters[region][]=outaouais&filters[region][]=mauricie&filters[region][]=bas-saint-laurent&filters[region][]=laurentides&filters[region][]=province-du-quebec&filters[region][]=abitibi-temiscamingue&filters[region][]=capitale-nationale&filters[region][]=centre-du-quebec&filters[region][]=chaudiere-appalaches&filters[region][]=cote-nord&filters[region][]=estrie&filters[region][]=gaspesie-iles-de-la-madeleine&filters[region][]=lanaudiere&filters[region][]=laval&filters[region][]=monteregie&filters[region][]=nord-du-quebec&filters[duree_intervalle][]=%2A-3600&filters[duree_intervalle][]=3600-10800&filters[duree_intervalle][]=10800-28800&filters[duree_intervalle][]=28800-230400&filters[duree_intervalle][]=230400-%2A

Or like this:

filters[persona]=positionner_1er,vendre,selectionner,maximiser,positionner_2e&filters[type]=programme,parcours,activite,capsule&filters[theme]=ressources-humaines,numerique,marketing-com,vente,finance,gestion,innovation&filters[difficulte]=avance,debutant,intermediaire&filters[region]=outaouais,mauricie,bas-saint-laurent,laurentides,province-du-quebec,abitibi-temiscamingue,capitale-nationale,centre-du-quebec,chaudiere-appalaches,cote-nord,estrie,gaspesie-iles-de-la-madeleine,lanaudiere,laval,monteregie,nord-du-quebec&filters[duree_intervalle]=%2A-3600,3600-10800,10800-28800,28800-230400,230400-%2A

But all with deepObject our params end-up overwritting each other:

filter[persona]=positionner_1er&filter[persona]=vendre&filter[persona]=selectionner&filter[persona]=maximiser&filter[persona]=positionner_2e&filter[type]=programme&filter[type]=parcours&filter[type]=activite&filter[type]=capsule&filter[theme]=ressources-humaines&filter[theme]=numerique&filter[theme]=marketing-com&filter[theme]=vente&filter[theme]=finance&filter[theme]=gestion&filter[difficulte]=avance&filter[difficulte]=debutant&filter[difficulte]=intermediaire&filter[region]=outaouais&filter[region]=mauricie&filter[region]=bas-saint-laurent&filter[region]=laurentides&filter[region]=province-du-quebec&filter[region]=abitibi-temiscamingue&filter[region]=capitale-nationale&filter[region]=centre-du-quebec&filter[region]=chaudiere-appalaches&filter[region]=cote-nord&filter[region]=estrie&filter[region]=gaspesie-iles-de-la-madeleine&filter[region]=lanaudiere&filter[region]=laval&filter[region]=monteregie&filter[region]=nord-du-quebec&filter[duree_intervalle]=*-3600&filter[duree_intervalle]=3600-10800&filter[duree_intervalle]=10800-28800&filter[duree_intervalle]=28800-230400&filter[duree_intervalle]=230400-*&filter[dispo_dans_catalogue]=true&sort=created_at&append=pourcentage_complete

Here is the schema if someone could help:

    formationFilter:
      schema:
        type: object
        properties:
          type:
            type: string
            example: 'parcours,programme,activite'
          theme:
            type: string
            example: 'innovation,gestion,numerique,ressources-humaines,marketing-com,vente,finance'
          region:
            type: string
            example: 'montreal,outaouais,mauricie,bas-saint-laurent,laurentides,province-du-quebec,abitibi-temiscamingue,capitale-nationale,centre-quebec,cote-nord,estrie,gaspesie–iles-de-la-madeleine,lanaudiere,laval,monteregie,nord-du-quebec,saguenay–lac-saint-jean'
          difficulte:
            type: string
            example: 'debutant,intermediaire,avance'
          etat:
            type: string
            example: 'complete,en_cours,non_achete'
          modes_de_presentation:
            type: string
            example: 'en_personne,a_distance,differe'
          persona:
            type: string
            example: 'positionner_1er,positionner_2e,vendre,selectionner,maximiser'
          duree_intervalle:
            type: string
            example: '0-300,3600-18000,360000-*'
          duree_plus_grand_egal:
            type: integer
            example: 10000
          duree_moins_de:
            type: integer
            example: 100
          dispo_dans_catalogue:
            type: boolean
            example: true
      in: query
      name: filter
      style: deepObject
      explode: true
      description: Filters available for Formations. Must be wrapped in "filter" object.

[dafeder]

Just chiming in, that on https://github.com/GetDKAN/dkan we have settled on what I guess I'll call "qs style" serialization for complex GET requests. FWIW this style essentially has native support in PHP and additional support in core Symfony libraries. Struggling with the same issues as everyone, probably going to add some minimal parameters and just add narrative text for now, pointing people to the JSON schema and suggestions for how to serialize.

[ElectroBuddha]

Five years later, this feature request is still relevant. Any news on this ?

[jakguru]

I'll also chime in that this is a common method of building complex queries for elasticsearch. While in their case, they're expecting the body to be passed via POST, the ability to deeply nest objects in arrays allows for a huge level of flexibility in generating queries.

Also, AdonisJS uses qs style parsing