Query for Post List

[EdwardHinkle]

Discussion for adding a way to query a list of posts from the Micropub endpoint (https://indieweb.org/Micropub-extensions#Query_for_Post_List)

Below are some thoughts from the brainstorming page.

Use Cases

• [[headless_CMS|headless cms]] / Separation of data storage and data display
• Private note taking / journaling
• Find and update existing posts eg. change a "want to read" to "read" after complete
• Full content management UIs
• Find and edit draft / hidden posts
• Synchronizing various documents eg. bookmarks
• Discover if a user has liked / replied to a specific page

Query Parameters

In order to achieve the above use cases there would need to be a robust query system in place. It would need to be able to query for:

• url (this filters and returns the single post with that url. This is how q=source currently functions)
• posts based on post type (similar to https://github.com/indieweb/micropub-extensions/issues/1)
• pages of posts with defined limits
• posts with specific properties
• posts with combinations of properties
• posts without certain properties
• posts with properties that match a certain value (eg, all posts that have in-reply-to == 'example.com')

Other Considerations

• Any returned item in a list would need to include a url property so that micropub clients could use the url for update and delete requests etc.
• Would require one or more scopes

[EdwardHinkle]

Here are the notes about IndieWeb implementation from the brainstorming page:

• j4y_funabashi implemented an extension to the q=source query on my micropub endpoint, if no url is provided it returns a list of posts. Not yet paginated but that is the plan.
• eddiehinkle implemented the q=source query extension.
  • When no url is provided, it returns a list of posts.
  • It doesn't support pagination yet, but currently returns all posts within the last month
  • It does support a post-type filter that will filter out what posts are returned based on the post type discovery, similar to https://github.com/indieweb/micropub-extensions/issues/1
• grantcodes implemented an experimental extension that uses the mongodb query syntax since his posts are stored in MongoDB

[EdwardHinkle]

@cleverdevil Post List Query discussion has moved here, for when you want to work on integrating into Known

[EdwardHinkle]

It has been discussing that using paging should probably be similar to Microsub (https://indieweb.org/Microsub-spec#Example_Paging_Workflow) where the server provides both after and before strings for pagination. The server will then return results as appropriate. When you reach the point that there are no more results available, the results don't contain an after string.

The pagination tokens are opaque to the client, so they can be anything the server wants: page numbers, date string, post ids, etc.

[EdwardHinkle]

It probably makes sense to include the limit parameter from Microsub as well. So that would put us at:

https://example.tld/micropub?q=source

• &post-type=type - filters the returned posts to only posts that match the included post type. This would be a post type from the Post Type Discovery
• &after=pagination token - returns older posts
• &before=pagination token - returns newer posts
• &limit=number - limits the returned results by the number provided

As already included in the original spec:

• &url=POST URL - returns a single post that matches this url

[grantcodes]

I think the specific property queries might be more difficult to map into the url parameters.

They could potentially follow a similar format to the micropub update spec

In particular like the delete parameter which can either specify just the property name to delete or the value to delete.

In my mind there needs to be 2 types of queries on properties at least:

1. A specific property exists
2. A specific property contains a given value - eg. post-status = draft This could get more complex, if you wanted to match an entire array (eg. category = ['foo', 'bar']) or just contains the given parameter

There are also a number of others that may be worth having, but I am not so sure are as vital:

1. A property does not exists / is empty
2. A property value does not equal a given value
3. A property value is less than, greater than etc. compared to a given value

[grantcodes]

Another one worth considering would be a sort or order parameter. I can imagine clients wanting to either retrieve posts in chronological or reverse chronological order at a minimum.

And potentially even ordering by other properties, eg. Alphabetical by post title. Although I can't think of a solid use case for that right now.

[grantcodes]

I've updated my implementation so I support a bunch of new features expanding on what @EdwardHinkle already mentioned

When using q=source there is now a lot of other parameters available:

• post-type=type - Filters the returned posts to only posts that match the included post type. This would be a post type from the Post Type Discovery
• after=pagination token - Returns older posts (I just use page numbers for now)
• before=pagination token - Returns newer posts (I just use page numbers for now)
• limit=number - Limits the returned results by the number provided
• order=reverse - Returns posts in a reverse order
• exists[]=property name - Returns posts that include the given properties
• not-exists[]=property name - Returns posts that do not include the given properties
• property-property-name=value - Returns posts that contain the given property value e.g. property-post-status=draft

[cleverdevil]

FYI, I have a local branch of Known that supports:

• q=source
• &url=
• &limit=
• &offset=

I'm closely watching this thread, and if we can come to a consensus, I should be able to have a fully compliant implementation for Known in short order.

Things that need to be decided, from what I can tell:

• [ ] What's the official approach to pagination?
• [ ] Of @grantcodes' and @EdwardHinkle's additional query parameters, which should be adopted?

I will say that the &post-type parameter would be a bit difficult to implement in Known at this stage, because Known doesn't currently have any such notion. That said, a mechanism could be created that would map Known "Entity" types to post-types from the Post Type Discovery spec. I'd likely need direction from @benwerd and @mapkyca on how to do it.

[grantcodes]

Can someone write down here a valid use case for post type queries? I'm sure there were some but I can't remember why they'd be needed if property queries are supported

[EdwardHinkle]

Here is a link to the previous reasons cited: https://chat.indieweb.org/dev/2018-08-09#t1533837539815600

To summarize, there are instances where just a single property existing does not define some more complicated post types. So it means you either have to define complicated requirements in URL parameters (including that certain properties DON’T exist) or you use the PTD by using a post-type parameter.

Some examples:

• both Podcasts, Articles and Issues have a Title. So how do you get all articles but not podcast episodes or issues when they all have titles?
• Determining the difference between Video and Photo posts because Videos often have a photo property.
• Using PTD protects against new experimental post types getting returned erroneously because it happens to have the same property as another post type.

[swentel]

I like the post-type filter, and I guess in combination with the post-type discovery, we can either expose them or not in the micropub clients. In the case of the Drupal module: I map every micropub post to a different content type in Drupal. When a q=source would come in with a post-type filter, it would be very easy to filter out, so I'm +1 on that filter :)

I'm going to experiment with this the next week to add this to the drupal module and into indigenous for android. Since I've added basic update post support, having a list to quickly edit (instead of having to copy over urls and such), a very simple list would make my life extremely easy!

[swentel]

So the post list and post-type filter is in the latest release of the indigenous android client and drupal module. The post list listens to the post-type discovery as well, so if none is found, filtering is not possible at the moment. I'll gradually add more filter options (like limit) and paging in the next week.

[grantcodes]

So now that this is becoming more widely spread, I am starting to have issues with implied post types and micropub queries. Mostly around whether something should be a specific post type or if it should be any post type with a specific property.

My 2 main use cases, (which I think are totally valid, and either in wide spread use or wanted):

1. Drafts - Obviously there is post-status = draft But drafts are not defined by post type discovery at the moment. So at the moment to get a list of drafts there would either need to be a post-type=draft query or the client and server would both need to support property-post-status=draft.
2. Private journal entries - Again there is no post type defined for journals, and this time no property that is specific to journals. Personally I just make a note with post-status=private and category=journal. If I wanted to expose these to a micropub client I would likely need to make my own custom post type.

Having written these out I think my main issue is around the creation of custom post types and whether they should be actual post types, or just properties and how that works with micropub queries

[EdwardHinkle]

@grantcodes it looks like half your post is missing because you only mention 1 use case

[grantcodes]

@EdwardHinkle Yeah, published early by accident, updated now.

[swentel]

What if we do something like this as query params for properties:

• property-name=value
• property-value=value

That would make sure we don't end up with a dozen of property-{property-name} options for the query.

[grantcodes]

@swentel how would you then do multiple values at once?

[swentel]

Sending an array then: property-name[0]=post-status&property-value[0]=draft&property-name[1]=name&property-value[1]=title

Which makes it probably a bit less readable, but still remains flexible, at least imo.

[EdwardHinkle]

I do think post-status should be a property query, and I think something like a journal is best handled as a category property query. So I think adding property-post-status and property-category (not tied to those names) are two properties that would be very widely used and wouldn’t be confused as “conflicting with post types”

[EdwardHinkle]

I don’t think the post-type filter should be the only filter, I just think it’s the simplest filter when you are trying to get a post type.

That said, I think it’s okay for a specialized Micropub app to support an experimental post type and for servers that want that to support it as well. For example: Teacup would query Ate or Drank posts which aren’t part of Post-Type Discovery. But I think general purpose Micropub (Quill, Indigenous, etc) should probably stick to mostly the widely accepted post-types.

[EdwardHinkle]

Hmmm interesting @swentel. I can’t tell how I feel about that. Good or bad lol. A part of me thinks it’s good and the other part cringes a little.

@aaronpk With your API experience what do you think about the properties as far as being defined or flexible like @swentel suggested?

[grantcodes]

@swentel That handles multiple values of different properties, but what about multiple values of a single property?

property-name[0]=category&property-value[0]=foo&property-name[1]=category&property-value[1]=bar

vs

property-category[0]=foo&property-category[1]=bar

In my case I used the property- prefix because I found it easier to handle on the server. As I look for parameters prefixed with property- then I check if they are an array or string and handle them appropriately.

[swentel]

Oh right, multiple values, good catch. Mmm, I'd say, use comma separated values then. I can see myself filtering on two post-types as well, so post-type=note,article should work (even though Indigenous filter screen only allows you to select a single value though atm).

Note: I'm totally fine with property-{property-name} as well. It probably makes the spec clearer in the end anyway.

[aaronpk]

I still have some catching up to do in this thread, but just want to say -1 for comma-separated values. That adds a lot of complexity into the spec. Instead, let's use the existing mechanism that Micropub has for multiple values, which is to use the array syntax instead: prop[]=val1&prop[]=val2. Also note that Micropub explicitly defines the array syntax to be the square brackets with no offset number. https://www.w3.org/TR/micropub/#form-encoded-and-multipart-requests

[dshanske]

The WordPress Micropub endpoint now supports q=source with an optional limit parameter to return the last x number of posts. Likely implement paging in a future version.

[manton]

Sounds like paging is still an open question. Because Microsub was mentioned a few times, why not just copy the Microsub spec exactly: wrap the response with items and paging fields and use JF2 syntax to simplify everything. (Obviously that's a breaking change with existing implementations, but it doesn't seem like this is widely deployed yet except in home-grown solutions.)

If that's not possible, I'd probably be in favor of simple limit and offset params. Just wondering if this is an opportunity to simplify some of the list/get/edit/delete parts of Micropub, though.

[swentel]

The Drupal module q=source call implements paging like @manton describes, wrapping the posts in 'items'. It also supports filtering via 'limit' and 'post-type' GET params. This is all supported in Indigenous for Android too.

[manton]

Thanks @swentel! And does it use JF2 like Microsub, or the more verbose MF2-style JSON syntax with a properties field and arrays for values? (Assuming the latter but want to be sure.)

[swentel]

@manton The simple microsub version. I think Wordpress does the same, @dshanske should be able to confirm that.

[dshanske]

Only limit so far for me, but paging is on the horizon

[grantcodes]

I don't think there should be any use of JF2 anywhere, it is not used anywhere else in the spec, you post mf2 and receive mf2 via source=url queries so might as well stick with that.

[manton]

I've been testing this on Micro.blog using MF2-style responses and plan to roll it out tomorrow. I wonder if a goal for IndieWeb Summit should be to finalize some of these extensions and update the spec with more examples? (I'm going to document this in Micro.blog's help as well, because I think it can be a little confusing to implement for developers who are not familiar with the IndieWeb already.)

[manton]

Support for q=source (and editing and deleting posts) has been deployed on Micro.blog. It uses the MF2-style JSON as described on the wiki for the response. Thanks y'all!

[jamietanna]

Is the expectation for this that will contain all of the properties? Or could just be the URL/UID?

For instance, I've got a static site, so my Micropub endpoint would proxy a generated file from the site. If I can reduce the amount of properties to add to this, it'd be good, especially if this is really only being used for post listing.

(Originally published at: https://www.jvt.me/mf2/2020/04/3qnnb/)

[jamietanna]

Follow-up for anyone interested, I have generated all the properties and return them when using q=source

(Originally published at: https://www.jvt.me/mf2/2020/06/maoih/)

[jamietanna]

Answer to my question: Indigenous for Android expects that this list is the full post's content, as it renders some of the information (as seen in https://media.jvt.me/5a9abbb797.png) and I found would break if any nulls were found.

(Originally published at: https://www.jvt.me/mf2/2020/07/sm4ec/)

[jalcine]

Do we have a sense of what properties we should consider “standardized” from this list? I’ve reworked my implementation (it includes support for custom pages and some markup for tags so I can add more information to their standalone pages). Nothing really changes with regards to that but it does means that I’m looking for the field type now as well to determine h-entry, h-x-page and h-x-category (the latter being custom for reasons).

(Originally published at: https://v2.jacky.wtf/post/ef485fa8-33af-4f02-8844-572373a4fd74)

[manton]

For whatever it's worth, here's an example of the "standard" properties that Micro.blog always returns. At a minimum, I'd think most servers should return content, published, and url. The URL will be needed for editing.

{
  "items": [
    {
      "type": "h-entry",
      "properties": {
        "uid": [ 12345 ],
        "name":[ "Title here" ],
        "content": [ "Hello world." ],    
        "published": [ "2020-07-14T18:54:18+00:00" ],
        "post-status":[ "published" ],
        "url": [ "https://www.manton.org/2020/..." ]
      }
    }
  ]
}

[jamietanna]

I'd prefer to keep it as a fairly minimal list, maybe even just the type, uid and url, and then require a follow-up call to q=source&url=... if a given post is required. This (possibly) reduces work on the server to render all the information but should make it simpler for folks using static sites as they then don't need to render everything for q=source compared to calls for q=source&url=...

[manton]

Servers should at least have the option to return name, content, etc. It doesn't have to be required, but it is really important for any app that is showing a list of posts. Doing an extra query if you're showing a list of 100+ posts would be really wasteful.

[jamietanna]

That's fair enough - if we can clarify that they're not required properties, then that'd allow clients to be built more resiliently :+1:

[jalcine]

Throwing a wrench into the mix? How do people see a query for all posts that have a property matching a particular value? Like if the audience is set to a particular URL?

[jalcine]

Is anyone implementing queries like exists[] and not-exists[] in their systems? I don't see an independent ticket for it on here but I'm curious to see implementation approaches.

[jalcine]

I have a tracking ticket for Koype for implementing exists[] and not-exists[]. I plan to use this in Lwa to highlight if a sort of reaction was made to an entry before (like, in a channel, a post has been liked, replied to or shared already) to prevent duplication of content.

[jalcine]

There's quite a few things here in this issue. Would it make sense to split it up maybe into some sort of list of items? I don't know if @EdwardHinkle would be willing to add them to the bottom of the top-entry so it'd be easier to link to (with headings) on the Wiki.

(Originally published at: https://jacky.wtf/2022/7/OWkL)

[manton]

Good idea @jalcine. I feel like some of the older suggestions in this issue (like basic support for ?q=source to get the post list) are widely supported and stable. We should mark those as "done" and create new issues for any experimental extensions where there's no consensus yet.

[dshanske]

@jalcine @manton Eddie has been busy with life...I can take a run at it...I split off some things in the past

[dshanske]

Issue closed as remaining sub-topics now split into separate issues.

[manton]

Thanks @dshanske!