REST API query parameters for requesting field-based filtering of index schemas

michael-valdron commented 1 year ago

Which area this feature is related to?

/area registry

Which functionality do you think we should add?

Why is this needed? Is your feature request related to a problem?

Allows a consumer of the REST API to request for filtered down responses when working with GET /index and GET /v2index.

Detailed description:

Currently for GET /index and GET /v2index, there are a few query parameters which provide filtering based on schema version and architecture type. In addition to these, it would be good to have the following querying capabilities:

Stack/Sample query (fuzzy search) filter using index schema fields excluding select one
New lastModified filter

These capabilities would reduce load on both the registry index server and REST API consumers such as the registry viewer.

Describe the solution you'd like

Tasks

[x] Stack/Sample query parameters for filtering by index schema properties
- [x] Filtering using AND
[X] ~Stack/Sample lastModified named query parameter~ See #1327
[x] Additional testing additions to account for changes done
[x] Additional documentation additions to REST API docs to describe these parameters

Examples

Request: GET /index?name=python or GET /v2index?name=python

Response:

[
  {
    "name": "python",
    "displayName": "Flask",
    "description": "Flask is a web framework, it’s a Python module that lets you develop web applications easily. It’s has a small and easy-to-extend core: it’s a microframework that doesn’t include an ORM (Object Relational Manager) or such features.",
    "type": "stack",
    "tags": [
      "Flask",
      "Python",
      "Pip"
    ],
    ...
  },
  {
    "name": "python-django",
    "displayName": "Django",
    "description": "Django is a high-level Python web framework that enables rapid development of secure and maintainable websites. Built by experienced developers, Django takes care of much of the hassle of web development, so you can focus on writing your app without needing to reinvent the wheel. It is free and open source, has a thriving and active community, great documentation, and many options for free and paid-for support.",
    "type": "stack",
    "tags": [
      "Django",
      "Python",
      "Pip"
    ],
    ...
  }
]

Describe alternatives you've considered

Without these changes, the filters in this issue would need to be applied on the client side which both requires the registry index server to create the entire list of index schemas to send as the response and put additional load on the consumer having to filter and pull out needed information after the fact.

Additional context

elsony commented 1 year ago

For the fuzzy search, what fields are we using to do the matching? Do we use name+description+tags or some other combination?
Do we need a search based on the schema version (or combination of schema version)?

schultzp2020 commented 1 year ago

@elsony

For the fuzzy search, what fields are we using to do the matching? Do we use name+description+tags or some other combination?

That works for me.

Do we need a search based on the schema version (or combination of schema version)?

We already have a search based on the schema version

michael-valdron commented 1 year ago

@elsony @schultzp2020

For the fuzzy search, what fields are we using to do the matching? Do we use name+description+tags or some other combination?

That works for me.

I will update the description to include this.

schultzp2020 commented 1 year ago

@michael-valdron Could we have a filter for all the logical filterable devfile properties?

What I am thinking... (You already pointed out a few of these in the pr)

name
type? Maybe we should change the API so it defaults to returning all the stacks/samples and then you need to specify ?type=<stacks || samples>
tags
projectType
language
provider
schemaVersion? Change the way this is done so its ?schemaVersion=2.1 or ?schemaVersion=2.1-2.2 where schema version accepts a SEMVER-like version or a range

Also, for the different versions, its so minimal I can filter it out easily on the frontend. So its up to you if you think its necessary to add. If so, instead of it being a query parameter can you add a property called availableVersions that contains the array of version numbers?

Lastly, is it possible to add a parameter called lastModified that returns the last modified date for the corresponding devfile? I already discussed this feature request with @elsony and he likes it. If so, can you make this queryable? This would help the registry viewer.

michael-valdron commented 1 year ago

@schultzp2020

tags

projectType

language

provider

Lastly, is it possible to add a parameter called lastModified that returns the last modified date for the corresponding devfile? I already discussed this feature request with @elsony and he likes it. If so, can you make this queryable? This would help the registry viewer.

These I am good with including.

type? Maybe we should change the API so it defaults to returning all the stacks/samples and then you need to specify ?type=<stacks || samples>

schemaVersion? Change the way this is done so its ?schemaVersion=2.1 or ?schemaVersion=2.1-2.2 where schema version accepts a SEMVER-like version or a range

It might be complicated as changing these might break functionality with current consumers.

Also, for the different versions, its so minimal I can filter it out easily on the frontend. So its up to you if you think its necessary to add. If so, instead of it being a query parameter can you add a property called availableVersions that contains the array of version numbers?

Not sure what you mean by adding a property called availableVersions. Do you mean instead of doing something like GET /index?versionList=true might be better to do GET /index/availableVersions or GET /index/available-versions?

schultzp2020 commented 1 year ago

@michael-valdron

Not sure what you mean by adding a property called availableVersions. Do you mean instead of doing something like GET /index?versionList=true might be better to do GET /index/availableVersions or GET /index/available-versions?

I mean we do something like this. Adding this index/functionality isn't necessary for the registry viewer though. GET /index

[
  {
     ...
    "language": "go",
    "provider": "Red Hat",
    "availableVersions": ["1.1.0", "1.2.0"],
    "versions": [
      {
        "version": "1.1.0",
        "schemaVersion": "2.0.0",
        "default": true,
        ...
      },
      {
        "version": "1.2.0",
        "schemaVersion": "2.1.0",
        "description": "Stack with the latest Go version with devfile v2.1.0 schema verison",
        "tags": [
          "testtag"
        ],
        "icon": "https://raw.githubusercontent.com/devfile-samples/devfile-stack-icons/main/golang.svg",
        ...
      }
    ]
  },
...
]

michael-valdron commented 1 year ago

@schultzp2020

Not sure what you mean by adding a property called availableVersions. Do you mean instead of doing something like GET /index?versionList=true might be better to do GET /index/availableVersions or GET /index/available-versions?

I mean we do something like this. Adding this index/functionality isn't necessary for the registry viewer though. GET /index
[
  {
     ...
    "language": "go",
    "provider": "Red Hat",
    "availableVersions": ["1.1.0", "1.2.0"],
    "versions": [
      {
        "version": "1.1.0",
        "schemaVersion": "2.0.0",
        "default": true,
        ...
      },
      {
        "version": "1.2.0",
        "schemaVersion": "2.1.0",
        "description": "Stack with the latest Go version with devfile v2.1.0 schema verison",
        "tags": [
          "testtag"
        ],
        "icon": "https://raw.githubusercontent.com/devfile-samples/devfile-stack-icons/main/golang.svg",
        ...
      }
    ]
  },
...
]

I don't see why we can't have this addition to the V2 index schema. I can do this instead of having a ?versionList query parameter to fetch this list from versions.

michael-valdron commented 1 year ago

@schultzp2020 @elsony

Was there a preferred datetime format for lastModified? Should we just use the date?

schultzp2020 commented 1 year ago

@michael-valdron ISO 8601 YYYY-MM-DD

michael-valdron commented 1 year ago

Based on recent changes to the description, this issue is no longer a dependency for #950.

schultzp2020 commented 1 year ago

@michael-valdron Can you add a query to show x number of devfiles or a range of devfiles?

e.g. GET /index?show=10 would show the first 10 devfiles and GET /index?show=11-20 would show the 11th through 20th devfiles.

schultzp2020 commented 1 year ago

@michael-valdron Also, I would look into https://github.com/google/go-querystring because I am using https://www.npmjs.com/package/query-string on the frontend with the array format as comma.

michael-valdron commented 1 year ago

As discussed in the community slack channel:

Decided form of the filters in the Registry REST APIs /index and /v2index query strings will be in the form ?field1=val1&field2=val2 along with reserved operational query parameters: e.g. show, minSchemaVersion, and maxSchemaVersion.

michael-valdron commented 1 year ago

Recommended to complete #972 before completing this issue.

michael-valdron commented 10 months ago

Need to wait for the completion of #1197 to include additions for deployment scope.

thepetk commented 10 months ago

@michael-valdron @elsony After the refinement on 04 Aug 2023 were wondering if this should be transformed into an EPIC and have as child issue the https://github.com/devfile/api/issues/1065 which is related to filters.

michael-valdron commented 10 months ago

@michael-valdron @elsony After the refinement on 04 Aug 2023 were wondering if this should be transformed into an EPIC and have as child issue the #1065 which is related to filters.

I believe this makes sense, #1065 covers a subset of this issue and we can separate the rest into other child issues.

elsony commented 10 months ago

Do we want to change this to an Epic only because we wanted to add #1065 as a child issue? Are there any other child issues that we need? If it is only for #1065, we can probably just link it as related. If there are more, I am fine with changing this to an epic

michael-valdron commented 10 months ago

Do we want to change this to an Epic only because we wanted to add #1065 as a child issue? Are there any other child issues that we need? If it is only for #1065, we can probably just link it as related. If there are more, I am fine with changing this to an epic

Related issue might be another option because though this issue could be split into similar child issues the addition of filtering mechanisms might not be that time consuming to implement and there is only two (not including #1065) so this could be left a moderate sized issue with a related issue of a smaller size.

michael-valdron commented 10 months ago

Do we want to change this to an Epic only because we wanted to add #1065 as a child issue? Are there any other child issues that we need? If it is only for #1065, we can probably just link it as related. If there are more, I am fine with changing this to an epic

Related issue might be another option because though this issue could be split into similar child issues the addition of filtering mechanisms might not be that time consuming to implement and there is only two (not including #1065) so this could be left a moderate sized issue with a related issue of a smaller size.

I am leaning towards this if there are no more filters to add, otherwise the epic option seems better.

thepetk commented 10 months ago

I think we can keep it as is :) I've updated the #1065 and its refinement date

michael-valdron commented 10 months ago

Based on changes to https://github.com/devfile/api/issues/1065, this issue will now include filtering devfile stack/sample deprecation within the "Stack/Sample query parameters for filtering by index schema properties".

michael-valdron commented 10 months ago

Need to wait for the completion of #1197 to include additions for deployment scope.

Also #964 for deprecated stack/sample support.

michael-valdron commented 7 months ago

Need to wait for the completion of #1197 to include additions for deployment scope.

Also #964 for deprecated stack/sample support.

Been determined that the content under #1197 are not blocking this issue and filters should be added before the additional field.

michael-valdron commented 7 months ago

Broken down into two issues, #1327 will now track the addition of the lastModified filter.

Resizing this issue to 5 days from 8 days which was sized on October 20, 2023.

michael-valdron commented 4 months ago

Defined additional parameters under OpenAPI spec: https://github.com/michael-valdron/devfile-registry-support/commit/d13aea0a2ef9275a8807cdfc90ffc0648d528eca

michael-valdron commented 3 months ago

Working through filtering functions, I have created getter functions tied to each index schema field to help with accessing field in reusable source.

michael-valdron commented 2 months ago

Field filtering is now usable through the index endpoints, still working through integration test cases and updating the REST API document.

michael-valdron commented 2 months ago

The remaining work on the issue will be blocked by #1484 as it prevents building of the devfile registry, will shift into next sprint for completion to cover #1484 before finalizing this issue.

michael-valdron commented 2 months ago

Unblocked with the merge of devfile/registry-support#226.

michael-valdron commented 2 months ago

PR https://github.com/devfile/registry-support/pull/202 is ready for review.

michael-valdron commented 1 month ago

I have addressed @thepetk comment with the following criteria:

Setting deprecated to true will return all deprecated stacks (or samples):

?deprecated=true

Setting deprecated to false will return all non-deprecated stacks (or samples):

?deprecated=false

Not setting deprecated will not trigger this filter.

devfile / api