Import from OpenSalt needs a paging or filtering option

[mparsons-ce]

@woodkri Jeff Grann wanted to import a framework from the OpenSalt sandbox. He entered the following and click the case connect button: https://sandbox.opensalt.net/

The first 100 frameworks were displayed, but the one he was interested in was not shown. There needs to be a paging or perhaps filtering capability. OR the ability to enter the actual URL. I did try to enter the following but it was invalid: https://sandbox.opensalt.net/cftree/doc/2328

I noticed the search results contained checkhttps://sandbox.opensalt.net/uri/ef88269b-9add-4517-938f-f479b8afaab0

So I tried entering the following but that didn't work either. https://sandbox.opensalt.net/uri/22aa2502-fb7c-5802-a6e4-09d190fa34bd

[jeannekitchens]

@mparsons-ce @woodkri The link to the above issue is to a PCGs' CASE Sandbox where every framework I checked is saved as DRAFT. PCG is a company that provides CASE services. Don't only published/public CASE Frameworks import via the CaSS Remote Server options? We're not going to import the entire open salt repo. Isn't CaSS set up to handle CASE uploads one at at time using the published/public CASE URI? We have CASE conformance so we know the import works. Let's absolutely make sure there's instructions on how to use a public/published CASE Framework URI to be ingested by CaSS.

@woodkri I remember you having some URIs that were public and published handy. Do you have any? Do they work?

@mparsons-ce what owuld paging and filtering with CaSS be for? It's not our intent to mass publish from CASE but what is already supported is remote server ingest one by one.

[woodkri]

@jeannekitchens We have the ability to import draft frameworks as long as they are public and don't require a login or API key to access.

We do not have the CASE import set up to work on individual framework URIs; you have to enter in the endpoint of the CASE instance as Mike noted that Jeff did, and then select frameworks from the available list. I agree that we need paging/filtering here.

[jeannekitchens]

Right, it's been awhile since I've seen what that looks like when the CASE instance endpoint is entered. So the problem is finding a framework when there's a long list not that the import isn't working, correct?

[woodkri]

Correct.

[mparsons-ce]

@woodkri Is this a feature that can be added to the CaSS tool? If so, would there be a timeline for inclusion? Jeff Grann is meeting with the OpenSalt group to discuss their frameworks, and would like a sense of what might be possible, including paging/filtering or entering exact framework URLs.

[woodkri]

Yes, I would like to at a minimum add paging. I'd like to add support for exact framework URLs as well but this may be trickier since some of the ones I've seen out in the wild don't have the JSON available at the framework URL. I'd have to do some more investigation of patterns being followed for these to determine feasibility.

In terms of timeline, I would guess this is a lower priority than Collections?

[jeannekitchens]

@woodkri Collections are higher priority for sure.

[mparsons-ce]

@gloverkari We have a partner from North Dakota that is trying to import one of their CASE frameworks, and are unable to as only one page of results show. You may not be familiar with this request yet, but could we get an estimate of the effort involved? @jeannekitchens We will need to decide where this is lies in the priority list.

[mparsons-ce]

@gloverkari I have been told that this is a high priority issue as it is important to a state.

[jeannekitchens]

@gloverkari the imports from OpenSalt need both a paging and a filtering option. Imagine having to page through a couple hundred or thousand frameworks. It would be untenable. Users need to be able to page so it's an option when there's not a very large quantity and they need a filter to be able to find the framework. The filter should allow either a name or the unique identifier.

We have a user with the state of North Dakota that needs to import but my understanding is that they cannot get to the framework as it doesn't appear because CASS is only displaying the first 100 frameworks for publishing.

[gloverkari]

As a short-term fix while we are implementing paging or filtering, I set OpenSalt's 'limit' query parameter to 1000 (default is 100) and was able to retrieve all of the frameworks.

[jeannekitchens]

Thank you @gloverkari!

[jmarks]

Paging the CFDocs list is needed but you might also filter or sort on the Publisher element like the home page of OpenSALT.

I will note some OpenSALT servers have thousands of frameworks. There is no limit.

[mparsons-ce]

@jmarks I did try using the filter parameter in a search, and it seems to be ignored. I didn't see any documentation on how to format a filter. Do you happen to know the syntax? I tried plain text https://case.nd.gov/ims/case/v1p0/CFDocuments?limit=50&offset=0&filter=Dance I tried JSON, including text that I would not expect to be found. https://case.nd.gov/ims/case/v1p0/CFDocuments?limit=50&offset=0&filter={"title":"north dakotaXXX"}

I always get 298 results, so clearly the filter parameter is being ignored. Any help would be appreciated.

[jmarks]

The easy way to solve this is to enable entering the URI for a specific document and package, e.g. https://case.nd.gov/uri/p07cb92d2-c8ec-11ed-a8b8-0242ac1f0002 in the case of OpenSALT.

with the case formal format of that get call being: https://case.nd.gov/ims/case/v1p0/CFPackages/07cb92d2-c8ec-11ed-a8b8-0242ac1f0002

[roverwolf]

OpenSALT does not currently support filtering when using the API (no one has wanted to use it previously).

As has been noted, pagination is supported using the offset and limit query parameters.

[jmarks]

As another option could you support uploading the CASE JSON file directly?

[mparsons-ce]

@jmarks The production CaSS application remote file import has been updated to have a page size of 1000. So now use the ND case endpoint (https://case.nd.gov) will show all 298 frameworks.

[gloverkari]

Updated the remote server import to use paging when retrieving OpenSalt frameworks. The import will attempt to retrieve all available frameworks from the specified endpoint.

[gloverkari]

I confirmed that importing from the referenced url (https://sandbox.opensalt.net/) now retrieves 233 results, instead of the original 100.

[sedula27]

Verified when importing from open salt shows all the frameworks, This issued can be closed