Caching on /FeatureServer endpoint

[apatriz]

When using the /FeatureServer endpoint with the built-in memory cache, it doesn't look like this response is cached.

I can see that the responses are cached when hitting the 'FeatureServer//query' or 'FeatureServer/' endpoints, but not '/FeatureServer/layers' or '/FeatureServer'

i.e. url: myprovider/someid/FeatureServer?f=json will always call the getData function

My provider is similar to the koop-provider-csv plugin in that it has to fetch a file remotely. I would like to avoid multiple calls to fetch the file (especially it seems the ArcGIS API for JS 4.x calls the server info endpoint multiple times in a row when calling Layer.from()

It seems reasonable to cache this response but I'm not sure where to do that - do I need a custom cache plugin ? can I do it in the provider plugin ? Any insight is greatly appreciated

Update: After some research, i discovered the 'pullLayer' and 'pullCatalog' methods in koop-core, which seem to be exactly what I need to properly cache the calls to /FeatureServer, /FeatureServer/layers, and /FeatureServer/layer/

Do you think these will be leveraged here for those routes or are you open to PRs to implement them ?

[rgwozdz]

@apatriz I'm surprised that '/FeatureServer/layers' or '/FeatureServer' are not being cached. They should all pass through the same section of code that upsert data to the cache. Have you been able to debug to see why either the upsert is not happening or the subsequent retrieve is failing? (Note, this caching code is found in koop-core).

We're definitely open to PRs. @haoliangyu developed the pullLayer and pullCatalog methods and may have thoughts. I think the critical thing is probably to include a fallback to getData as not every provider will have implemented the other methods.

[haoliangyu]

The pullLayer and pullCatalog methods are designed to fetch the metadata of a catalog service (like FeatureSever) and its layers. In the current implementation, calling these methods will throw an error if the provider doesn't implement the getLayer and getCatalog. It could be possible to fall back to getData and provider minimal metadata.

[apatriz]

Thanks @rgwozdz @haoliangyu -- I did some more digging and I think the problem is actually in koop-cache-memory here -- the retrieve method doesn't account for cached {layers} -- it assumes cached objects are always features, which doesn't work when caching /FeatureServer and /FeatureServer/layers responses . I can open a separate issue there.

I'd like your opinions on how to implement the behavior I'm looking for if you have a chance:

my getData function downloads a KMZ file, parses it into multiple layers based on geometry type, and always builds geoJson in the following format

{
    layers,
    metadata: {
      name,
      description,
    },
    ttl: TTL,
}

It then checks the req.params.layer and returns the appropriate layer geoJSON if requested, otherwise it returns everything

What I want is to cache that entire object (each layer in the layers array also contains features), using just the req.params.id for createKey -- any subsequent requests for layers i.e. /FeatureServer/0, /FeatureServer/1, should check if that layer is stored in the cache first

This is to avoid having to download the KMZ file once for /FeatureServer and then once for each /FeatureServer/layerid , which is unnecessary network calls and processing time, especially when KMZ files are large

My initial thought for workaround is to call this.cache.retrieve in my getData function to handle getting the appropriate layer from the cache, but doesn't seem quite right to me.

Or overriding the pull method, but I also don't think that is quite right

Appreciate the time and any insight

[rgwozdz]

Sorry to have dropped off on this. Did you move forward with a workaround?

[rgwozdz]

Ah, I see you opened a PR some time ago. I will review and comment there.