Feature request: Dedup same-key loader

[renatomariscal]

The Loader function is called on each cache miss:

sequenceDiagram
    %% Request 1
    Service->>+CacheAPI: get X
    CacheAPI->>+CacheMemory: get x
    CacheMemory->>-CacheAPI: miss
    CacheAPI->>+Loader: load X
    Loader->>+Resource: load x

    %% Response 1
    Resource->>-Loader: value X
    Loader->>+CacheAPI: set X
    CacheAPI->>+CacheMemory: set X
    CacheMemory->>-CacheAPI: 
    CacheAPI->>-Loader: 
    Loader->>-CacheAPI: value X
    CacheAPI->>-Service: value X

but there is cached signal that such key is already being loaded.

If the resource behind the loader is overwhelmed and slowing down, it will increase the parallelism on these scenarios, which increases the load, therefore making it worse (cascading failure).

This scenario is most likely on a cold-start.

Diagram showing 2 parallel loads of the same key, it could be N:

sequenceDiagram
    %% Request 1
    Service->>+CacheAPI: get X
    CacheAPI->>+CacheMemory: get x
    CacheMemory->>-CacheAPI: miss
    CacheAPI->>+Loader: load X
    Loader->>+Resource: load x

    %% Request 2
    Service->>+CacheAPI: get X
    CacheAPI->>+CacheMemory: get x
    CacheMemory->>-CacheAPI: miss
    CacheAPI->>+Loader: load X
    Loader->>+Resource: load x

    Note right of Resource: 2 parallel request on the resource

    %% Response 1
    Resource->>-Loader: value X
    Loader->>+CacheAPI: set X
    CacheAPI->>+CacheMemory: set X
    CacheMemory->>-CacheAPI: 
    CacheAPI->>-Loader: 
    Loader->>-CacheAPI: value X
    CacheAPI->>-Service: value X

    %% Response 2
    Resource->>-Loader: value X
    Loader->>+CacheAPI: set X
    CacheAPI->>+CacheMemory: set X
    CacheMemory->>-CacheAPI: 
    CacheAPI->>-Loader: 
    Loader->>-CacheAPI: value X
    CacheAPI->>-Service: value X

If the cache would internally hold a sort of promise:

sequenceDiagram
    %% Request 1
    Service->>+CacheAPI: get X
    CacheAPI->>+CacheMemory: get x
    CacheMemory->>-CacheAPI: miss
    CacheAPI->>+CacheMemory: set X (promise)
    CacheMemory->>-CacheAPI: 
    CacheAPI->>+Loader: load X
    Note right of CacheAPI: req 1 wait for promise
    Loader->>+Resource: load x

    %% Request 2
    Service->>+CacheAPI: get X
    CacheAPI->>+CacheMemory: get x
    CacheMemory->>-CacheAPI: found X (pending)
    Note right of CacheAPI: req 2 wait for promise

    %% Response 1
    Resource->>-Loader: value X
    Loader->>-CacheAPI: value X

    Note right of CacheAPI: resolve promise,<br/>req1 and req2 are unblocked

    CacheAPI->>-Service: value X

    %% Response 2
    CacheAPI->>-Service: value X

[swithek]

Perhaps SuppressedLoader might be something you're looking for?

[renatomariscal]

🤦🏻‍♂️ Thanks @swithek, I had skimmed over the documentation, and had assumed by the name this would suppress the loader on some scenarios.

[XuefengLiVincent]

@swithek Can you please guide me on this? Thanks I must have missed something

1. Late loader supposes to return an Item pointer.
2. The Item is sorta immutable obj with no public constructor how do I create an item instance to return in my Loader func?

[swithek]

Something like this should work:

func main() {
    loader := ttlcache.LoaderFunc[string, string](
        func(c *ttlcache.Cache[string, string], key string) *ttlcache.Item[string, string] {
            item := c.Set("key from file", "value from file") // create/set an item
            return item // pass it through the cache - it won't be set again
        },
    )
    cache := ttlcache.New[string, string](
        ttlcache.WithLoader[string, string](ttlcache.NewSuppressedLoader(loader, nil)),
    )

    item := cache.Get("key from file")
}

[renatomariscal]

@swithek but the official (internal) constructor of the Item constructor, has a touch on its initialization, won't it cause some trouble with the internal state to be missing that, such as by being considered expired right away?

https://github.com/jellydator/ttlcache/blob/6c6fce49fcd399ef4bb1f0b6cc14aee7c09ed4cf/item.go#L50

[swithek]

touch() in that case only initialises the expiration date (if enabled), it doesn't do any additional processing and the item that is returned from the loader is immediately returned from the Get() method as well: https://github.com/jellydator/ttlcache/blob/v3/cache.go#L229

[XuefengLiVincent]

Thanks @swithek ! Via Set function. As every loader must insert the item to cache. It would be good to have a better name. Hdyt? We can probably improve our implementation later @renatomariscal

[swithek]

As every loader must insert the item to cache. It would be good to have a better name.

Can you explain this more?