Proper example for getMany

[tomcatmurr]

Hi guys, could you provide a proper example for getMany (with idResolver etc). Missing it here and in node-service-template (which is great btw). Aiming to implement it with mikro-orm.

I want to manage stuff like getAll or pagination, i.e. collections or complex objects while I am limited with Promise in get and Promise<LoadedValue[]> in getMany using DataSource interface for a loader.

The dumb way just to use get with keys like 'ALL' (for getAll) but yet in coflicts with interface.

[kibertoad]

Can you share more details about you use-case? What is the structure of your ids, and what would you like to compute it from?

Naive example from tests:

const idResolver: IdResolver<string> = (value) => {
  const number = value.match(/(\d+)/)?.[0] ?? ''
  return `key${number}`
}

const value = await operation.getMany(['key1', 'key2'], idResolver)

This resolver will extract first number sequence from a given string value and prefix it with key. More commonly you would have IdResolver<MyObject> and then concatenate different fields of it into what would be a unique id. In case your objects can be identified by just a single field (e. g. id), idResolver could be as simple as this:

const idResolver: IdResolver<MyObject> = (value: MyObject) => {
  return value.id
}

I will update our documentation, as you are right, this part is pretty confusing right now.

[tomcatmurr]

Thanks for quick reply. I am a bit confused as your getMany is actually focused on retrieving/caching multiple keys, or, with db example, something like this: select('*').whereIn('id', keys.map(parseInt)).

I am interested in trivial thing like cache the plain collection of records with some custom cacheKey, let's say menu list, where cacheKey is a simple string but the collection array requires me to use getMany according to DataSource interface.

Or cursor pagination results, where again, I want to make cacheKey totally customized for each data set. Probably need to create different loaders for the same entity as I am limited with DataSource interface.

[kibertoad]

Exactly, getMany is intended to be used for whereIn scenarios.

Can't you type your Loader as Loader<MyObject[]>? In that case you will be able to store multiple entries per key, if that is what you ultimately want.

[tomcatmurr]

Ok, imagine trivial scenario, let's say I have news entity, and I want various news collections for frontpage (3 items for instance), latest (10), latest per region (5 regions x 10 items), and then paginated (5 pages).

I will need to create bunch of loaders to have get(key) serve each usecase, while all I want is just specify custom cacheKey and let your machinery do the magic. Maybe sticking to strict DataSource interface is overkill here?

[kibertoad]

See above suggestion. My suggestion would be to define NewMultiLoader: Loader<NewsItem[]>, and pass loadParams with whatever pagination, date and region filter that you want. I guess separation between id and loadParams is the most confusing part right now. id is whatever unique identifier that is used for a key-value lookup in the cache, you can aggregate whatever you want into a single string there, could be a simple concatenation of all the query parameters involved. now loadParams is anything that you can use from datasource to actually load the value. It can be an object with a bunch of optional fields, so that you could customize your single datasource logic however you like.

[kibertoad]

See this example:

type MyLoaderParams {
  region?: string
  pageNum?: number
  pageSize?: number
}

export function resolveKey (params: MyLoaderParams): string {
  return `${params.region ?? '*'}-${params.pageNum ?? '*'}-${params.pageSize ?? '*'}`
}

class MyParametrizedDataSource implements DataSource<MyCacheValue[], MyLoaderParams> {
  async get(_key: string, params?: MyLoaderParams): Promise<MyCacheValue[] | undefined | null> {
    if (!params) {
      throw new Error('Params were not passed')
    }

    const resolvedValue = await someResolutionLogic(params.region, params.pageNum, params.pageSize)
    return resolvedValue
  }
}

[tomcatmurr]

Thanks, will try it. By making DataSource interface less strict I thought of making getMany optional.

[kibertoad]

@tomcatmurr You can simply do throw new Error('Not implemented') from it if you don't intend to use it.

[tomcatmurr]

@kibertoad, thanks for your assistance, everything works like a charm. Greetings from Kharkiv)

[kibertoad]

glad to hear that! and appreciate you reaching out, I'll update documentation to explain these more advanced use-cases with greater details.

Слава Україні!