LaunchDarkly KV lookups don't work within Cloudflare Miniflare V3

[JordanFaust]

Describe the bug When ran within Miniflare V3 the KV lookup on the key fails. Not matter the content written to the miniflare KV namespace the lookup fails.

To reproduce

Setup a simple hono app using the cloudflare-server-sdk

import { init } from '@launchdarkly/cloudflare-server-sdk';
import { deserializePoll } from '@launchdarkly/js-server-sdk-common';
import { BasicLogger, EdgeFeatureStore, LDClient } from '@launchdarkly/js-server-sdk-common-edge';
import { Context, Hono } from 'hono';

const keySuffix = 'maintenance.enabled';
const ctx = { kind: 'user', key: 'maintenance' };

class KVWrapper {
  constructor(private readonly kv: KVNamespace) {
    this.kv = kv;
  }

  // Wrapping this works for some reason. I don't know if there is a mismatch in the override function on the KVNamespace type. Miniflare maps over the types and converts them to a new type. Not sure if this is missed and this fails without wrapping it with something like this. b
  public async get(rootKey: string): Promise<string | null | undefined> {
    console.log(`attempting to get: ${rootKey} with suffix .flags`)
    const result = await this.kv.get(`${rootKey}.flags`)
    console.log(`get(rootKey): ${JSON.stringify(deserializePoll(result!))}`)
    return result
  }
}

export async function maintenance(c: Context, next: Next): Promise<Response | void> {
  let ldclient = c.get('ldclientl');
  if (ldclient == undefined) {
    const ldclient = init(c.env.LD_CLIENT_SIDE_ID, c.env.ld_kv, {});
    await ldclient.waitForInitialization();
    c.set('ldclient', ldclient);
  }

  const key = `${c.env.NAME}.${keySuffix}`;
  console.log(`looking up LD key: ${key}`)
  const maintenanceFlag = await ldclient.variation(key, ctx, false);
  // Get the data directly from KV
  const directKey = `LD-Env-${c.env.LD_CLIENT_SIDE_ID}.flags`
  const kvDirectFlag = await c.env.ld_kv.get(directKey)
  // Use the KVWrapper and use EdgeFeatureStore directly
  const logger = new BasicLogger({ level: 'debug' })
  const fs = new EdgeFeatureStore(new KVWrapper(c.env.ld_kv), c.env.LD_CLIENT_SIDE_ID, 'Cloudflare', logger)
  const _result = await fs.get({ namespace: 'features'}, key, (res) => console.log(`get result: ${JSON.stringify(res)}`))
  console.log("used edge feature store")
  console.log(`direct flag value is flag: ${directKey} value: ${JSON.parse(kvDirectFlag)}`);
  console.log(`deserializedPoll value: ${JSON.stringify(deserializePoll(kvDirectFlag))}`);

  if (maintenanceFlag) {
    return c.json({ "maintenance": true}, 503)
  }

  await next();

}

const app = new Hono();
app
  .all('*', maintenance)
  .all('*', async (c: Context) => fetch(c.req.raw)));

export default app

Setup miniflare to use the worker and seed expected flags within the KV:

import { Log, LogLevel, Miniflare } from 'miniflare';
import { packageDirectory } from 'pkg-dir';

// The path to the current npm package containing the worker
const packageRoot = await packageDirectory();
// The path to the precompiled worker
const scriptPath = `${packageRoot}/dist/worker.js`;

// Worker config
const name = process.env.NAME || 'zone-gateway-worker-local';
const host = process.env.HOST || '0.0.0.0';
const port = parseInt(process.env.PORT || '8080');
const clientSideId = 'abc123';

// Create a new Miniflare instance, starting a workerd server
const mf = new Miniflare({
  name: name,
  modules: true,
  scriptPath: scriptPath,

  host: host,
  port: port,
  compatibilityDate: '2023-08-14',
  compatibilityFlags: ['nodejs_compat'],

  log: new Log(LogLevel.DEBUG),

  // Environment Configurtion
  bindings: {
    NAME: name,
    // Add the Client Side ID used as the suffix in the KV key
    LD_CLIENT_SIDE_ID: clientSideId,
  },

  // Add the LaunchDarkly KV Namespace
  kvNamespaces: ['ld_kv'],
});

// mock adding expected flag values within the KV
const ns = await mf.getKVNamespace('ld_kv')
const maintenanceFlag = `${name}.maintenance.enabled`
const flags = {"flags": {}}
flags["flags"][maintenanceFlag] = {"key":`${maintenanceFlag}`, "on":false, "variations":[true,false]}
console.log(`flags: ${JSON.stringify(flags)}`)
const ldKey = `LD-Env-${clientSideId}.flags`
await ns.put(ldKey, JSON.stringify(flags))
console.log("KV List result:")
console.log(await ns.list())
console.log(`KV.get(${ldKey}) result:`)
console.log(await ns.get(ldKey))

console.log(`${name} started on ${host}:${port}`);

Expected behavior

The launchdarkly SDK should work with the local mocking of KVNamespaces within Miniflare.

Logs If applicable, add any log output related to your problem.

[mf:inf] Ready on http://0.0.0.0:8080
[mf:inf] - http://127.0.0.1:8080
[mf:inf] - http://10.244.0.37:8080
flags: {"flags":{"zone-gateway-worker-local.maintenance.enabled":{"key":"zone-gateway-worker-local.maintenance.enabled","on":false,"variations":[true,false]}}}
KV List result:
{
  keys: [ { name: 'LD-Env-abc123.flags' } ],
  list_complete: true,
  cacheStatus: null
}
KV.get(LD-Env-abc123.flags) result:
{"flags":{"zone-gateway-worker-local.maintenance.enabled":{"key":"zone-gateway-worker-local.maintenance.enabled","on":false,"variations":[true,false]}}}
zone-gateway-worker-local started on 0.0.0.0:8080
setting client side ID to: abc123
Using LD options: {"stream":false,"sendEvents":false,"useLdd":true,"diagnosticOptOut":true,"logger":{"logLevel":0,"name":"LaunchDarkly"},"featureStore":{"edgeProvider":{},"sdkKey":"abc123","description":"Cloudflare","logger":{"logLevel":0,"name":"LaunchDarkly"},"rootKey":"LD-Env-abc123"}}
Using LD options: {"stream":false,"sendEvents":false,"useLdd":true,"diagnosticOptOut":true,"logger":{"logLevel":0,"name":"LaunchDarkly"},"featureStore":{"edgeProvider":{},"sdkKey":"abc123","description":"Cloudflare","logger":{"logLevel":0,"name":"LaunchDarkly"},"rootKey":"LD-Env-abc123"}}
looking up LD key: zone-gateway-worker-local.maintenance.enabled
Requesting zone-gateway-worker-local.maintenance.enabled from LD-Env-abc123.flags
Error: LD-Env-abc123.flags is not found in KV.
    at Xa.get (file:///usr/src/app/dist/worker.js:34666:15)
Requesting zone-gateway-worker-local.maintenance.enabled from LD-Env-abc123.flags
KVWrapper lookup on LD-Env-abc123.flags
KVWrapper result: {"flags":{"zone-gateway-worker-local.maintenance.enabled":{"key":"zone-gateway-worker-local.maintenance.enabled","on":false,"variations":[true,false]}}}
get result: {"key":"zone-gateway-worker-local.maintenance.enabled","on":false,"variations":[true,false]}
used edge feature store
direct flag value is flag: LD-Env-abc123.flags value: "{\"flags\":{\"zone-gateway-worker-local.maintenance.enabled\":{\"key\":\"zone-gateway-worker-local.maintenance.enabled\",\"on\":false,\"variations\":[true,false]}}}"
deserializedPoll value: {"flags":{"zone-gateway-worker-local.maintenance.enabled":{"key":"zone-gateway-worker-local.maintenance.enabled","on":false,"variations":[true,false]}}}

SDK version

  "dependencies": {
    "@cloudflare/workers-types": "^4.20240314.0",
    "@launchdarkly/cloudflare-server-sdk": "^2.4.0",
    "@launchdarkly/js-sdk-common": "^2.2.0",
    "@launchdarkly/js-server-sdk-common": "^2.2.1",
    "@launchdarkly/js-server-sdk-common-edge": "^2.2.0",
    "@microlabs/otel-cf-workers": "1.0.0-rc.20",
    "@opentelemetry/sdk-trace-base": "^1.18.1"
  }

Language version, developer tools

  "dependencies": {
    "hono": "^3.12.10",
    "tsconfig-paths": "^4.2.0",
    "typescript": "^5.4.2",
    "vitest": "^1.3.1",
    "workerd": "^v1.20240314.0",
    "wrangler": "^3.30"
  },

OS/platform

WorkerD/Cloudflare. See above

Additional context

I think the problem is the EdgeProvider interface defined here:

export interface EdgeProvider {
  get: (rootKey: string) => Promise<string | null | undefined>;
}

The above example forcable wraps the passed KVNamespace used within the Miniflare V3 environment. This forcful wrapping solves the problem as seen from the log output:

KVWrapper lookup on LD-Env-abc123.flags
KVWrapper result: {"flags":{"zone-gateway-worker-local.maintenance.enabled":{"key":"zone-gateway-worker-local.maintenance.enabled","on":false,"variations":[true,false]}}}
get result: {"key":"zone-gateway-worker-local.maintenance.enabled","on":false,"variations":[true,false]}

I don't know if this is a miss on the Miniflare V3 side or within this SDK. I don't see an override for KV that only accepts a single parameter within worker-types but several accept optional parameters beyond the key. Miniflare to my understanding maps over those types to keep the API the same.

[yusinto]

We are investigating this. Internally logged as 237244.

[yusinto]

@JordanFaust looking at your code, it looks like you set up your mock data incorrectly:

    // This is incorrect
    // const ldKey = `LD-Env-${clientSideId}.flags`

    // The correct way is to just use `LD-env-clientSideId` without .flags
    const ldKey = `LD-Env-${clientSideId}`;
    await ns.put(ldKey, JSON.stringify(flags));

See index.test.ts where we use miniflare v2 to setup mock data. I don't think this is a v2/v3 issue. We will update to v3 in time separately.

Please try this and if you still have questions and issues, please reach out to our support team.

[JordanFaust]

You are right I got this working. Closing this out.