Platform context fallbacks

[hmnd]

Describe the problem

• When developing with svelte-kit dev, the event.platform object is always empty, with no great way to mock it.
• When building using an adapter like Cloudflare Workers, event.platform is empty for prerendering.

Describe the proposed solution

Ability to provide a platform object in svelte.config.js that is

• Substituted for event.platform when event.platform is undefined.
• Merged with event.platform, useful for specifying optional/default platform values.
• Both?

Alternatives considered

Perhaps allowing a 'transform' function that accepts an event may be better, in case platform needs to change based on inputs?

Importance

would make my life easier

Additional Information

No response

[dominikg]

relatated #2304

ideally adapters come with their own dev setup where needed, eg miniflare for cf

[Rich-Harris]

Yeah, this is one facet of a much larger topic that also includes #3535. The platform shouldn't be provided by svelte.config.js, it should be provided by adapters (which could expose options for controlling how the adapter constructs the default). But when we get to the specifics (e.g. exposing KV bindings and DO namespaces during development with the Cloudflare adapter) it doesn't actually get us very far by itself.

[TheHadiAhmadi]

ideally adapters come with their own dev setup where needed, eg miniflare for cf

what do you think if developer specify the contents of platform?

for example I need a database for my project. for Deno I can set database in platform object that can use Mongodb driver from deno.land like below:

// adapter.deno.js

// PRODUCTION mode 
// we have access to Deno,WebSocket,crypto.... in deno

const platform = {
  db: {
    get(id) {...},
    insert(data) {...}
    update(id, data) {...},
    remove(id) {...}
  }
}

export default platform;

then I can import and use this file in server.js

and for dev-mode I can simulate this functionality in handle (hooks.js) using in-memory or filesystem based database.

// hooks.js

export async function handle ({event, resolve }) {
  if(!platform.db) { // DEV mode
    platform.db = {
      get(id) {...},
      insert(data) {...},
      update(id, data) {...},
      remove(id) {...}
    }
  }
  ....

}

this way our sveltekit project is not dependent to cloudflare/deno and we can always move from one provider to another because the developer is the creator of this abstraction.

// adapter.cloudflare.js

const platform = {
  db: {/*TODO: same abstraction using cloudflare's KV or DO.*/}
}

export default platform;

Ability to provide a platform object in svelte.config.js

yes, this way we can manage platform object from svelte.config.js instead of hooks.js and we can provide different implementations for different adapters

[denizkenan]

My proposal would be, having adapters optionally provide an adapt_dev function similar to adapt where adapter author provides a mocked platform before request is sent for render. Setting up mocked platform should be adapter specific though.

I have managed fix this issue when using adapter-cloudflare-workers. it is a great workaround that served me well so far.

in hooks I use:

//src/hooks/index.ts
export const interceptPlatform: Handle = async ({ event, resolve }) => {
  event.platform = await cloudflareAdapterPlatform(event.platform)
  return resolve(event)
}
...

export const handle: Handle = sequence(interceptPlatform, ...)

Every request is intercepted by following code:

//src/cloudflareAdapterPlatform.ts

import { dev } from '$app/env';
import type { CFENV } from '../../app';

let context:ExecutionContext
const exposeCFGlobals = (globalObjects:object,ctx:ExecutionContext)=>{
    Object.entries(globalObjects).forEach(([key,val])=>{
         global[key]=val;
    })
    context = ctx;
}
const fn = (ctx:ExecutionContext) => {
        exposeCFGlobals({crypto},ctx)
        return;
}
export default async (_platform:App.Platform) => {
        if(!dev){
            return _platform;
        }
        if(_platform){
            return _platform;
        }

        console.log("!!!!!INITIALIZED!!!!!")
        const dotenv = await import("dotenv");
        const esbuild = await import("esbuild")
        const path = await import("path")
        const toml = await import("toml")
        const fs = await import("fs");
        const sourcefile = path.join(process.cwd(),"/durable_objects/src/objects.ts")
        const tsconfpath = path.join(process.cwd(),"/tsconfig.json")
        const wranglerPath = path.join(process.cwd(),"/wrangler.toml")
        const sourceCode = fs.readFileSync(sourcefile).toString('utf8')
        const tsconfigRaw = fs.readFileSync(tsconfpath).toString('utf8')
        const wranglerConfigRaw = fs.readFileSync(wranglerPath).toString('utf8')
        const wranglerConfig = toml.parse(wranglerConfigRaw)
        const bindings = wranglerConfig?.vars ??{}
        const durableObjects = (wranglerConfig?.durable_objects?.bindings ?? []).reduce((p,{name,class_name})=>{
            p[name] = class_name;
            return  p;
        },{})

        const {code } =  esbuild.transformSync(
            `
            const fn =  ${fn.toString()};
            export default {
                fetch: async (request, env2, ctx2) => {
                    fn(ctx2);
                    return new Response("Hello Miniflare!");
                }
            };
            ${sourceCode}
            `
            ,{
            loader: 'ts',
            sourcefile,
            tsconfigRaw,
            sourcemap:"inline"
        });
        const {parsed} =  dotenv.config()
        const miniflare = await (await import("miniflare")).Miniflare;
        const mf = new miniflare({
            modules:true,
            durableObjectsPersist:true,
            wranglerConfigPath:false,
            envPath:path.join(process.cwd(),"/.env"),
            script: code,
            durableObjects,
            bindings,
            globals:{exposeCFGlobals}
            })
            await mf.dispatchFetch("https://host.tld")
            const env = {...parsed, ...bindings} as unknown as CFENV;
            for await (const [k,_] of Object.entries(durableObjects)){
                env[k] = await mf.getDurableObjectNamespace(k) as DurableObjectNamespace;
            }  

        const platform:App.Platform = {env,context}
        return platform;

    }

In brief cloudflareAdapterPlatform.ts:

• checks if it is called in dev mode if not exits without any action
• does platform specific logic.
  • spin up miniflare.
  • load Durable Objects
  • expose runtime specific modules back to node such as Crypto
  • load environment variables defined in wrangler.toml
  • etc..
  • attach mocked platform to event

This logic should be part of adapter and every adapter should fulfil its platform specific requirements. I understand that there is a desire for keeping codebase free from adapter specific logic. However, I don't see this happening when endpoint events expose adapter specific platform.

[RicardoViteriR]

Hi @denizkenan, your looks great! Would you by any chance have an public repo with this implementation available? I am trying to figure out how you are piecing everything together but I can't figure it out.

[kalepail]

Hi @denizkenan, your looks great! Would you by any chance have an public repo with this implementation available? I am trying to figure out how you are piecing everything together but I can't figure it out.

It's not perfect but here's what I've been using for my base repo: https://github.com/tyvdh/test-kit

[denizkenan]

Hi @denizkenan, your looks great! Would you by any chance have an public repo with this implementation available? I am trying to figure out how you are piecing everything together but I can't figure it out.

It's not perfect but here's what I've been using for my base repo: https://github.com/tyvdh/test-kit

Ah nice, I see that you have wrapped it as boilerplate(also added cache).

[CanRau]

👍🏼 yes would be amazing to have, when I played with SolidStart couple weeks ago I was amazed that dev already uses miniflare under the hood which makes it much simpler to develop code depending on R2 etc

[UnlimitedBytes]

Hi there,

I understand the frustration expressed in the original post regarding the inability to use platform specific context when in development mode. While there are some workarounds available, such as running the building process in watch mode and using the development software provided by the platform on the build output, they fall short of providing a seamless developer experience, especially with regards to features like HMR.

As someone who has developed applications that rely on the platform attribute, I can attest to the pain of not being able to use it in development mode. It's disappointing that Svelte, which is known for its developer-friendliness, has not yet found a solution to this issue, especially when other frameworks have already done so.

I would like to voice my support for finding a solution to this issue and urge the Svelte team to address it as soon as possible. Please let me know when there are any kind of solutions to this yet.

Thank you.

[UnlimitedBytes]

For Cloudflare Pages / Workers it would be possible to use Miniflare to archive this. I build a basic sample showcasing how fallbacks can already be implemented using this, it would still be awesome to have this already povided by @sveltejs/adapter-cloudflare.

// /lib/server/miniflare.ts
type StorageOptionsMemory = {
    type: 'memory';
};

type StorageOptionsFile = {
    type: 'file';
    path: string;
};

export type StorageOptions = StorageOptionsMemory | StorageOptionsFile;

export const createCache = async (storageOptions: StorageOptions) => {
    const { Cache } = await import('@miniflare/cache');

    if (storageOptions.type === 'memory') {
        const { MemoryStorage } = await import('@miniflare/storage-memory');
        return new Cache(new MemoryStorage());
    } else if (storageOptions.type === 'file') {
        const { FileStorage } = await import('@miniflare/storage-file');
        return new Cache(new FileStorage(storageOptions.path));
    }

    throw new Error('StorageType not found');
};

export const createD1 = async (storageOptions: StorageOptions) => {
    const { createSQLiteDB } = await import('@miniflare/shared');
    const { D1Database, D1DatabaseAPI } = await import('@miniflare/d1');

    if (storageOptions.type === 'memory') {
        const sqliteDb = await createSQLiteDB(':memory:');
        return new D1Database(new D1DatabaseAPI(sqliteDb));
    } else if (storageOptions.type === 'file') {
        const sqliteDb = await createSQLiteDB(storageOptions.path);
        return new D1Database(new D1DatabaseAPI(sqliteDb));
    }

    throw new Error('StorageType not found');
};

export const createR2 = async (storageOptions: StorageOptions) => {
    const { R2Bucket } = await import('@miniflare/r2');

    if (storageOptions.type === 'memory') {
        const { MemoryStorage } = await import('@miniflare/storage-memory');
        return new R2Bucket(new MemoryStorage());
    } else if (storageOptions.type === 'file') {
        const { FileStorage } = await import('@miniflare/storage-file');
        return new R2Bucket(new FileStorage(storageOptions.path));
    }

    throw new Error('StorageType not found');
};

export const createKV = async (storageOptions: StorageOptions) => {
    const { KVNamespace } = await import('@miniflare/kv');

    if (storageOptions.type === 'memory') {
        const { MemoryStorage } = await import('@miniflare/storage-memory');
        return new KVNamespace(new MemoryStorage());
    } else if (storageOptions.type === 'file') {
        const { FileStorage } = await import('@miniflare/storage-file');
        return new KVNamespace(new FileStorage(storageOptions.path));
    }

    throw new Error('StorageType not found');
};

export const createDOStorage = async (storageOptions: StorageOptions) => {
    const { DurableObjectStorage } = await import('@miniflare/durable-objects');

    if (storageOptions.type === 'memory') {
        const { MemoryStorage } = await import('@miniflare/storage-memory');
        return new DurableObjectStorage(new MemoryStorage());
    } else if (storageOptions.type === 'file') {
        const { FileStorage } = await import('@miniflare/storage-file');
        return new DurableObjectStorage(new FileStorage(storageOptions.path));
    }

    throw new Error('StorageType not found');
};

// /src/hooks.server.ts
import { dev } from '$app/environment';
import { createKV, createD1 } from '$lib/server/miniflare';

export const handle = ({ event, resolve }) => {
    if (dev) {
        // We fake the platform for local development.
        event.platform ??= {
            env: {
                COUNTER: createKV({ type: 'file', path: '.mf/kv-counter' }),
                DATABASE: createD1({ type: 'file', path: '.mf/d1-database.sqlite3' }),
            },
        };
    }
    return resolve(event);
};

[hd-4]

I used the same technique to fill out the platform in dev for a custom adapter I was working with:

export async function handle({ event }) {
    if (dev) {
        const dev_platform = await import('./dev/platform.server.js');
        event.platform = dev_platform.platform;
    }
}

[carsonmccue]

What's the proper way to define classes for durable objects using SvelteKit and the modular @miniflare libraries?

[55lj]

@UnlimitedBytes thanks for sharing. Would it also be possible to access remote D1 dbs from wrangler pages dev (assuming I just use another remote D1 as dev instance next to my production D1)?

[shivan-s]

@UnlimitedBytes thanks for sharing. Would it also be possible to access remote D1 dbs from wrangler pages dev (assuming I just use another remote D1 as dev instance next to my production D1)?

I wonder about this also. Following.

[UnlimitedBytes]

@UnlimitedBytes thanks for sharing. Would it also be possible to access remote D1 dbs from wrangler pages dev (assuming I just use another remote D1 as dev instance next to my production D1)?

Yes, it would be possible. Although D1 is intended to be only used from Cloudflare Workers or Cloudflare Pages Functions you can also access it from any place you like. In order to achieve something like you described you would need to mock the whole D1 Object in your local environment and let it run the queries against Cloudflare's D1 API Endpoints.

It is worth nothing though that Cloudflare currently (as D1 is only intended to be used from workers/pages and it's in alpha) does not document their D1 API Endpoints. So you will need to "reverse engineer" it from the wrangler command-line tool which uses this endpoints. Another project you can take a look at is D1-Console which extracted most of the D1 parts out of wrangler.

Sadly I cannot provide an example for this specific use-case as it's not mine and I don't have enough time to spear to develop things I don't need myself.

[jahir9991]

wrangler pages dev .svelte-kit/cloudflare --local --d1=local_db vite build -w These two commands worked for me when I run both parallelly

[xdivby0]

For Cloudflare Pages / Workers it would be possible to use Miniflare to archive this. I build a basic sample showcasing how fallbacks can already be implemented using this, it would still be awesome to have this already povided by @sveltejs/adapter-cloudflare.

// /lib/server/miniflare.ts
type StorageOptionsMemory = {
    type: 'memory';
};

type StorageOptionsFile = {
    type: 'file';
    path: string;
};

export type StorageOptions = StorageOptionsMemory | StorageOptionsFile;

export const createCache = async (storageOptions: StorageOptions) => {
    const { Cache } = await import('@miniflare/cache');

    if (storageOptions.type === 'memory') {
        const { MemoryStorage } = await import('@miniflare/storage-memory');
        return new Cache(new MemoryStorage());
    } else if (storageOptions.type === 'file') {
        const { FileStorage } = await import('@miniflare/storage-file');
        return new Cache(new FileStorage(storageOptions.path));
    }

    throw new Error('StorageType not found');
};

export const createD1 = async (storageOptions: StorageOptions) => {
    const { createSQLiteDB } = await import('@miniflare/shared');
    const { D1Database, D1DatabaseAPI } = await import('@miniflare/d1');

    if (storageOptions.type === 'memory') {
        const sqliteDb = await createSQLiteDB(':memory:');
        return new D1Database(new D1DatabaseAPI(sqliteDb));
    } else if (storageOptions.type === 'file') {
        const sqliteDb = await createSQLiteDB(storageOptions.path);
        return new D1Database(new D1DatabaseAPI(sqliteDb));
    }

    throw new Error('StorageType not found');
};

export const createR2 = async (storageOptions: StorageOptions) => {
    const { R2Bucket } = await import('@miniflare/r2');

    if (storageOptions.type === 'memory') {
        const { MemoryStorage } = await import('@miniflare/storage-memory');
        return new R2Bucket(new MemoryStorage());
    } else if (storageOptions.type === 'file') {
        const { FileStorage } = await import('@miniflare/storage-file');
        return new R2Bucket(new FileStorage(storageOptions.path));
    }

    throw new Error('StorageType not found');
};

export const createKV = async (storageOptions: StorageOptions) => {
    const { KVNamespace } = await import('@miniflare/kv');

    if (storageOptions.type === 'memory') {
        const { MemoryStorage } = await import('@miniflare/storage-memory');
        return new KVNamespace(new MemoryStorage());
    } else if (storageOptions.type === 'file') {
        const { FileStorage } = await import('@miniflare/storage-file');
        return new KVNamespace(new FileStorage(storageOptions.path));
    }

    throw new Error('StorageType not found');
};

export const createDOStorage = async (storageOptions: StorageOptions) => {
    const { DurableObjectStorage } = await import('@miniflare/durable-objects');

    if (storageOptions.type === 'memory') {
        const { MemoryStorage } = await import('@miniflare/storage-memory');
        return new DurableObjectStorage(new MemoryStorage());
    } else if (storageOptions.type === 'file') {
        const { FileStorage } = await import('@miniflare/storage-file');
        return new DurableObjectStorage(new FileStorage(storageOptions.path));
    }

    throw new Error('StorageType not found');
};

// /src/hooks.server.ts
import { dev } from '$app/environment';
import { createKV, createD1 } from '$lib/server/miniflare';

export const handle = ({ event, resolve }) => {
    if (dev) {
        // We fake the platform for local development.
        event.platform ??= {
            env: {
                COUNTER: createKV({ type: 'file', path: '.mf/kv-counter' }),
                DATABASE: createD1({ type: 'file', path: '.mf/d1-database.sqlite3' }),
            },
        };
    }
    return resolve(event);
};

THIS has to be included in the docs. I can't explain how confused I was, just learning sveltekit but not being able to get it running locally. Somehow weird for being the "most-loved" framework. Simply add the above to the docs, it'll save sooo many people hours of research.

[eltigerchino]

Ideally, we'd just pass the namespaces to the adapter options such as:

// svelte.config.js
import adapter from '@sveltejs/adapter-cloudflare';

/** @type {import('@sveltejs/kit').Config} */
const config = {
    kit: {
        adapter: adapter({
            kvNamespaces: ['YOUR_KV_NAMESPACE'] // populates platform for us
        })
    }
};

export default config;

Otherwise, we could export helper methods as demonstrated in https://github.com/sveltejs/kit/issues/4292#issuecomment-1550596497

As a third option (or something we can add now), we can document the pattern below on using Miniflare directly:

// src/lib/dev/miniflare.js
import { Miniflare, Log, LogLevel } from 'miniflare';

// See https://latest.miniflare.dev/
// for further configuration

/** @type {import('miniflare').MiniflareOptions} */
const opts = {
    log: new Log(LogLevel.WARN),
    modules: true,
    script: `
    export default {
        fetch () {
            return new Response(null, { status: 404 });
        }
    }
    `,
    // These namespaces should also be added to
    // `app.d.ts` `App.Platform.env`
    // (and `wrangler.toml` if using wrangler)
    kvNamespaces: ['YOUR_KV_NAMESPACE'],
    kvPersist: './.mf/kv'
};

/**
 * @returns {Promise<App.Platform>}
 */
export async function getPlatform() {
    const mf = new Miniflare(opts);

    /** @type {App.Platform} */
    const platform = {
        env: await mf.getBindings()
    };

    return platform;
}

// src/hooks.server.js
import { dev } from '$app/environment';

/** @type {import('@sveltejs/kit').Handle} */
export async function handle({ event, resolve }) {
    if (dev) {
        const { getPlatform } = await import('$lib/dev/miniflare');
        event.platform ??= await getPlatform();
    }

    const response = await resolve(event);
    return response;
}

https://github.com/s3812497/sveltekit-cloudflare

[sdarnell]

I originally used a version based on: @UnlimitedBytes code https://github.com/sveltejs/kit/issues/4292#issuecomment-1550596497 but this stops working with the latest miniflare, and in a similar way to @s3812497 I've produced a version that uses the new v3 api. The key differences in my starter are:

• Reads configuration from wrangler.toml file (top level at least) so is generic
• Currently supports D1 and KV bindings (including multiple)
• Works with the same paths as wrangler, so you can mix wrangler local commands (e.g. migrations)
• Includes the cf properties object, using a cached cf.json that wrangler uses.

See https://github.com/sdarnell/cf-svelte/blob/main/src/lib/server/miniflare.ts

But for the more general issue, it would be really good if adapters could hook into the server hooks processing, and obviously if the adapter-cloudflare could include the above it would relieve a lot of pain when people try to use CF and Sveltekit together.

[ssbm-oro]

I originally used a version based on: @UnlimitedBytes code #4292 (comment) but this stops working with the latest miniflare, and in a similar way to @s3812497 I've produced a version that uses the new v3 api. The key differences in my starter are:

• Reads configuration from wrangler.toml file (top level at least) so is generic
• Currently supports D1 and KV bindings (including multiple)
• Works with the same paths as wrangler, so you can mix wrangler local commands (e.g. migrations)
• Includes the cf properties object, using a cached cf.json that wrangler uses.

See https://github.com/sdarnell/cf-svelte/blob/main/src/lib/server/miniflare.ts

But for the more general issue, it would be really good if adapters could hook into the server hooks processing, and obviously if the adapter-cloudflare could include the above it would relieve a lot of pain when people try to use CF and Sveltekit together.

I've been trying to get this working all morning and finally found my way to this post. However, I get the following error when running vite build:

[commonjs--resolver] Failed to resolve entry for package "fs". The package may have incorrect main/module/exports specified in its package.json.
error during build:
Error: Failed to resolve entry for package "fs". The package may have incorrect main/module/exports specified in its package.json.

I originally had a bunch of errors for missing basic modules, but i was able to npm i them all away, but this one I'm still stuck on. Any idea why it might be failing to build for me?

[sdarnell]

@ssbm-oro Could it be the version of node/npm that you're using? I happen to be using v16.20.0, though I'd advise going for v18. I just switched to v18 locally and it works for me. In you package.json file, do you have a line with: "type": "module" ? Also note that the miniflare.ts file must only be loaded on the server side (i.e. node). Which is why in my example, it is in a lib/server folder. One other thing to try, though not sure why it might be needed, see this SO answer: https://stackoverflow.com/questions/74743694/quasar-vite-failed-to-resolve-entry-for-package-fs-the-package-may-have-inc

[ssbm-oro]

@ssbm-oro Could it be the version of node/npm that you're using? I happen to be using v16.20.0, though I'd advise going for v18. I just switched to v18 locally and it works for me.

yeah, i'm using v18.6.0 already, so i don't think that's the issue.

In you package.json file, do you have a line with: "type": "module" ?

i do, should it be something else?

Also note that the miniflare.ts file must only be loaded on the server side (i.e. node). Which is why in my example, it is in a lib/server folder.

yeah, i have it in the same src/lib/server path, and i'm only referencing it from hooks.server.ts

One other thing to try, though not sure why it might be needed, see this SO answer: https://stackoverflow.com/questions/74743694/quasar-vite-failed-to-resolve-entry-for-package-fs-the-package-may-have-inc

i saw this post when i was doing my initial search, but i don't think i'm using Quasar and don't have a quasar.config.js.

A related issue discussion linked from that one suggested adding "rollup-plugin-node-builtins" as the alias to resolve this error, but doing this gave me the error when vite tries to load the config file:

failed to load config from ~/git/chatdraft/vite.config.ts
error during build:
TypeError: __require.resolve is not a function

Thanks for the suggestions!

[ssbm-oro]

I found something that works for me, at least, it'll let me continue developing and testing locally with the same code that'll run on Cloudflare in production. I created two NPM commands, one that just does vite build -w to watch for changes to my code, and the other that runs this command:

wrangler pages dev .svelte-kit/cloudflare --d1=DB -k KV -e dev

Not ideal since i'll need to keep my package.json and wrangler.toml in sync if i add/change any bindings, but it'd be a much better developer experience if the adapter could take care of these details.

[zwergius]

@ssbm-oro So I actually got all this working yesterday with a local db through mini flare and wrangler using dev build. Be minded I don't have KV setup, so removed this parts of the code.

Here is my start script to get it all running

"start": "wrangler pages dev --d1=DB --compatibility-date=2023-09-22 --ip '127.0.0.1' --proxy 5173 -- pnpm run dev"

I also needed to populate my local DB, I use this script: "seed:local:db": "wrangler d1 execute {db-name} --local --file schemas/dev-schema.sql"

[ssbm-oro]

@zwergius Thanks for the suggestion, this is similar to how I was trying to run it before. Maybe it's because I'm using npm instead of pnpm, but when I run the command like this instead of looking at the build output directory, the DB and KV objects are undefined.

[gerhardcit]

For the many others that will end up with this issue: I've tried wrangler-proxy with great success. https://www.npmjs.com/package/wrangler-proxy

There is pull request on svelte-demo-d1 which let you test this quickly: https://github.com/elithrar/svelte-demo-d1/pull/3

[longrunningprocess]

I was able to get this working today locally see: https://github.com/cloudflare/workers-sdk/issues/3622#issuecomment-1773909149

[gerhardcit]

@longrunningprocess , yes, that's possible, but it's not "dev" space.. that is testing of a build.. in a larger project, that is painful. The idea is to have platform bound when running: npm run dev

[eltigerchino]

https://github.com/sveltejs/kit/pull/11730 has been merged which opens the door for adapters to populate event.platform during vite dev and vite preview. Stay tuned for updates to the cloudflare adapters :)