[🐞] Cloudflare Pages: Ability to run cloudflare pages dev server with vite

[sujith97]

Which component is affected?

Qwik Rollup / Vite plugin

Describe the bug

The current Cloudflare pages setup makes it hard to develop qwik-city with Cloudflare page functions. Cloudflare has D1, KV etc objects that should be accessible during the development mode but at the moment we should first build. The build drops the static files to dist folder which can then be accessed using Cloudflare pages. Ideally, we should be able to have the HMR and wrangler reload during local development.

Can we provide guidance on how to develop locally with Cloudflare and qwik-city vite? (https://developers.cloudflare.com/pages/platform/functions/local-development/)

export const useListLoader = routeLoader$(async (context: any) => {
  const ps = (context.env.get("cloudflaredb") as any)?.prepare(
    "SELECT * FROM users"
  );
  if (ps) {
    const users = await ps.first();
    return {
      users,
    };
  } else {
    console.log("No access to D1", context.env.get("cloudflaredb"));
  }

  return {
    users: [],
  };
});

Reproduction

https://stackblitz.com/edit/qwik-starter-joopk8?file=src%2Froutes%2Findex.tsx

Steps to reproduce

No response

System Info

System:
    OS: macOS 13.0
    CPU: (8) arm64 Apple M1 Pro
    Memory: 279.02 MB / 32.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 18.15.0 - ~/.volta/tools/image/node/18.15.0/bin/node
    Yarn: 1.22.19 - ~/.volta/tools/image/yarn/1.22.19/bin/yarn
    npm: 9.5.0 - ~/.volta/tools/image/node/18.15.0/bin/npm
  Browsers:
    Chrome: 111.0.5563.64
    Safari: 16.1
  npmPackages:
    @builder.io/qwik: 0.20.1 => 0.20.1
    @builder.io/qwik-city: 0.5.2 => 0.5.2
    undici: 5.20.0 => 5.20.0
    vite: 4.1.4 => 4.1.4

Additional Information

No response

[stackblitz[bot]]

Start a new pull request in StackBlitz Codeflow.

• qwik-starter-joopk8 (Open reproduction)

[sujith97]

Solid start's Cloudflare adapter supports these options in vite configuration: link

+  plugins: [
+   solid({
+     adapter: cloudflare({
+       kvNamespaces: ["MY_KV"],
+       d1Databases: ["MY_DB"]
+     }),
+   }),
+  ],

We should have something similar to Qwik's Cloudflare pages adapter

[sujith97]

Workaround

1. Install following dev dependencies: npm i @miniflare/d1 @miniflare/shared

2. Create a separate file to fetch the DB instance:

   
   let getDevDb: any = () => {};

if (import.meta.env.DEV) { const { D1Database, D1DatabaseAPI } = await import("@miniflare/d1"); const { createSQLiteDB } = await import("@miniflare/shared");

let devDb: any;

getDevDb = async () => { if (!devDb) { const sqlLite = await createSQLiteDB( ".wrangler/state/d1/cloudflared1db.sqlite3" ); devDb = new D1Database(new D1DatabaseAPI(sqlLite)); } return devDb; }; }

const getDb = async (context: any) => { if (context.env.get("cloudflared1db")) { return context.env.get("cloudflared1db"); }

return getDevDb(); };

export default getDb;

3. Ignore these `@miniflare` deps in build
> /adapters/cloudflare-pages/vite.config.ts

rollupOptions: { input: ['src/entry.cloudflare-pages.tsx', '@qwik-city-plan'],

• external: ['@miniflare/d1', '@miniflare/shared'] },

That's it. Now whereever you need db instance just fetch it using await getDb(). Local dev server will use miniflare (local sqllite db) whereas the production builds use the actual cloudflare instances.

export const useListLoader = routeLoader$(async (context: any) => {
  const db = await getDb(context);

  if (db) {
    const ps = db.prepare("select * from users");
    const firstUser = (await ps.all())?.results?.[0];
    return {
      list,
      user: { // DB objects have some issue if we directly send them so create a new object
        ...firstUser,
      },
    };
  }
});

[mattp0123]

I have the same problem with KV. here's the workaround modified from @sujith97's:

// src/lib/kv.ts
import type { KVNamespace } from '@miniflare/kv';

let getDevKVNamespace: () => KVNamespace;

if (import.meta.env.DEV) {
  const { KVNamespace } = await import('@miniflare/kv');
  const { MemoryStorage } = await import('@miniflare/storage-memory');

  let kvNamespace: KVNamespace;

  getDevKVNamespace = () => {
    if (!kvNamespace) {
      kvNamespace = new KVNamespace(new MemoryStorage());
    }
    return kvNamespace;
  };
}

export const getKVNamespace = (
  platform: QwikCityPlatform,
  KVBindingKey: string
): KVNamespace => {
  if (platform.env) {
    return platform.env[KVBindingKey] as KVNamespace;
  }
  return getDevKVNamespace();
};

// src/routes/index.tsx
export const useData = routeLoader$(async ({ platform }) => {
  const MY_KV = getKVNamespace(platform, 'MY_KV');
  return await MY_KV.list();
});

[lyonsbp]

Has anyone been able to get this working with R2 yet as well or only KV?

[austin-raey]

I'd like to note that from the workaround comment above for D1, https://github.com/BuilderIO/qwik/issues/3345#issuecomment-1475385715

For whatever reason, my builds began to fail recently which had something to do with the miniflare package,

Putting this in the main vite.config.ts solved the build issue. :)

    build: {
        rollupOptions: {
            external: ['@miniflare/d1', '@miniflare/shared'],
        },
    },

As far as I can tell miniflare code is not being bundled, bundle sizes do not change even if I just comment out all the code between if (import.meta.env.DEV) stuff. So I'm not really entirely sure what is going on 🤔

For instance, if I were to completely remove the build.rollupOptions.external configuration in both places, and replace if (import.meta.env.DEV) { with if (false) { ..., vite seems to know not to bundle the miniflare code and there is no error. With import.meta.env.DEV there is one.

It seems I just have a gap in my knowledge for this lmao

[serbyxyz]

I also can not get this to work, the main issue I believe is the qwik-auth package, I have installed from npm run qwik add auth

this is the original way I had it set up :

export const { onRequest, useAuthSession, useAuthSignin, useAuthSignout } =
  serverAuth$(({ env, platform }) => ({
    secret: env.get("PRIVATE_AUTH_SECRET"),
    adapter: DrizzleAdapter(platform.env?.DB),

Is there a way to set default "Env" or "PLATFORM" to be of the type needed by miniflare or any other package ? instead of needing to call const db = await getDb(platform.env.YOUR_DB)

So that when the Vite npm run dev starts up it check for theimport.meta.DEV so it loads with the PLATFORM necessary ?

[mhevery]

Qwik-auth is very simple: https://github.com/BuilderIO/qwik/blob/main/packages/qwik-auth/src/index.ts But I am not an export on it. What needs to be changed there? Can you propose a PR?

[austin-raey]

@mhevery If I ever find the time I wouldn't mind looking into such a thing, although it does seem a little complex.

If you look at the source code for what solid-start does for Cloudflare Pages, it seems they apply some deeper integration with Miniflare.

This is what allows for the platform variable to have the appropriate Cloudflare bindings like D1, R2, KV etc in the dev environment i.e. npm run dev, like what @serbyxyz is asking.

Such a fix is not necessarily specific to qwik-auth, rather, it would likely need to be applied somehow with Qwik's Cloudflare adapter. (@builder.io/qwik-city/adapters/cloudflare-pages/vite I believe)

Currently it seems Qwik just wants you to npm run build then wrangler pages dev ./dist, but this is not ideal for rapid development.

As an aside, it may be possible to run wrangler pages dev and vite dev at the same time and get access to platform, but I haven't really looked to much into this. Integrating miniflare to the Cloudflare Pages adapter seems like the most elegant solution.

[serbyxyz]

yes how @austin-meadows explained it. it has nothing to do with qwik-auth just realized it when I had to modify qwik-auth to change the drizzle adapter from drizzle D1 to cloudflare D1. Which would work by just adding {env, platform} to the plugin. Instead of using the getDB() inside of the plugin

[serbyxyz]

okay so if you add this to package.json or just run it in terminal : @austin-meadows npx wrangler pages dev --proxy 5173 --compatibility-date=2023-10-06 -- npm run dev ( if you don't add the --proxy it runs but the returned page is just blank..)

also just running the npm run build and then the npm run serve would work but you need to add the wrangler.toml, it doesn't "dev mode" --reload though

assuming you have wrangler v3+ it will do all the miniflare "V3" / workerd ... stuff for you and allows you to do the platform.env anywhere. just need to add a wrangler.toml with your bindings etc.

The docs are very mixed between versions like the run flags --local etc but there is a list of deprecated ones https://developers.cloudflare.com/workers/wrangler/deprecations/

... now to figure out what all this proxy errors are with it (the authjs adapter d1-adapter not drizzle) also complains about some file that isnt getting bundled in the ... -- npm run dev.

I tried in the plugin@auth.ts doing this instead of using the getDb() in there since its much less modifications

serverAuth$(({ env, platform }) => ({
    secret: env.get("PRIVATE_AUTH_SECRET"),
    adapter: DrizzleAdapter(drizzle(platform.env?.DB, { schema: sqliteDB })),

it doesn't give the bundle error just the proxy error which it seems like might be something with the provider callback ( which I've tried all the ip addresses 127.0.0.1:8788 :5173... localhost: ...)

but that's as far as I got so far

[serbyxyz]

I actually found something interesting when messing about with Hono and Cloudflare workers ( very similar to FastAPI which I know well) … but hono does something that is typescript specific with global variables. That maybe could be added to the cloudflare or any other adapter. The platform works with the wrangler v3 + with the adapter. @mhevery , But I’m not sure if the adapter is looking for wrangler.toml file , in the case of cloudflare , not sure what vercel or others have . But if the adapter reads the wrangler.toml file and in some way “loads” or caches the bindings into the env, ( similar to how in tsconfig you can add the types to cloudflare and it’s “globally” available). If the adapter reads the wrangler.toml and makes the bindings available in that “global” way then I think it might solve a lot of the “type”ing problems.

for example it seems it’s pretty standard to make a src/db directory similar to the qwik insights (using libSQL/Turso based on env vars). I noticed it has this getDb() function/ method …

So with most using drizzle for example purpose , You can do the same IE

Export const db=drizzle(client,{schema:myschema})

But with cloudflare D1 client = platform.env.MyBinding

but platform can’t be used or “type”d in a seperate directory it always needs to be called inside an event / fetch handler style function by using the platform that’s provided in the method / function.

So in every routeLoader or routAction the db needs to be initiated for d1 , vs importing db from src/db …

I could totally be missing something here that might be common knowledge that I’m missing out on, that lead me down this path.( I have no problems with any other way of doing this except on cloudflare d1 … yes it’s beta so there is not much docs examples to go off).

I did a test run of installing qwik through the cloudflare C3 method and it does add extra functions to the package.json but I think the main thing that’s missing is the loading or parsing of the wrangler.toml file which could add the bindings as global vars or something like name space? In typescript ? I’m not very good at typescript so I can be off on that.

[mhevery]

A good investigation, but I have to be honest with you it's ower my head. Any chance someone could propose a PR?

[serbyxyz]

I think if your going to investigate , from what I have found out, looking at the qwik insights repo, make a branch off that. Last time I looked at it you have the getDB for turso / libSQL already in place and you already have the Qwik-auth plug-in installed. But I’m not sure if I’m production mode you have an adapter connected to it or not?

Here is some findings I tried and maybe you can reproduce results or get better “insight” into the subject ( for anyone that is stuck on this ).

Before even starting install qwik cloudflare plug-in via the builder.io - qwik adapters method and then install it via the cloudflare -C3 method , then compare the difference between the package.json “scripts”, first you will notes a “pages:dev” that is similar to the qwik - cloudflare adapters run dev that points to the .entry files. Second you will notice that the C3 create method adds a wrangler.toml file which is needed for the newer wrangler v3 which automatically has mini flare when in NODE_ENV= (!’not’ “production”) So you can do .local or .dev whatever with your env files ( it gets confusing in the wrangler.toml) when you set( see below ) [env.production] [env.dev] … or [vars] [vars.production] still confused about dev.vars file instead of .env file I think it’s the same or Vite will load the env for you but not the dev.vars . But one thing you will find in cloudflare docs is that your [d1database.bindings] needs a preview_id=“can be same as database_id”

[env.production] NODE_ENV= “production” [env.dev] NODE_ENV= “development”

^^^^ But for example Vite has automatic like search or parse for .env .env.local so I tried add [env.local] NODE_ENV= “development” [env.local.d1database] database_name= “database” database_id= “your db id” database_binding = “ your DB binding preview_id= “ I think you can make it whatever you want but I copied the database_id here”

but in env.local file I also added NODE_ENV=development

According to the wrangler v3 docs you don’t need to use —local anymore when running pages:dev , I had issue with qwik page loading using both install methods without —proxy “(Vite port)5173” then it would load the page And at the end of the pages:dev command adding — npm run dev ( I think I posted link / command in previous post)

but with the wrangler.toml file and running npm run pages:dev you will notice a new .sqlite file under the .wrangler/cache( or something)/{the preview_id you gave it} in the root of your project folder

So I think the main thing is the adapter install or add command just needs to be modified by adding / creating a wrangler.toml file to it similar to how C3 does it (that’s why I mention installing both for investigation) to compare what’s different ( I think it’s just what’s different in the updated wrangler v3)

But for me the database did not get migrated even by running the migrate commands for d1 with the —local flag in wrangler d1 migrate —local ( I think it’s like that) it says it migrates but in sqlite viewer extension still says 0 tables but if you run migrate again it says nothing to migrate …

so what I ended up doing was just using drizzle to push:sqlite -> that .wrangler/cache… .sqlite file using better-sqlite adapter and ended up just making the getDB() function when NODE_ENV ===“development” use drizzle( better-sqlite3) and when production use drizzle( d1 )

but you can’t just add better-sqlite3 with top-level import because when you push to GitHub ( and cloudflare runs build) it will complain about better-sqlite ( even if you have node_compat= true in cloudflare settings )… the drizzle adapter is fine it’s just the import of Database from /better-sqlite

so a work around was to make second function that you do like const Database = await import (“better-sqlite”).default but since Database is a class and it expects it to be async it needs to be awaited that’s why it needs to be in its own function and then called from inside of getDB ( when I get to pc I will paste example) I’m not very good with typescript so maybe it can be refined , it’s mainly that it will complain inside of the auth-js adapter that type of Database is not compatable but by sticking it in its own function then calling it in getDB it goes away ( this still doesn’t solve the fact that d1 is not running local on mini flare mainly due to the fact the local db doesn’t want to migrate the tables, so it does work without better-sqlite3) but the tables are not there so it throws error that table can not be found … so only way I found was using better-sqlite, ( which is similar to how qwik-insights uses libSQL for isLocal and turso for production)

** another thing to note is that the drizzle d1 adapter already has mini flare added to it …

[serbyxyz]

Hmm I’m not the best at this pr stuff, I don’t know where to begin… I know there is some sort of format that needs to be done 🤷‍♂️Sorry kind of new to all this “working in teams / GitHub” stuff :/ ( point me in the right direction and I might figure it out but will probably be scrutinized the first couple times)

** I don’t want my lack of experience on PR overshadow the actual issue so if I figure it out bear with me, I also need to sit down and actually think this out because I kind of just write things as they come into my head, but I think their is enough of that here that I can compile something more concise

On another note, based off the qwik-insights I think I tried connecting an adapter with libSQL - turso to the plug-in auth and the drizzle adapter didn’t play nice ( I took the fact it was netlify into consideration, and this is not exactly part of cloudflare topic / issue but does highlight some sort of “bug” in the db adapter in auth js and drizzle adapter)

[serbyxyz]

@mhevery you wrote angular you wrote qwik , trust me we just not on the same page, if I figure out the proper way to show you the issue ( I’m not at your level by any means ) , but I know that you can fix this in 15 minutes or less, it some trivial thing that I can’t explain properly without showing you where I hit walls, and those walls are typescript , because I look at hono and other repos and it’s something trivial like having a bindings.d.ts file that is somewhat equivalent to the qwik city PLATFORM = … but unless I figure out a way to show you all my brute force attempts and where they fail , it’s always going to seem over your head the communication on my end is poor because im no where near your level of understanding the Framework you made… so I can sit here and try in every way I know how to explain it but we never gonna get anywhere… because my typescript is mediocre at best.

But there are about 3 different issues that I have found that play into each other

Like hono and even itty-router anything that uses cloudflare does something ( similar to PLATFORM) using a .d.ts file called bindings or environment or Env, that in some shape or form needs to be adopted into the cloudflare pages plug-in ( if it’s something You want working properly, with all your other “integrations”) But maybe it already works that way just something I’m missing or once you see what I’m talking about your going to be like oh , that just needs to be like this, but I mean only the person that wrote the thing( or has the level of understanding ) is going to find the “oh” moment heh… and it’s some thing so simple I think but it’s a huge hurdle , let me figure out a way to make a good example repo on GitHub where I can try to highlight ( or have a compare code section) that might help, because the project in working on I can’t just show ( NDA) so I’ll try to just make a blank project and use one of. My personal cloudflare accounts to get a working mock up, you’ll see it’s some trivial thing

** remember I have this working , and using builder.io as cms and using figma -> builder plug-in ( a lot on my plate ATM) but the issue comes when I need to come out of production mode and into dev mode to try and build the Structured data plug-in for builder.io

** the structured data plug-in I got to be able to use dev mode to with drizzle ( or any adapter, drizzle because i can just change some env vars to make it reusable )

but the auth-js unless builder.io is going to integrate a sort of plug-in ( yes I can use supabase and it’s auth system off CF workers ) I can maybe use turso ( even though the auth-js plug-in complains about types with drizzle adapter as an adapter for auth js) I got it working , but I’m almost at the point of monkey patching it with python ( which I’m good at) but that breaks the whole work flow …

[serbyxyz]

Take this for example https://github.com/OultimoCoder/cloudflare-planetscale-hono-boilerplate/blob/main/bindings.d.ts

But look at the over all structure of the project ( you don’t need to look beyond the root directory)

but hono uses “c” as context which is equivalent to “platform” in qwik city … but that file right there I think is what where this issue stems from , like where do we implment something like this in qwik?

** like it seems like this would need to be tied into the entry file for pages , or imported somehow I could be way of basis on that though

[adamsoderstrom]

Thanks for the workaround, @sujith97! I tried it out and i found an alternative way to work with a local database (note: i've only tested this with Cloudflare D1):

1. The qwikCity plugin supports a platform property.
2. You could skip better-sqlite if you're already using the wrangler v3 CLI (in conjunction with the miniflare API).

---

Workaround

Note: This assumes that you've populated a local version of the database via the wrangler CLI. Though, It seems that wrangler d1 migrations create isn't useful for the moment , since it doesn't do anything(?). Instead, you could use wrangler d1 execute *database_name* --local --file path/to/file.sql.

Thanks to this, we can add the database connection once in vite.config.ts:

import { defineConfig, loadEnv } from "vite";
import { qwikVite } from "@builder.io/qwik/optimizer";
import { qwikCity } from "@builder.io/qwik-city/vite";
import tsconfigPaths from "vite-tsconfig-paths";
import { Miniflare, MiniflareOptions } from 'miniflare'

export default defineConfig(async ({ mode }) => {
  let platform = {}
  if (mode === 'development') {// or mode: 'ssr'
     // The following miniflare statements are heavily inspired by the source of the `wrangler execute` CLI command.
     // @see https://github.com/cloudflare/workers-sdk/blob/24d1c5cf3b810e780df865a0f76f1c3ae8ed5fbe/packages/wrangler/src/d1/execute.tsx#L236-L251
    /**
     * The `d1Persist` directory is created after you've executed `wrangler execute *database_name* --local`. 
     */
    const d1Persist = '.wrangler/state/v3/d1'
    const mf = new Miniflare({
      modules: true,
      script: "",
      d1Persist,
      d1Databases: { DB: 'database-id' }, // the value of `database_id` or `preview_database_id` in your `wrangler.toml`. Depends on what values that are / were present when you called `wrangler execute` mentioned earlier.
    });

    const db = await mf.getD1Database('DB') // The key of the `d1Databases` object prop.

    platform = {
      env: { DB: db }
    }
  }

  return {
    build: {
      rollupOptions: {
        external: new RegExp('stories.tsx')
      },
    },
    devTools: {
      clickToSource: false,
    },
    plugins: [qwikCity({
      platform // Here we pass the mocked `platform` property.
    }), qwikVite(), tsconfigPaths()],
    preview: {
      headers: {
        "Cache-Control": "public, max-age=600",
      },
    },
  }
});

Then, you should be able to use the database as follows:

export const useListLoader = routeLoader$(async (context: any) => {
  const db = context.platform.env.DB

  if (db) {
    const ps = db.prepare("select * from users");
    const firstUser = (await ps.all())?.results?.[0];
    return {
      list,
      user: { // DB objects have some issue if we directly send them so create a new object
        ...firstUser,
      },
    };
  }
});

[emmanuelchucks]

I wonder if Hono's new vite dev server package can provide any guidance here: https://github.com/honojs/vite-plugins/blob/main/packages/dev-server

[babie]

FYI, These could be used to address this issue:

• Wrangler's getPlatformProxy(): https://developers.cloudflare.com/workers/wrangler/api/#getplatformproxy
• Vite's Runtime API: https://vitejs.dev/blog/announcing-vite5-1.html#vite-runtime-api