bug: ssrServer ENOENT firestore.proto

Bandit commented 2 years ago

Describe the Bug

Have previously released/deployed without any problems. Tried to release tonight and when loading a page that relies on a load function with a Firebase call, it fails. Other pages seem to work.

This is the load function:

import { fetchProducts } from "$lib/stripeutils";
export async function load({ page, fetch, session, stuff }) {
    const products = await fetchProducts();
    return {
        props: {
            products,
        },
    };
}

This is the fetchProducts function

import { firestore, functions } from "$lib/firebase/client";
import { getDocs, query, where, orderBy, doc, addDoc, collection, onSnapshot } from 'firebase/firestore';
[...]

export async function fetchProducts() {
    let products = {
        [...]
    };

    const docSnap = await getDocs(query(
        collection(firestore(), "products"),
        where("active", "==", true),
        orderBy("metadata.order")
    ));

    await Promise.all(docSnap.docs.map(async (doc) => {
        let product = {
            [...]
        };

        const priceSnap = await getDocs(query(
            collection(doc.ref, "prices"),
            where("active", "==", true)
        ));

        priceSnap.docs.forEach((pdoc) => {
            const price = {
                [...]
            };

            product.prices = [price, ...product.prices];
        });

        const type = product.metadata.type || "addon";
        products[type].items.push(product);
    }));

    return products;
}

The firestore function

import { initializeApp, getApps } from "firebase/app";
import { getFirestore } from 'firebase/firestore';
[...]

const config = {
    [...]
};

function firebase() {
    return getApps().length === 0 ? initializeApp(config) : getApps()[0];
}

function firestore() {
    return getFirestore(firebase());
}

The errors:

ssrServer

[2021-10-28T07:36:58.392Z]  @firebase/firestore: Firestore (9.0.0): INTERNAL UNHANDLED ERROR:  Error: ENOENT: no such file or directory, open '/workspace/ssrServer/src/protos/google/firestore/v1/firestore.proto' 

Error: ENOENT: no such file or directory, open '/workspace/ssrServer/src/protos/google/firestore/v1/firestore.proto'
    at Object.openSync (fs.js:498:3)
    at Object.readFileSync (fs.js:394:35)
    at fetch2 (/workspace/ssrServer/index.js:11082:30)
    at Root2.load2 [as load] (/workspace/ssrServer/index.js:11111:11)
    at Root2.loadSync (/workspace/ssrServer/index.js:11121:19)
    at Object.loadProtosWithOptionsSync (/workspace/ssrServer/index.js:14289:31)
    at Object.loadSync (/workspace/ssrServer/index.js:14440:33)
    at loadProtos (/workspace/ssrServer/index.js:106181:43)
    at newConnection (/workspace/ssrServer/index.js:106185:20)
    at OnlineComponentProvider2.createDatastore (/workspace/ssrServer/index.js:109513:26)

Steps to Reproduce

Have previously deployed without issue, however this is my first release where I've used a Firestore call inside a page's load function. Mainly looking for a place to start my debugging for something like this.

Expected Behaviour

Code should work fine?

svelte-adapter-firebase version

0.9.2

sveltejs/kit version

1.0.0.next.160

Bandit commented 2 years ago

I assume I don't need to manually manage dependencies in functions/package.json. Firestore isn't in there (but also neither is anything else I use in my site)

{
  "name": "functions",
  "description": "Cloud Functions for Firebase",
  "scripts": {
    "lint": "eslint .",
    "serve": "firebase emulators:start --only functions",
    "shell": "firebase functions:shell",
    "start": "npm run shell",
    "deploy": "firebase deploy --only functions",
    "logs": "firebase functions:log"
  },
  "engines": {
    "node": "14"
  },
  "main": "index.js",
  "dependencies": {
    "cors": "^2.8.5",
    "express": "^4.17.1",
    "firebase-admin": "^9.8.0",
    "firebase-functions": "^3.15.5",
    "stripe": "^8.164.0"
  },
  "devDependencies": {
    "eslint": "^7.6.0",
    "eslint-config-google": "^0.14.0",
    "firebase-functions-test": "^0.2.0"
  },
  "private": true
}

Bandit commented 2 years ago

This page is not working either, with the same ENOENT error

import { firestore } from "$lib/firebase/client";
import { collection, query, where, getDocs } from 'firebase/firestore';

export async function load({ page, fetch, session, stuff }) {
    const profilesRef = collection(firestore(), "profiles");
    const profileQuery = query(profilesRef, where("slug", "==", page.params.slug));

    const querySnap = await getDocs(profileQuery);

    if (querySnap.empty) return {
        status: 404,
        error: new Error(`Profile not found`),
    };

    return {
        props: {
            profile: {
                uid: querySnap.docs[0].id,
                ...querySnap.docs[0].data(),
            },
        },
    };
}

But if I navigate to this page using the SvelteKit router (e.g. clicking a link in the client) it works, so definitely entirely related to a bug with the ssrServer function.

Also worth noting the SSR-aspect of all these pages work fine locally, so nothing to do (as far as I can tell) with including client-side only dependencies or something like that.

jthegedus commented 2 years ago

I would suggest starting with updates to both SvelteKit and the adapter. There have been quite a few versions released since those versions you list in the issue. Also, there will be some breaking changes from Kit for the adapter API relatively soon which will affect the compilation of the SSR code, so I expect behaviour to change again in another breaking change.

jthegedus commented 2 years ago

I assume I don't need to manually manage dependencies in functions/package.json

Correct, although the SSR code runs on the Cloud Functions, all dependencies for the SSR code are compiled in-line during build phase so you shouldn't need to touch your Cloud Function deps for the Kit APIs or SSR code.

jthegedus commented 2 years ago

first release where I've used a Firestore call inside a page's load function

Many people have had issues with including Firestore directly in load, it can usually be mitigated by performing the Firestore data load in a Kit endpoint and then fetching that kit endpoint like any other REST API

Bandit commented 2 years ago

I updated to v1.0.0-next.192 and 0.13.1 and now get this error on a page that worked prior to upgrading:

TypeError [ERR_INVALID_ARG_VALUE]: The argument 'filename' must be a file URL object, file URL string, or absolute path string. Received undefined
    at Function.createRequire (internal/modules/cjs/loader.js:1173:11)
    at Object.<anonymous> (/workspace/ssrServer/index.js:133837:38)
    at Module._compile (internal/modules/cjs/loader.js:1072:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1101:10)
    at Module.load (internal/modules/cjs/loader.js:937:32)
    at Function.Module._load (internal/modules/cjs/loader.js:778:12)
    at Module.require (internal/modules/cjs/loader.js:961:19)
    at require (internal/modules/cjs/helpers.js:92:18)
    at /workspace/index.js:273:31
    at cloudFunction (/workspace/node_modules/firebase-functions/lib/providers/https.js:50:16)

/EDIT I get this error on every page, and the client gets served with Error: could not handle the request

Ideas?

jthegedus commented 2 years ago

How are you running your app? Just npm run dev in your SvelteKit app directory?

Bandit commented 2 years ago

npm run dev --

Runs fine locally though, just get those errors on the server in the ssrServer function

jthegedus commented 2 years ago

can you share more of your actual code? Hard to debug without it.

Bandit commented 2 years ago

Which part of the code should I share? No page can be served by the ssrServer function, even a static page with no code at all on it.

E.g. a markdown file parsed with mdsvex into svelte that has literally no JS in it triggers this error:

 ssrServer

TypeError [ERR_INVALID_ARG_VALUE]: The argument 'filename' must be a file URL object, file URL string, or absolute path string. Received undefined
    at Function.createRequire (internal/modules/cjs/loader.js:1173:11)
    at Object.<anonymous> (/workspace/ssrServer/index.js:133837:38)
    at Module._compile (internal/modules/cjs/loader.js:1072:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1101:10)
    at Module.load (internal/modules/cjs/loader.js:937:32)
    at Function.Module._load (internal/modules/cjs/loader.js:778:12)
    at Module.require (internal/modules/cjs/loader.js:961:19)
    at require (internal/modules/cjs/helpers.js:92:18)
    at /workspace/index.js:273:31
    at cloudFunction (/workspace/node_modules/firebase-functions/lib/providers/https.js:50:16)

Bandit commented 2 years ago

firebase.json

{
  "functions": {
    "source": "functions"
  },
  "hosting": {
    "public": "public",
    "ignore": [
      "firebase.json",
      "**/.*",
      "**/node_modules/**"
    ],
    "rewrites": [
      {
        "source": "**",
        "function": "ssrServer"
      }
    ],
    "predeploy": ["npm run build"]
  }
}

svelte.config.js

import { mdsvex } from "mdsvex";
import mdsvexConfig from "./mdsvex.config.js";
import firebase from "svelte-adapter-firebase";
import preprocess from "svelte-preprocess";

/** @type {import('@sveltejs/kit').Config} */
const config = {
    "extensions": [".svelte", ...mdsvexConfig.extensions],

    kit: {
        adapter: firebase(),
        // hydrate the <div id="svelte"> element in src/app.html
        target: '#svelte',
        vite: {
            ssr: {
                external: ['firebase']
            }
    },

    preprocess: [
        mdsvex(mdsvexConfig),
        preprocess({
            "postcss": true
        })
    ],

    onwarn: (warning, handler) => {
        const { code, frame } = warning;
        if (code === "a11y-missing-content") return;

        console.log(code);
        handler(warning);
    }
};

export default config;
// Workaround until SvelteKit uses Vite 2.3.8 (and it's confirmed to fix the Tailwind JIT problem)
const mode = process.env.NODE_ENV;
const dev = mode === "development";
process.env.TAILWIND_MODE = dev ? "watch" : "build";

Top of package.json

"scripts": {
        "dev": "svelte-kit dev",
        "build": "npx rimraf public && svelte-kit build --verbose",
        "preview": "svelte-kit preview",
        "lint": "prettier --check --plugin-search-dir=. . && eslint --ignore-path .gitignore .",
        "format": "prettier --write --plugin-search-dir=. ."
    },

Firebase versions: "firebase": "^9.0.0", "firebase-admin": "^9.11.0"

I also get this in my console when running firebase deploy

12:53:27 PM [vite-plugin-svelte] The following packages did not export their `package.json` file so we could not check the "svelte" field. If you had difficulties importing svelte components from a package, then please contact the author and ask them to export the package.json file.
- @firebase/firestore

Bandit commented 2 years ago

This is line 133860 of ssrServer - looks like some Firebase code to me

var require2 = import_module.default.createRequire(import_meta.url);

This is further up in the file, and it looks like import_meta is never written to after initiation here because there's only 2 references of it according to my search (which explains the URL error because import_meta.url is undefined)

// node_modules/@firebase/firestore/dist/index.node.mjs
[...]
var import_meta = {};

/EDIT dug out the code for this part of the index.node.mjs file above (L14241), and it's preceded by these comments:

// This is a hack fix for Node ES modules to use `require`.
// @ts-ignore To avoid using `--module es2020` flag.
const require = module.createRequire(import.meta.url);

Could be something there? Some sort of incompatibility with the way ssrServer is executed or something?

jthegedus commented 2 years ago

12:53:27 PM [vite-plugin-svelte] The following packages did not export theirpackage.jsonfile so we could not check the "svelte" field. If you had difficulties importing svelte components from a package, then please contact the author and ask them to export the package.json file. -@firebase/firestore

You can ignore this.

I would try bumping your Cloud Function Node.js version from 14 to 16 and seeing what happens.

If that doesn't work I have this to say: using firestore in load can have issues, as I said before you can move it to a Kit endpoint and mitigate most of the issues.

If the error is coming from firebase/firestore package, my recommendation is to not use the firebase lib for firestore in code that can run on the server. load can run in SSR and on the Client. While the firebase@9.0.0 lib can be used in Node.js environments as well as web, it should only be used when on on the web. Firestore data fetch should be performed by using firebase-admin@10.0.0 on the server side.

So in your load function, you should first check if the execution environment is Server or Client and use firebase-admin to get firestore on the server & firebase to get firestore on the client (web).

This matters because the packages are different with firebase-admin specifically targeting Node.js envs. The authentication for each of the libs is also different:

server: firebase-admin > get firestore instance > execute query > bypasses security rules because it is run on server using the special admin lib > hits firestore database
client: firebase > get firestore instance > execute query > pass through security rules with user auth token > hits firestore database

Using the Client lib on the Server raises questions around whether the query bypasses the sec rules on the Firestore server or not, and also, if it does go through the security rules, how can the SSR environment know which user initiated the request.

Given these factors, I recommend NOT using the Client (web) firestore libs on the Server and switching the use of the firebase vs firebase-admin packages depending on the Kit $app/env browser value (I have found I need to dynamically import these libs and that code splitting cannot accurately remove each one from the other bundle, causing errors).

It is possible to have an authenticated user pass their auth token to the initial page call that executes the page SSR, so the web lib can be used in SSR envs and know which users initiated the request and perform security rules appropriately, however this wiring needs to be done manually within the Kit request lifecycle and requires handling Firebase auth tokens yourself, which defeats the point of the Firebase.

Server data fetch on SSR load should be of data that does not require an auth token using the firebase-admin lib.

Client data fetch on client load should use the firebase lib.

Hope this helps you debug.

Bandit commented 2 years ago

@jthegedus thanks for your advice.

What you say about Firestore makes a lot of sense, and keen to do that (though it will obviously take quite some time), but the error I'm now receiving didn't happen before upgrading to Sveltekit v1.0.0-next.192 and svelte-adapter-firebase 0.13.1 so doesn't that imply some sort of bug / issue there?

Especially given my ssrServer function literally can't run due to some sort of compiled code error as per the above (where import_meta.url is undefined)?

I tried bumping to Node v16 but the only thing that changes is now I don't get given a reason for the crash in the logs:

jthegedus commented 2 years ago

the error I'm now receiving didn't happen before upgrading to Sveltekit v1.0.0-next.192 and svelte-adapter-firebase 0.13.1 so doesn't that imply some sort of bug / issue there

Logically, sure, actually, maybe.

Updating Kit actually updates Vite, the Svelte-Vite Plugin, which all effect how code is bundled. The adapters currently then bundle the Vite SSR bundle again using esbuild. So there can be issues between those. You haven't shared your Kit package.json:dependencies which include your firebase imports, I imagine those are version ranges with ^ and not pinned to specific versions, so as you dev, reinstall deps, update deps your firebase version may have also changed where the bug could be introduced in that lib and how it is compiled by either Vite or esbuild.

I watch the Kit changelog & PRs for breakages to the adapter API and update accordingly. I currently only test to ensure the Kit todo template works as expected once parsed through the adapter. There is a specific compatibility table of Kit & adapter versions in the readme, so sticking to those might help.

I do not currently test how firebase packages & their usage in Kit work in the tests here. I do consume Kit & this adapter for my own projects which do use Firebase.

I have been unable to get Firestore working in the way you describe, ever, in any version of Kit & this adapter. With the firebase and firebase-admin libs going through rewrites to better support ESM for usage in ESM-based tools like Vite, I decided not to delve into their specific issues and debug them. Kit, this adapter, Vite, Vite Svelte Plugin & both Client & Server Firebase libs are all being actively developed. Even though Firebase v9 lib is officially released with ESM support, ESM issues surface regularly and effect Vite projects, which in turn would effect Kit.

As I mentioned in my first response

there will be some breaking changes from Kit for the adapter API relatively soon which will affect the compilation of the SSR code, so I expect behaviour to change again in another breaking change.

So the aspect of Kit & this adapter that effect SSR code generation is going to change again, so we could spend all week debugging this to only have our efforts rendered useless in a weeks time.

Further suggestions:

remove this snippet from you svelte.config.js and seeing what happens:
```
          vite: {
              ssr: {
                  external: ['firebase']
              }
```
I've not see people do this before, it may have been a workaround during a beta release of the firebase@9 package
wrap the function call to the Kit server in your Cloud Function with a try/catch to log the error and see what it is in Node.js 16. It may be the same root cause, or something new. The adapter uses your Cloud Function Node.js runtime version as the compile target version so changing the Function runtime does effect the compilation of your SSR code.
reverting back to your older solution that worked and waiting for both Kit and the adapter to come out of Beta
pull your Firestore usage out of the component load and into the Kit endpoint (using firebase-admin in endpoint) as I know that works

Bandit commented 2 years ago

wrap the function call to the Kit server in your Cloud Function with a try/catch to log the error and see what it is in Node.js 16. It may be the same root cause, or something new. The adapter uses your Cloud Function Node.js runtime version as the compile target version so changing the Function runtime does effect the compilation of your SSR code.

How to do this? Manually edit functions/ssrServer/index.js and tell firebase to deploy it without running a sveltekit build?

Bandit commented 2 years ago

Removing that external: ['firebase'] snippet causes this locally in my Svelte terminal:

11:36:28 AM [vite] Error when evaluating SSR module /node_modules/@firebase/functions/dist/index.esm2017.js?v=82ef9ed5: ReferenceError: self is not defined at eval (/node_modules/@firebase/functions/dist/index.esm2017.js?v=82ef9ed5:674:30) at async instantiateModule (/node_modules/vite/dist/node/chunks/dep-85dbaaa7.js:66541:9)

Looks like this issue: https://github.com/firebase/firebase-js-sdk/issues/4846 so I used the workaround and dynamically load the functions lib client-side only.

jthegedus commented 2 years ago

wrap the function call to the Kit server in your Cloud Function with a try/catch to log the error and see what it is in Node.js 16. It may be the same root cause, or something new. The adapter uses your Cloud Function Node.js runtime version as the compile target version so changing the Function runtime does effect the compilation of your SSR code.

How to do this? Manually edit functions/ssrServer/index.js and tell firebase to deploy it without running a sveltekit build?

svelte-adapter-firebase outputs code to the terminal on first run which you need to add to your Cloud Functions index.js file, wrap the code there. Looking at the e2e test as an example:

https://github.com/jthegedus/svelte-adapter-firebase/blob/main/tests/end-to-end/scaffold/functions/index.js

const functions = require('firebase-functions');

let sveltekitServer;
exports.sveltekit = functions.https.onRequest(async (request, response) => {
    if (!sveltekitServer) {
        functions.logger.info('Initialising SvelteKit SSR Handler');
        sveltekitServer = require('./sveltekit/index').default;
        functions.logger.info('SvelteKit SSR Handler initialised!');
    }

    functions.logger.info('Requested resource: ' + request.originalUrl);
-   return sveltekitServer(request, response);
+   let result;
+   try {
+       result = await sveltekitserverServer(request, response);
+   } catch(err) {
+       functions.logger.error(err)
+   }
+   return result;

This way you can see the errors when the call the the Kit Server fails.

jthegedus commented 2 years ago

Removing that external: ['firebase'] snippet causes this locally in my Svelte terminal:

11:36:28 AM [vite] Error when evaluating SSR module /node_modules/@firebase/functions/dist/index.esm2017.js?v=82ef9ed5: ReferenceError: self is not defined at eval (/node_modules/@firebase/functions/dist/index.esm2017.js?v=82ef9ed5:674:30) at async instantiateModule (/node_modules/vite/dist/node/chunks/dep-85dbaaa7.js:66541:9)

Looks like this issue: firebase/firebase-js-sdk#4846 so I used the workaround and dynamically load the functions lib client-side only.

This is exactly what I mean when I say Firebase still has many bugs when working with Vite/Kit. I have not been pursuing resolving those bugs here for the firebase SDKs because they're not yet worked out upstream.

The dynamic import of the lib client-side is what I do for all firebase libs, with load data coming from Kit endpoints that use firebase-admin. It is the only way I have been able to get this all to work as of ~3 months ago and I haven't revisited since.

Bandit commented 2 years ago

@jthegedus off topic in a way, but do you have a code example of importing firebase-admin in a server route?

With this import import { initializeApp, applicationDefault } from 'firebase-admin/app';

I get this error:

[vite] Error when evaluating SSR module products.json.js
Error: Cannot find module 'firebase-admin/app'`

firebase-admin is a devDependency so I don't understand it...

/EDIT looks like this issue? https://github.com/firebase/firebase-admin-node/issues/1488

Bandit commented 2 years ago

Just checking in to say that moving ALL firebase calls to dynamic imports (behind a browser check) as you suggested has fixed the problem. Working with v1.0.0-next.193 and 0.13.1 with no silly crap in svelte.config.js 👍

It's been a journey. Thanks for the help once again @jthegedus

jthegedus commented 2 years ago

@jthegedus off topic in a way, but do you have a code example of importing firebase-admin in a server route?

This is what I was testing with:

// src/lib/firebase/admin.js
import admin from "firebase-admin";

const app = admin.apps.length === 0 ? admin.initializeApp() : admin.apps[0];
const firestore = admin.firestore(app);

export { firestore };

// src/routes/blog/[slug].json.js
import { firestore } from '$lib/firebase/admin';

export async function get(request) {
    const slug = request.params.slug;
    console.log(`SvelteKit Endpoint: ${request.path}`);
    const postPageSnapshot = await firestore.doc(`posts/${slug}`).get();

Importantly I was using firebase-admin@9.12.0 at the time and when v10 released did not upgrade because it had issues. v9 didn't require anything special. While I generally do recommend updating to the latest packages, server-side code is less critical and firebase-admin@9 is very stable

jthegedus commented 2 years ago

It's been a journey. Thanks for the help once again @jthegedus

Glad you got it working.

jthegedus / svelte-adapter-firebase