tursodatabase / libsql

libSQL is a fork of SQLite that is both Open Source, and Open Contributions.
https://turso.tech/libsql
MIT License
12k stars 304 forks source link

LibsqlError: URL_INVALID when deploying to Render #1121

Open rakis10 opened 9 months ago

rakis10 commented 9 months ago

Localy works, but when deploying on Render.

db.js:

let dotenv = require('dotenv').config()
const { createClient } = require('@libsql/client')
const { drizzle } = require('drizzle-orm/libsql')
const client = createClient({
  url: dotenv.parsed.TURSO_URL,
  authToken: dotenv.parsed.TURSO_AUTH_TOKEN,
})
module.exports = drizzle(client)

"dependencies": { "@libsql/client": "^0.5.2", "cors": "^2.8.5", "dotenv": "^16.4.5", "drizzle-orm": "^0.29.4", "ejs": "^3.1.9", "events": "^3.3.0", "express": "^4.18.2", "express-session": "^1.17.3", "fs": "^0.0.1-security", "nodemon": "^3.0.2", "passport": "^0.7.0", "passport-local": "^1.0.0" },

/opt/render/project/src/node_modules/@libsql/core/lib-cjs/uri.js:12 throw new api_js_1.LibsqlError("The URL is not in a valid format", "URL_INVALID"); ^ LibsqlError: URL_INVALID: The URL is not in a valid format at parseUri (/opt/render/project/src/node_modules/@libsql/core/lib-cjs/uri.js:12:15) at expandConfig (/opt/render/project/src/node_modules/@libsql/core/lib-cjs/config.js:35:39) at createClient (/opt/render/project/src/node_modules/@libsql/client/lib-cjs/node.js:28:52) at Object. (/opt/render/project/src/db.js:4:16) at Module._compile (node:internal/modules/cjs/loader:1376:14) at Module._extensions..js (node:internal/modules/cjs/loader:1435:10) at Module.load (node:internal/modules/cjs/loader:1207:32) at Module._load (node:internal/modules/cjs/loader:1023:12) at Module.require (node:internal/modules/cjs/loader:1235:19) at require (node:internal/modules/helpers:176:18) { code: 'URL_INVALID', rawCode: undefined,

} Node.js v20.11.1

notskamr commented 8 months ago

Facing the same issue with Cloudflare workers

niklauslee commented 8 months ago

Same issue with Cloudflare pages (works well in local)

Error: Failed to publish your Function. Got error: Uncaught Error: LibsqlError: URL_INVALID: The URL is not in a valid format

Using Astro, Drizzle and Turso:

// db.ts
import { drizzle } from "drizzle-orm/libsql";
import { createClient } from "@libsql/client/web";

export const client = createClient({
  url: import.meta.env.TURSO_URL,
  authToken: import.meta.env.TURSO_AUTH_TOKEN,
});

export const db = drizzle(client);

Environment variables in Cloudflare:

TURSO_URL=libsql://xxxxxxxxxxxx.turso.io
TURSO_AUTH_TOKEN=...
niklauslee commented 8 months ago

When I use url string directly in the client (instead of using import.meta.env.), it works. Maybe this is a cloudflare's environment variable issue.

The related issue: https://github.com/withastro/astro/issues/6130

RafaelRamosR commented 8 months ago

It worked for me.

import { createClient } from '@libsql/client/web';

const dbClient = () => createClient({
  url: import.meta.env.TURSO_URL,
  authToken: import.meta.env.TURSO_AUTH_TOKEN,
});
AirBorne04 commented 8 months ago

Hi @niklauslee the issue here is that the environment vars are not replaced at compile time. They are instead proxied to the request env vars at runtime (in cloudflare env), therefore you need to access the vars after a request has been made to your worker. This means that the suggested change from @RafaelRamosR works when called from inside an api request or inside an astro file, or middleware.

ArshErgon commented 7 months ago

@RafaelRamosR @AirBorne04 I am getting this error: Error [LibsqlError]: URL_INVALID: The URL is not in a valid format on vercel build failed, local development is working fine but deployment is failing


import { createClient } from '@libsql/client';
import { drizzle } from 'drizzle-orm/libsql';
import * as schema from './schema';

export const client = createClient({
    url: import.meta.env.TURSO_URL!,
    authToken: import.meta.env.TURSO_AUTH_TOKEN
});

export const db = drizzle(client, { schema });```
AirBorne04 commented 7 months ago

@ArshErgon you need to use a factory function createClient = () => because the env vars can only be read when a request is ongoing not statically. Just as the example from @RafaelRamosR above.

ArshErgon commented 7 months ago

Hey @AirBorne04 thanks for the reply but error is still the same db.ts

import { createClient } from '@libsql/client';
import { drizzle } from 'drizzle-orm/libsql';
import * as schema from './schema';

export const client = () => createClient({
    url: import.meta.env.TURSO_URL!,
    authToken: import.meta.env.TURSO_AUTH_TOKEN
});

export const db = drizzle(client(), { schema }); 

or can this file cause error? drizzle.config.ts

import { Config } from 'drizzle-kit';
import 'dotenv/config';

export default {
    schema: './src/lib/db/schema.ts',
    out: './migrations',
    driver: "turso",
    dbCredentials: {
        url: process.env.TURSO_URL!,
        authToken: process.env.TURSO_AUTH_TOKEN
    }
} satisfies Config;

edit: I finally found out what I was doing wrong: https://stackoverflow.com/questions/78328688/sveltekit-deployment-on-vercel-failed-because-of-error-libsqlerror-url-invali/78329787#78329787

AirBorne04 commented 7 months ago

@ArshErgon yeah ok, good you got it working. That is correct in a standard VITE project you need to prefix the vars. I thought you would also be in the AstroJS / Cloudflare environment (as @niklauslee ) where the case is different. So i think also the initial report is rather related to wrong env reading than turso? @rakis10 ?

Zaprovic commented 6 months ago

Hello! I am facing this issue but in development. I am using Bun, Drizzle ORM and Turso and when I run the very first time the local environment the page renders without problem, but when I refresh the website it shows me in the client the same error and points it to my src/db/main.ts file

import { createClient } from "@libsql/client";
import { drizzle } from "drizzle-orm/libsql";

const client = createClient({
  url: process.env.DATABASE_URL!,
  authToken: process.env.DATABASE_AUTH_TOKEN!,
});

export const db = drizzle(client);

Right now I have no idea what would the problem be :( . However it seems to be a problem in the client because the logs in the terminal do not show any issues at all. The error only appears in the browser

image

If I hardcode the variables in both drizzle.config.ts and src/db/main.ts it works without any issue. But I would like it to work with the enviroment variables

ArshErgon commented 6 months ago

@Zaprovic where are you hosting it on vercel? are you sure you have declared the environment variable correctly on your hosts? Do re-check. if it still failed, try to console log the process.env.DATABASE_URL its mostly happening because the URL is not found.

Zaprovic commented 6 months ago

@ArshErgon I had just launch to production on Vercel, so yes, there's where I am hosting my app. However in production the error does not happen. I copied and pasted the .env file in the Vercel environment variables and the deployment was successful. I still get the same issue in development though.

It is very weird, because the error points to src/db/main.ts but in the drizzle.config.ts I defined the environment variables the exact same way and it doesnt show any errors

import { defineConfig } from "drizzle-kit";

export default defineConfig({
  schema: "./src/db/schema.ts",
  out: "./drizzle/migrations",
  dialect: "sqlite",
  driver: "turso",
  dbCredentials: {
    url: process.env.DATABASE_URL!,
    authToken: process.env.DATABASE_AUTH_TOKEN!,
  },
});
import { createClient } from "@libsql/client";
import { drizzle } from "drizzle-orm/libsql";

// todo: fix .env issue
const client = createClient({
  url: process.env.DATABASE_URL!,
  authToken: process.env.DATABASE_AUTH_TOKEN!,
});

export const db = drizzle(client);

Here's how the error appears in my vscode as well

image

If I console log the variables as you say, I first get undefined, but then I get the correct value, maybe that's where the problem is

image

I am guessing the environment variables should be loaded ahead, before adding them to the libsqlClient configuration variable, but I don't know how to do that

ArshErgon commented 6 months ago

@Zaprovic I was once in your shoes: (https://stackoverflow.com/questions/78328688/sveltekit-deployment-on-vercel-failed-because-of-error-libsqlerror-url-invali/78329787#78329787) So I know hard all this is going for you, I will suggest to build the project locally and running the preview of it.

  1. Make sure you print that your createClient is getting your DATABASE_URL before the client variable try to print the DATABASE_URL (in both locally and production, in production you can see better error messages in vercel terminal as its only happenening because of DATABASE_URL is empty and so does the DATABASE_AUTH_TOKEN)
  2. Build the project locally npm run build && npm run preview

For me it work just adding VITE as a prefix maybe its the same for you: https://bun.sh/guides/runtime/set-env

Zaprovic commented 6 months ago

Here's the thing

image

This happens when I execute once, notice that the variable gets logged with no problem. But after like 1 second the log changes and I get this

image

I get 4 logs, the first three of them are undefined, and the last one is the url. Btw, I am using Nextjs, not Sveltekit

ArshErgon commented 6 months ago

I get 4 logs, the first three of them are undefined, and the last one is the url. Btw, I am using Nextjs, not Sveltekit

Yeah, I did read your very first comment which told me you are using Nextjs not Svelte, but the problem you are stuck is same as I was stuck in.

For me it work just adding VITE as a prefix maybe its the same for you: https://bun.sh/guides/runtime/set-env

Can you try to add this: Bun.env.API_TOKEN; (https://bun.sh/guides/runtime/set-env) and again uplaod the .env file to vercel not paste just upload the full file, and keep that console.log for your production build it will help you debug if you again hit with error

Zaprovic commented 6 months ago

@ArshErgon I thought of using Bun but it also does not work either. I got around the error by using the NEXT_PUBLIC_ prefix before DATABASE_URL. However I got another problem after that hahaha.

This is my .env.local

NEXT_PUBLIC_DATABASE_URL=libsql://******************
DATABASE_AUTH_TOKEN=ey**********************
/* src/db/main.ts */
import { createClient } from "@libsql/client";
import { drizzle } from "drizzle-orm/libsql";

// fix: added NEXT_PUBLIC_ to environment variables

const client = createClient({
  url: process.env.NEXT_PUBLIC_DATABASE_URL!,
  authToken: process.env.DATABASE_AUTH_TOKEN!,
});

export const db = drizzle(client);
/* drizzle.config.ts */
import * as dotenv from "dotenv";
import { defineConfig } from "drizzle-kit";
dotenv.config({
  path: ".env.local",
});

export default defineConfig({
  schema: "./src/db/schema.ts",
  out: "./drizzle/migrations",
  dialect: "sqlite",
  driver: "turso",
  dbCredentials: {
    url: process.env.NEXT_PUBLIC_DATABASE_URL!,
    authToken: process.env.DATABASE_AUTH_TOKEN!,
  },
});

I tried to get around the environment variables so I was able to perform migrations and this is what I ended up with. However, now I am not able to make a fetch call in a client component. For example this component throws the error in the browser

"use client";

import { Button } from "@/components/ui/button";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Checkbox } from "@/components/ui/checkbox";
import {
  Form,
  FormControl,
  FormDescription,
  FormField,
  FormItem,
  FormLabel,
  FormMessage,
} from "@/components/ui/form";
import { db } from "@/db/main";
import { CategoryTable } from "@/db/schema";
import { zodResolver } from "@hookform/resolvers/zod";
import { useEffect, useState } from "react";
import { useForm } from "react-hook-form";
import { z } from "zod";

// todo: data fetch cannot be performed in the client, only works on the server

const CategoryFormSchema = z.object({
  categories: z
    .array(z.string())
    .refine((value) => value.some((item) => item), {
      message: "Debes seleccionar al menos una categoria",
    }),
});

const CategoryOptions = () => {
  const [categories, setCategories] = useState<{ id: string; name: string }[]>(
    [],
  );

  useEffect(() => {
    const fetchCategories = async () => {
      try {
        const allCategories = (await db.select().from(CategoryTable).all()).map(
          ({ id, name }) => ({
            id: name.toLowerCase(),
            name,
          }),
        );

        setCategories(allCategories);
      } catch (error) {
        if (error instanceof Error) {
          throw new Error(error.message);
        }
      }
    };

    fetchCategories();
  }, []);

  const form = useForm<z.infer<typeof CategoryFormSchema>>({
    resolver: zodResolver(CategoryFormSchema),
    defaultValues: {
      categories: [],
    },
  });

  const onsubmit = async (data: z.infer<typeof CategoryFormSchema>) => {
    console.log(data);
  };

  return (
    <Card>
      <CardHeader>
        <CardTitle>Selecciona una categoria</CardTitle>
      </CardHeader>
      <CardContent>
        {categories.length > 0 ? (
          <Form {...form}>
            <form onSubmit={form.handleSubmit(onsubmit)} className="space-y-8">
              <FormField
                control={form.control}
                name="categories"
                render={() => (
                  <FormItem>
                    <div className="mb-4">
                      <FormLabel className="text-base">Categorias</FormLabel>
                      <FormDescription>
                        Selecciona como minimo una categoria
                      </FormDescription>
                    </div>
                    {categories.length > 0 &&
                      categories.map((item) => (
                        <FormField
                          key={item.id}
                          control={form.control}
                          name="categories"
                          render={({ field }) => {
                            return (
                              <FormItem
                                key={item.id}
                                className="flex flex-row items-start space-x-3 space-y-0"
                              >
                                <FormControl>
                                  <Checkbox
                                    checked={field.value?.includes(item.id)}
                                    onCheckedChange={(checked) => {
                                      return checked
                                        ? field.onChange([
                                            ...field.value,
                                            item.id,
                                          ])
                                        : field.onChange(
                                            field.value?.filter(
                                              (value) => value !== item.id,
                                            ),
                                          );
                                    }}
                                  />
                                </FormControl>
                                <FormLabel className="text-sm font-normal">
                                  {item.name}
                                </FormLabel>
                              </FormItem>
                            );
                          }}
                        />
                      ))}
                    <FormMessage />
                  </FormItem>
                )}
              />
              <Button type="submit">Submit</Button>
            </form>
          </Form>
        ) : (
          <span className="text-sm font-light">No categories to show</span>
        )}
      </CardContent>
    </Card>
  );
};

export default CategoryOptions;

image

And this one makes the fetch call with no issues

import { IconFilter } from "@tabler/icons-react";
import { db } from "../db/main";
import { CategoryTable } from "../db/schema";
import { Badge } from "./ui/badge";
import { Button } from "./ui/button";

const CategoriesNavbar = async () => {
  const categories = await db.select().from(CategoryTable).all();

  return (
    // todo: tweak properly the 'hidden' class if there are more categories
    <nav className="mb-5 flex w-full justify-end md:justify-center">
      <Button size={"icon"} variant={"secondary"} className="md:hidden">
        <IconFilter />
      </Button>
      <ul className="hidden gap-2 md:flex">
        {categories.map((category) => (
          <li key={category.id}>
            <Badge variant={"secondary"} className="py-1 hover:cursor-pointer">
              {category.name}
            </Badge>
          </li>
        ))}
      </ul>
    </nav>
  );
};

export default CategoriesNavbar;
ArshErgon commented 6 months ago

@Zaprovic the 401 error means its Unauthorized. means its now TOKEN. Can you try adding NEXT_PUBLIC_ infront of DATABASE_AUTH_TOKEN!, in the main.ts the same files you have added the NEXT_PUBLIC_ in the DATABASE_URL

Zaprovic commented 6 months ago

@ArshErgon I was considering to add the prefix to the token as well but wouldn't that expose the token to the browser? Or there is no problem in local development as long as in the production build I don't add that variable with the prefix?

ArshErgon commented 6 months ago

@Zaprovic yup, it will be visible to the client-side. I'm not expert in nextjs and I dont think theres other ways of accessing env in next, I google const url = import.meta.env.VITE_TURSO_DATABASE_URL?.trim(); use in next but its not possible I think.

Heres my current drizzle.config.ts file, its working try to copy my code in your file and lets pray it will work.

import type { Config } from 'drizzle-kit';
import * as dotenv from 'dotenv';
dotenv.config();

export default {
    dialect: 'sqlite',
    schema: './src/lib/server/database/drizzle-schemas.ts',
    out: './migrations',
    driver: "turso",
    dbCredentials: {
        url: process.env.VITE_TURSO_DATABASE_URL as string,
        authToken: process.env.VITE_TURSO_AUTH_TOKEN as string,
    }
} satisfies Config;

Edit: for nextjs the env file on vercel is name .env or .env.local? You can see when are uploading your .env file there, if thats .env then you may try to update this code of yours:

dotenv.config({
  path: ".env.local",
});
Axibord commented 5 months ago

So basically, everyone has the same problem, and there is still no fix for this. I guess it's the downside of using an SQL variant that nobody really uses in production...

penberg commented 5 months ago

@Axibord Please keep the comments relevant to the issue at hand. I am sure there are other good outlets for funny remarks like that.

notrab commented 5 months ago

@Zaprovic I highly recommend not to instantiate the "libSQL client" in client components and query data. If you're using Next.js, you can use server components to fetch data, and pass the data fetched to client components. If you want to show a loading spinner, you can use React Suspense with a fallback UI. Next.js docs has some good insights into using the App Router and Best Practices for Data Fetching.

notrab commented 5 months ago

Hey @rakis10

I'm looking at your original code snippet, and thinking….

You could simplify things by just using process.env.TURSO_URL and process.env.TURSO_AUTH_TOKEN directly.

Here’s what I mean:

require(‘dotenv’).config();

const { createClient } = require('@libsql/client')
const { drizzle } = require('drizzle-orm/libsql')

const client = createClient({
  url: process.env.TURSO_URL,
  authToken: process.env.TURSO_AUTH_TOKEN,
})

module.exports = drizzle(client)

Then in development, dotenv will load your .env. file automatically, and on Render, the process.env is populated with your environment variables at runtime.

If you have a full reproducer I can deploy to Render (if the above doesn’t work), please let me know.

feri-irawan commented 5 months ago

This problem is caused by undefined env variables, not libsql, to fix this make sure the env is added on the server and call it the right way. This solved my problem.

antscript commented 4 months ago

In my case, it was found that the reference to the database in the client-side rendered components caused the issue.

antscript commented 4 months ago

This problem is caused by undefined env variables, not libsql, to fix this make sure the env is added on the server and call it the right way. This solved my problem.

In client components, these environment variables cannot be accessed.

rahuls360 commented 2 months ago

In my case I had forgotten to do the following

import dotenv from "dotenv";
dotenv.config();