Bun binding problems: `dyld[XXX]: missing symbol called` and `error: could not locate the bindings file` causing unkillable process!

[buzzy]

Running the migrator.make() completely freezez the process and I am not even able to kill it using "kill -9"!

Diffing schema:
  • Capturing pragmas...dyld[80832]: missing symbol called

I am running this on an empty database + a very simple schema:

PRAGMA foreign_keys = off;
BEGIN TRANSACTION;

-- Table: sessions
CREATE TABLE "sessions" (
  id blob PRIMARY KEY NOT NULL,
  user blob NOT NULL,
  lastChecked timestamp NOT NULL
) WITHOUT ROWID;

-- Index: user
CREATE INDEX user ON sessions (user);

COMMIT TRANSACTION;
PRAGMA foreign_keys = on;

I am running this using Bun on a M2 Mac. Seems it's not compatible at all. Most likely this library (or a dependency) does not support non-x86 cpus yet. You might want to look into that.

[SanderGi]

Thanks for making me aware of this issue! I have been running things on an M-series Mac as well but using Node.js (compiled for arm64). Does it work for you with Node.js? I will look into the Bun compatibility to see if that might be the cause

[SanderGi]

Another note is, you don’t need the transaction in your schema. The migrator takes care of applying your schema within a transaction and temporarily disabling the foreign keys if they are enabled in your schema so you can remove those parts and just leave the one “PRAGMA foreign_keys = on;” at the top

[buzzy]

Thanks! I just exported the schema using SQLiteStudio without editing it as a test. Node might work. Haven't tested it as I don't want to mix runtimes in my environment. I also write in typescript, so running it in node would mean adding extra layers like transpilers etc, which I don't need with Bun. My CI-environment runs "normal" x86, so I am testing if it works fine there, but it would be nice to be able to test things locally on the Mac :)

[buzzy]

Nope, actually does not work on Bun on x86 either. My CI creates a production ready Bun-binary with all code and Bun-runtime in one file. Running that file gives error that package.json is missing. I do not get that problem if I don't import the "sqlite-auto-migrator" library, so something is definitely not playing nice with Bun.

Basically I am including the current .sql file inside my project (Bun binary file) and on first run, I have a "--update-database" flag, that compares the embedded .sql-file with the current database and performs the changes needed. That run fails with Bun.

[SanderGi]

Thanks for taking the time to investigate with x86 Bun. I just confirmed your schema works on M1 Mac with Node.js but the migrator breaks with Bun. I have pin-pointed the issue to be with running node-sqlite3 with Bun. Specifically, the Database.each() method does not bind correctly with Bun. The error can be recreated with:

import sqlite3 from 'sqlite3';

const db = new sqlite3.Database('data.db');
db.each('SELECT * FROM sessions', (err, row) => {
    console.log(row);
});

Which yields dyld[65381]: missing symbol called.

I have filed an issue with node-sqlite3. In the mean time, I have pushed a temporary patch to sqlite-auto-migrator. Let me know if upgrading sqlite-auto-migrator to version 1.1.4 does not fix things for you. It seems to work with my Bun version (1.1.29)

[buzzy]

Wow, that was quick work! I will test it now. Btw, Bun uses a own built-in sqlite library that is much faster than node-sqlite3, so I am not sure if the node-sqlite3 team will care :)

[buzzy]

Not sure if I am doing something wrong, but this is what I get:

[SanderGi]

Weird. What is your version of Bun? If it is an older version, you might need to add sqlite3 as a trusted dependency (https://github.com/oven-sh/bun/issues/5187)

[buzzy]

I tried it with 1.1.20 and latest 1.1.29. Same issue. I even tried clearing the Bun cache. Looks like it searches for the bindings in the wrong path:

I also tried removing my node_modules and bun.lockb and re-downloaded all libs.

[SanderGi]

Hmm, not sure why it searches in the wrong path for you but not for me. Did you install bun with curl -fsSL https://bun.sh/install | bash? Does the process.arch say 'arm64' and what is your macOS version? I'll try to see if I can reproduce your issue by clearing my cache

[buzzy]

Yes, I installed the official way with curl. Yes, it's a arm64 binary. The x86 Bun binary does not even start. Not even if I would run it with Rosetta.

[buzzy]

My test-script is very basic:

[buzzy]

Ah, for some reason, Bun is trying to get sqlite3 from it's cache instead of node_modules and in the cache, there is no sqlite3 binary:

[buzzy]

If I copy the sqlite3 binary from node_modules to cache, I get error, so I guess something is wrong with Bun.

[SanderGi]

It might be worth submitting a report to the Bun team. How did you copy things over to the cache? This is how my working cache looks: cache.zip

What does your bunfig.toml look like? Perhaps disabling the cache will fix things:

[install.cache]
# the directory to use for the cache
dir = "~/.bun/install/cache"

# when true, don't load from the global cache.
# Bun may still write to node_modules/.cache
disable = true

# when true, always resolve the latest versions from the registry
disableManifest = true

[buzzy]

Your cache looks exactly like mine. You can clearly see that there is no node_sqlite3.node binary in there. So if it works for you, it is definitely not using your cache, but the binary that exists in your node_modules dir. I do not use a bunfig.toml at all, so I use all defaults. I can try to disable the cache, but as the config says, it will still write to the .cache dir, so it sounds more like it just switches to another location.

For my test, I simply copied the entire contents from node_modules/sqlite3/* to the bun cache dir.

[SanderGi]

I see. I think the better way to get the binary into the cache directory is to cd ~/.bun/install/cache/sqlite3@5.1.7@@@1 and then run bun install (should be equivalent to copy pasting the installed node_modules/sqlite3 or setting the trustedDependencies but perhaps it makes a difference). I don't usually use Bun so I cannot explain why my installation finds the correct binary in the node_modules and yours does not, esp. since we both have no bunfigs. Perhaps you can try forcing it to use the node_modules one by providing a relative path? I'll keep trying to reproduce the issue on my M1 Mac

[SanderGi]

Update - @buzzy I reproduced your error message (from before messing with the missing binary in the cache) by running Bun from the parent directory:

And then reproduced the segmentation fault by running Bun globally with bunx:

As well as by adding the binary to the cache with cd ~/.bun/install/cache/sqlite3@5.1.7@@@1 && bun install and running from the parent directory:

A similar (but more informative) error happens with Node.js (since it does not default to running the global cached package and instead complains there is no local package found when running this way):

And no error occurs running using the global Node.js package (since it installs and binds the binary correctly):

Now that I can somewhat reproduce the error, I have found a potential work-around. Ideally Bun would work with binaries in the cache, but since it doesn't, you can make it aware of the node_modules binary by setting its working directory relative to the file being executed to guarantee it points at the folder with the package.json and node_modules:

import path from 'path';

process.chdir(path.join(__dirname, '..'));

import { Migrator } from 'sqlite-auto-migrator';

const migrator = new Migrator();

await migrator.make();
await migrator.migrate();

Something similar might work for you. Let me know if it does.

[buzzy]

Thank you! That actually solved the problem! So I guess Bun has a bug. I haven't tested it on "bun build" yet though. Hopefully it will still work when Bun bundles everything to one binary.

One question; Do I really need that "migration" table in the database? Is there a way to turn that off? As my CI simply has a sql-file with the latest schema and the CI always compares the "production" database with the schema and changes it, what exactly would I need the migration table for? I take a backup of the database before, so I don't really need any revert functionality.

[buzzy]

I have issues with it getting stuck and never finishes. It only happens randomly though.

When that happens, the tables are left in a locked state:

Also, is there a way to prevent it from creating the "migrations" table in the database and "migrations" directory? I do not need those as I am not checking in any versions or rollback features in git. I simply want it to compare the .sql-file with the database-file and create a "migration-file" in RAM and then execute it without leaving any traces.

[SanderGi]

Great! Happy to hear the workaround worked for you as well. Let me know whether it works with "bun build" when you test it

These are excellent inquiries. I have split them up into separate issues to make the conversation easier to follow. This issue will remain for the bun binary binding, #2 will be for the deadlock, #3 for the "migrations" table and directory issue