aep
commented
1 year ago npm test on nfs succeeds just fine
Closed aep closed 1 year ago
aep
commented
1 year ago npm test on nfs succeeds just fine
aep
commented
1 year ago the code that throws the error is:
async close(...e){
return LevelDatabase.#e.delete(this.#a),await this.compactFull(),super.close(...e)
}i dont know much javascript. doesnt this close the db before compact?
juliangruber
commented
1 year ago try this:
async close(...e){
await LevelDatabase.#e.delete(this.#a)
await this.compactFull()
return super.close(...e)
}You were running some operations in parallel
aep
commented
1 year ago i cant patch the code, it's proprietary software by @aaclayton
any idea how i can work around that, or if thats even the root cause?
aep
commented
1 year ago npm test also succeeds inside our production system. so clearly this must be related to how they use this library. i will attempt to figure out how to patch nodejs with gdb
aep
commented
1 year ago i figured out how to patch it. that solves that message but here's another one:
Error: Database is not open
at LevelDatabase.keys (/home/foundry/resources/app/node_modules/abstract-level/abstract-level.js:697:13)
at LevelDatabase.compactFull (file:///home/foundry/resources/app/dist/database/backend/level-database.mjs:8:445)
at LevelDatabase.close (file:///home/foundry/resources/app/dist/database/backend/level-database.mjs:4:20)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Promise.all (index 2)
at async Module.disconnect (file:///home/foundry/resources/app/dist/database/database.mjs:1:1667)
import {
ClassicLevel
} from "classic-level";
import SublevelDatabase from "./sublevel-database.mjs";
import fs from "fs";
export default class LevelDatabase extends ClassicLevel {
constructor(e, a, {
sublevels: t = [],
...s
} = {}) {
if (!e || !a) throw new Error("You must provide a unique database name and file path location");
if (LevelDatabase.#e.has(e)) throw new Error(`The database "${e}" is already open and cannot be re-created.`);
s.keyEncoding = "utf8", s.valueEncoding = "json", super(a, s), this.#a = e, LevelDatabase.#e.set(e, this), this.setMaxListeners(Math.max(10, t.length + 1));
const i = {
keyEncoding: s.keyEncoding,
valueEncoding: s.valueEncoding
};
for (const e of t) this.#t[e] = this.sublevel(e, i)
}
static async connect(e, a, t = {}) {
const s = new this(e, a, {
passive: !0,
createIfMissing: !0,
...t
});
try {
await s.open(), global.logger.info(`${global.vtt} | Connected to database "${e}"`)
} catch (a) {
throw a.message = `Failed to connect to database "${e}": ${a.message}`, a
}
return s
}
get name() {
return this.#a
}
#a;
get sublevels() {
return this.#t
}
#t = {};
static get databases() {
return LevelDatabase.#e
}
static #e = new Map;
static formatKey(...e) {
return e.join(".")
}
async close(...e) {
await LevelDatabase.#e.delete(this.#a)
await this.compactFull()
return super.close(...e)
}
async clone(e, a) {
if (this.constructor.databases.has(e) || a === this.location) throw new Error("The cloned database name and location must be unique");
const t = await this.constructor.connect(e, a, {
sublevels: Object.keys(this.sublevels)
}),
s = t.batch(),
i = this.iterator();
for await (const [e, a] of i) s.put(e, a);
return await i.close(), await s.write(), t
}
async destroy() {
await this.close(), fs.rmSync(this.location, {
recursive: !0
})
}
async compactFull() {
const e = this.keys({
limit: 1,
fillCache: !1
}),
a = await e.next();
await e.close();
const t = this.keys({
limit: 1,
reverse: !0,
fillCache: !1
}),
s = await t.next();
return await t.close(), this.compactRange(a, s, {
keyEncoding: "utf8"
})
}
async size() {
const e = this.keys({
limit: 1,
fillCache: !1
}),
a = await e.next();
await e.close();
const t = this.keys({
limit: 1,
reverse: !0,
fillCache: !1
}),
s = await t.next();
return await t.close(), this.approximateSize(a, s, {
keyEncoding: "utf8"
})
}
_sublevel(e, a) {
return new SublevelDatabase(this, e, a)
}
}```
vweevers
commented
1 year ago Hard to say what is happening here, without full (meaning non-minified) code and a way to reproduce. Typically "Database is not open" means there was an explicit call to db.close(), FWIW.
aep
commented
1 year ago i dont think its in their code. it looks like the call to s.open() is failing, and the rest is just spurious. gdb hasnt been helpful.
i finally figured out that node_modules/abstract-level/abstract-level.js is hiding the error in this._open(options, (err) => { from c which is
[Error: IO error: /data/Data/worlds/kmkm/data/messages/000005.ldb: No such device] {
code: 'LEVEL_IO_ERROR'
}the underlying issue is that that leveldb is unlinking a file and then writing to the open fd.
[pid 1963] unlink("/data/Data/worlds/kmkm/data/messages/000005.ldb") = 0
[pid 1953] write(1, "[Error: IO error: /data/Data/wor"..., 112[Error: IO error: /data/Data/worlds/kmkm/data/messages/000005.ldb: No such device] {
this is likely broken in NFS4 (?). I just dont understand why this only affecting this particular library, and not anything else leveldb related. is there a specific flag maybe
vweevers
commented
1 year ago why this only affecting this particular library
Perhaps because we use a rather old version of LevelDB here. What version(s) did you test against (in other libraries)?
aep
commented
1 year ago i tested leveldb 1.23, nothing else. maybe that just has an NFS workaround? its also possible this is a very unlucky combination, since NFS4 is fairly new and should natively support this, but i can see in strace that it clearly didnt work. i will investigate the nfs kernel client now.
aep
commented
1 year ago i still havent figured this out. Writing a deleted but open file works just fine on NFS.
I'd apprechiate if anyone has a small repro.
#include <stdio.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
int main() {
int fd = open("foo", O_CREAT | O_RDWR | O_TRUNC, 0644);
if (fd < 0) {
perror("open");
exit(1);
}
unlink("foo");
char *s = "hello world\n";
int len = strlen(s);
if (write(fd, s, len) != len) {
perror("write");
exit(1);
}
lseek(fd, 0, SEEK_SET);
char buf[1024];
int n = read(fd, buf, sizeof(buf));
if (n < 0) {
perror("read");
exit(1);
}
write(STDOUT_FILENO, buf, n);
close(fd);
return 0;
}
the story continues here: https://github.com/nfs-ganesha/nfs-ganesha/issues/980
I believe this is actually a kernel bug
what a rabbit hole
aep
commented
1 year ago i found it. the actual syscall failing is mmap. i think its just straight up broken on nfs3, and supposed to be correct on nfs4 but nobody can really confirm? this is likely the reason they say "nfs bad".
the only reason i did run into this and @felddy did not was that we tunnel NFS through virtiofs (i.e. fuse) and that enforces coherence, while nfs itself just accepts mmap anyway under conditions where it can lead to data loss.
https://gitlab.com/virtio-fs/virtiofsd/-/issues/105
thanks for providing a unique testcase. We have over 3K users and zero reports like this although eventually someone would have run into something related. I found plenty unrelated nfs bugs during this involuntary adventure,
we're (well.. the wife) users of the propriatary software mentioned in #63
they just say "must not use NFS" , and the reason appears somewhere in this library, since leveldb itself works just fine on NFS.
When opening a db on NFS, we just get
unfortunately the relevant code is obfuscated.
strace reveals no obvious reason why NFS would be a relevant factor. there is no call to flock.
https://github.com/felddy/foundryvtt-docker/issues/735