Acquiring more than 32 file locks prevents any other file operations

[Sembiance]

Version: Deno 1.40.5

This code attempts to .lock() on a file, read/write a different file and then release the lock. It's called 33 times all at once. All 33 correctly wait on .lock(true) and 1 correctly acquires the lock. However that 1 then hangs forever on the read operation Deno.readTextFile() (though Deno.open here also locks).

If you change the counter from 33 to 32 then everything works perfectly. Feels like a deno bug?

Is this some sort of internal deno resource limit I'm hitting?

ulimit -x shows unlimited and nothing else from ulimit a is anywhere close to 32.

Run with: deno run --allow-write --unstable-fs --allow-read lockLimitTest.js

const COUNTER_FILE_PATH = "/tmp/test.counter";
await Deno.writeTextFile(COUNTER_FILE_PATH, "0");

async function loadCounter()
{
    console.log("opening lock file");
    const lockFile = await Deno.open("/tmp/test.lock", {append : true, create : true});

    console.log("acquiring lock");
    await lockFile.lock(true);

    console.log("lock acquired, reading counter");
    const counter = +(await Deno.readTextFile(COUNTER_FILE_PATH));  // this is where it sticks

    console.log("writing counter");
    await Deno.writeTextFile(COUNTER_FILE_PATH, (counter+1).toString());

    console.log("releasing lock");
    await lockFile.unlock();

    console.log("closing lock file");
    lockFile.close();

    return counter;
}

const r = [];
for(let i=0;i<33;i++)   // if you reduce this to 32, it works fine
    r.push(loadCounter());

console.log(await Promise.all(r));

[littledivy]

What is your file system and kernel (assuming Linux)?

[littledivy]

Oh I see what's happening:

• We have a max tokio threadpool limit of 32.
• Each flock takes up a thread (because we use spawn_blocking)
• Any other threadpool tasks like readFile and writeFile hang because of the flock.

I'm not sure what a fix for this would look like.

[Sembiance]

I’m not too familiar with rust. But my questions would be:

• If the tokio threadpool is increased, what performance/memory impact would that have?
• Can the threadpool be made to dynamically grow at runtime as needed?
• Could a deno CLI flag or environmental variable be added to change the threadpool size?
• Does this limit also mean that no more than 32 IO operations can take place at once? File and/or network?
• Could .flock() be optimized to check first if the same file is already locked in the current deno process and then instead of consuming a tokio thread it would wait in some other manner for the other flock to release? Although I suppose I could do that on the user side.

[rracariu]

Oh I see what's happening:

• We have a max tokio threadpool limit of 32.

would this be the cause for this issue also: https://github.com/denoland/deno/issues/22207 ?

[bartlomieju]

@rracariu no, it's unrelated. Worker threads use OS threads, not Tokio threadpool.

[lucacasonato]

We could use OS threads for async flock operations rather than spawn_blocking? Or we could manually serialize the order of flock calls, and only ever call tokio with flock once per path?

[littledivy]

Using OS threads directly sounds better than serializing flock calls in tokio threadpool (it could still block with >32 files i think?)