search

fangfufu / httpdirfs

A filesystem which allows you to mount HTTP directory listings or a single file, with a permanent cache. Now with Airsonic / Subsonic support!
Other
778 stars 59 forks source link

Error when accesing a file from different processes #126

Open juanep97 opened 1 year ago

juanep97 commented 1 year ago

I'm trying to access (read) a file from different processes, but httpdirfs (-f --cache) crashes:

thread 140111547192896: pthread_mutex_lock() failed, 22, Invalid argument

Oops! HTTPDirFS crashed! :(
backtrace() returned the following 12 addresses:
httpdirfs(exit_failure+0x32)[0x555acc1c78e2]
httpdirfs(+0x4996)[0x555acc1c7996]
httpdirfs(+0x4f1f)[0x555acc1c7f1f]
httpdirfs(Cache_read+0x168)[0x555acc1cb148]
httpdirfs(+0x81fe)[0x555acc1cb1fe]
/lib/x86_64-linux-gnu/libfuse.so.2(fuse_fs_read_buf+0x1e8)[0x7f6f24cea2f8]
/lib/x86_64-linux-gnu/libfuse.so.2(+0xd4c6)[0x7f6f24cea4c6]
/lib/x86_64-linux-gnu/libfuse.so.2(+0x16f1f)[0x7f6f24cf3f1f]
/lib/x86_64-linux-gnu/libfuse.so.2(+0x1769c)[0x7f6f24cf469c]
/lib/x86_64-linux-gnu/libfuse.so.2(+0x1421d)[0x7f6f24cf121d]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f6f24707ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126a40)[0x7f6f24799a40]

PS: thanks for the nice software!

fangfufu commented 1 year ago

Could you describe what you did in a bit more detail please?

juanep97 commented 1 year ago

I mounted a directory with some 100mb-300mb files in it and executed a program that needs to read them / load several of them. When this program executes in a single thread, it just takes time and they are downloaded and cached normally. But when this program executes in parallel (~6 processes), I get the error message above (using httpdirfs -f --cache).

fangfufu commented 1 year ago

What program is this? Which HTTP server are you using? I need to be able to reproduce your problem at my local end.

fangfufu commented 1 year ago

Also could you compile HTTPDirFS with this SHA 1e808448313b96b6080cc659a8929f4235ee5cb4 and see if the problem still occurs.

juanep97 commented 10 months ago

Hi, sorry for the long delay, I've been quite busy with academic life. I use HTTPDirFS as a workaround in https://github.com/juanep97/iop4 to mount a remote archive, and avoid having to download ~35GB of files during the CI. But as you can see, tests fail most of the time, because the program does not find the file where it should be / it stopped being mounted at some point (maybe after a connection error?). The version I use is the one in the ubuntu repo. I will try to check if the problem still occurs using that commit you mention.

fangfufu commented 10 months ago

No worries, I am a busy person too. ~I actually have a rough suspicion of what might be causing the problem. Since you are using this for science, I am going to put this on higher priority. When do you need this to be done by?~

Edit: Apologies, I got confused this issues with the other two issues that's due to curl_easy_escape. Debugging lock problem is not exactly easy.

Is this issue only observed within the CI, or does it happen in real life too? Could you point me to a CI run where this failure occur?

You could try and run httpdirfs in single thread mode by passing in -s. This is mentioned in the README under "Useful FUSE options"