search

madler / pigz

A parallel implementation of gzip for modern multi-processor, multi-core machines.
http://zlib.net/pigz/
2.65k stars 175 forks source link

pigz: can't destroy locked resource (pigz.c:2015:mutex_destroy) pigz: abort: internal threads error #115

Closed VOMAY closed 9 months ago

VOMAY commented 9 months ago

Hello,

I am currently running this command on a 4GB+ file as part of a csh script hundreds of times on similar files.

pigz -c test.ntd > /mnt/data/workspace/test.ntd.gz

The file test.ntd is located on a local SSD and /mnt/data/workspace/ is a network drive (which is not particularly quick). I have been running the script hundreds of times and occasionally get this error.

pigz: can't destroy locked resource (pigz.c:2015:mutex_destroy) pigz: abort: internal threads error

I cannot reproduce intentionally and I assume it is somehow related to IO-load or network traffic (CPU and RAM load is fine). This is what I get if I test the output file: pigz -t /mnt/data/workspace/test.ntd.gz pigz: skipping: /mnt/data/workspace/test.ntd.gz: corrupted -- incomplete deflate data

For the time being, I am rerunning the command in a while loop until it works.

System:

madler commented 9 months ago

I tried replicating this with a slow network drive, without success. How often is "occasionally", as a fraction of the number of runs?

VOMAY commented 9 months ago

It occurs about 1/10.

Something else, which is likely out of the norm: I am working on a powerful Intel machine with 64 logical CPU. About half of the cores are busy crunching numbers.

I am not limiting the number of pigz threads (-p option). Maybe the issue is related to uncommonly high/unlimited number of threads.

Anyway, I forgot to mention how much I like pigz. It does help me a lot! Thanks.for your efforts.

madler commented 9 months ago

What versions of gcc and glibc were used to compile pigz?

VOMAY commented 9 months ago

gcc version 7.4.1 20190424 [gcc-7-branch revision 270538] (SUSE Linux) ldd (GNU libc) 2.26

madler commented 9 months ago

Ah. SUSE. That rings a bell. You may want to try updating your operating system, as there was a bug in SUSE that manifested in pthreads and caused issues with pigz.

VOMAY commented 9 months ago

If the choice of OS was only my choice... Thanks for your effort anyway. I'll keep promoting the tool.

Something unrelated: The '-c' option works, but I'd much rather specify the output destination explicitly like one can do with tar. I understand this conflicts with the group compatibility idea, but would improve user-friendliness. E.g. pigz test.txt -C /mnt/netstorage/, which will result in /mnt/netstorage/test.txt.gz`

madler commented 9 months ago

I will add a -o, --output option.