Segmentation fault while creating an archive

ayushnix commented 1 month ago

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

BUG/ISSUE

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

borg 1.4.0

Operating system (distribution) and version.

Arch Linux, kernel version 6.6.49-1-lts

Hardware / network configuration, and filesystems used.

borg repository was being created on a 16TB Western Digital Ultrastar DC HC550 from a ZFS RAID 10 array locally.

How much data is handled by borg?

Approximately 2.2TB for now.

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg -p create -s -c 300 -C auto,lz4 -e data/videos/_testing /mnt/backup::{now}-{borgversion}-videos /data/videos

Describe the problem you're observing.

Fatal Python error: Segmentation faultata/videos/english-movies/sample-folder-name/sample.mkv                                                                                                

Current thread 0x000076c73cd25b80 (most recent call first):
  File "/usr/lib/python3.12/site-packages/borg/repository.py", line 1713 in write_put
  File "/usr/lib/python3.12/site-packages/borg/repository.py", line 1266 in put
  File "/usr/lib/python3.12/site-packages/borg/cache.py", line 956 in add_chunk
  File "/usr/lib/python3.12/site-packages/borg/archive.py", line 1257 in chunk_processor
  File "/usr/lib/python3.12/site-packages/borg/archive.py", line 1269 in process_file_chunks
  File "/usr/lib/python3.12/site-packages/borg/archive.py", line 1467 in process_file
  File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 699 in _process_any
  File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 797 in _rec_walk
  File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 834 in _rec_walk
  File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 834 in _rec_walk
  File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 834 in _rec_walk
  File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 619 in create_inner
  File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 687 in do_create
  File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 191 in wrapper
  File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 5309 in run
  File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 5391 in main
  File "/usr/sbin/borg", line 8 in <module>

Extension modules: borg.crypto.low_level, msgpack._cmsgpack, borg.chunker, borg.algorithms.checksums, borg.platform.posix, borg.platform.syncfilerange, borg.platform.linux, borg.hashindex, borg.item, borg.compress, llfuse (total: 11)
fish: Job 1, 'borg -p create -s -c 300 -C aut…' terminated by signal SIGSEGV (Address boundary error)

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Include any warning/errors/backtraces from the system logs

I'll resume the creation of my borg repository from the last checkpoint and I'll report back if I face more segfaults. Meanwhile, I also found a coredump. Here's the output from systemd-coredump.

Stack trace of thread 217360:
#0  0x000076c73c4a53f4 n/a (libc.so.6 + 0x963f4)
#1  0x000076c73c44c120 raise (libc.so.6 + 0x3d120)
#2  0x000076c73c44c1d0 n/a (libc.so.6 + 0x3d1d0)
#3  0x000076c73a50fcc0 n/a (checksums.cpython-312-x86_64-linux-gnu.so + 0x6cc0)
#4  0x000076c73c7a500d PyObject_Vectorcall (libpython3.12.so.1.0 + 0x1a500d)
#5  0x000076c73c789d71 _PyEval_EvalFrameDefault (libpython3.12.so.1.0 + 0x189d71)
#6  0x000076c73c84e395 PyEval_EvalCode (libpython3.12.so.1.0 + 0x24e395)
#7  0x000076c73c87263a n/a (libpython3.12.so.1.0 + 0x27263a)
#8  0x000076c73c86d4ef n/a (libpython3.12.so.1.0 + 0x26d4ef)
#9  0x000076c73c887b14 n/a (libpython3.12.so.1.0 + 0x287b14)
#10 0x000076c73c8873a1 _PyRun_SimpleFileObject (libpython3.12.so.1.0 + 0x2873a1)
#11 0x000076c73c886aff _PyRun_AnyFileObject (libpython3.12.so.1.0 + 0x286aff)
#12 0x000076c73c87f2c4 Py_RunMain (libpython3.12.so.1.0 + 0x27f2c4)
#13 0x000076c73c839eac Py_BytesMain (libpython3.12.so.1.0 + 0x239eac)
#14 0x000076c73c434e08 n/a (libc.so.6 + 0x25e08)
#15 0x000076c73c434ecc __libc_start_main (libc.so.6 + 0x25ecc)
#16 0x00005cb8329d4045 _start (python3.12 + 0x1045)
ELF object binary architecture: AMD x86-64

I can upload and link the coredump file if it helps. Let me know if I should.

ThomasWaldmann commented 1 month ago

Thanks for the bug report! The stack trace is not very useful due to missing symbols.

Can you check the discussion in #5899 about what can be done / what's needed to fix this?

I am personally not very experienced with debugging at that level, but it seems that you could maybe help with this (as long as you can reproduce it), please?

ThomasWaldmann commented 1 month ago

@ayushnix did you see my comment?

ayushnix commented 1 month ago

@ThomasWaldmann apologies for the late response, yes I did. Unfortunately, I don't have experience with using debug symbols and gdb to be able to pinpoint the issue. I thought the stack traces and core dump output might help so I posted it here.

I can try to follow specific steps if this issue issue happens the next time if I can get a short guide about what I should do.

ThomasWaldmann commented 1 month ago

Guess someone needs to find out how to build the compiled parts of borg with debug symbols and modify our setup.py accordingly. There is a line defining "cflags" in there.

Maybe using a python binary built with debug symbols is helpful also.

borgbackup / borg