search

trapexit / mergerfs-tools

Optional tools to help manage data in a mergerfs pool
ISC License
385 stars 44 forks source link

mergerfs.dup fails to backup symlinks #80

Closed anthonyryan1 closed 5 years ago

anthonyryan1 commented 5 years ago

I seem to be seeing an issue with files sometimes not being created with the required xattrs.

I have a script that caches thousands of bits of data to flat files on a mergerfs system, I deleted all the files, re-created them and the issue persisted.

In mergerfs.dup this manifests like

Traceback (most recent call last):
  File "/usr/bin/mergerfs.dup", line 347, in <module>
    main()
  File "/usr/bin/mergerfs.dup", line 291, in main
    relpath  = mergerfs_relpath(fullpath)
  File "/usr/bin/mergerfs.dup", line 193, in mergerfs_relpath
    return lgetxattr(fullpath,b'user.mergerfs.relpath').lstrip(b'/')
AttributeError: 'NoneType' object has no attribute 'lstrip'

What would be the best way to begin debugging this to provide you with a helpful report?

trapexit commented 5 years ago

It's not possible to not have the attribute. That don't actually exist.

How about the details of what you did? The command line? The file it fails on?

anthonyryan1 commented 5 years ago

The application is Plex, the file is actually hundreds of files in .config/plex/Plex Media Server/Metadata/Movies/f/*.bundle/Contents/_combined/Info.xml and many other small files in there. Info xmls, thumbnails, etc.

I deleted every failing file and even purged the entire Metadata directory then re-generated it and the issue persisted.

The precise command I'm running is mergerfs.dup -c 2 .config/

Here's an exact example from today:

# copy
rsync -avHAXWES --numeric-ids --progress --relative '/raw/41fdf564-3089-4eed-b154-f1fdf765a405/./.config/plex/Plex Media Server/Metadata/Movies/f/53709a8d81c12cb0f4f8d58004a25dd063de67c.bundle/Contents/_combined/Info.xml' /raw/e8ae6859-13a6-45f3-b724-8e11253256cc/

filepath = /storage/.config/plex/Plex Media Server/Metadata/Movies/f/53709a8d81c12cb0f4f8d58004a25dd063de67c.bundle/Contents/_combined/art/com.plexapp.agents.imdb_d4b35dc174b52b68886c072f755ee7fd9a907cce

Traceback (most recent call last):
  File "/usr/bin/mergerfs.dup", line 347, in <module>
    main()
  File "/usr/bin/mergerfs.dup", line 291, in main
    relpath  = mergerfs_relpath(fullpath)
  File "/usr/bin/mergerfs.dup", line 193, in mergerfs_relpath
    return lgetxattr(fullpath,b'user.mergerfs.relpath').lstrip(b'/')
AttributeError: 'NoneType' object has no attribute 'lstrip'

The filepath = line there was my own addition to the dup script to see which file it was checking when it errored out, on.

mergerfs.fsck ./ from the same directory also yielded no change.

trapexit commented 5 years ago

I need the actual file info. Run stat on the file in question.

anthonyryan1 commented 5 years ago 
snow /storage/.config # stat '/storage/.config/plex/Plex Media Server/Metadata/Movies/f/53709a8d81c12cb0f4f8d58004a25dd063de67c.bundle/Contents/_combined/art/com.plexapp.agents.imdb_0438bbbfec44df9d599a91248a1d36c8de8ce13f'
  File: /storage/.config/plex/Plex Media Server/Metadata/Movies/f/53709a8d81c12cb0f4f8d58004a25dd063de67c.bundle/Contents/_combined/art/com.plexapp.agents.imdb_0438bbbfec44df9d599a91248a1d36c8de8ce13f -> ../../com.plexapp.agents.imdb/art/0438bbbfec44df9d599a91248a1d36c8de8ce13f
  Size: 74          Blocks: 0          IO Block: 4096   symbolic link
Device: 1fh/31d Inode: 8804864972284  Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (  108/    plex)   Gid: (  117/    plex)
Access: 2019-04-03 08:57:56.801838972 -0400
Modify: 2019-04-03 08:57:56.801838972 -0400
Change: 2019-04-03 08:57:56.801838972 -0400
 Birth: -
anthonyryan1 commented 5 years ago

So I finished making a minimal test case now, the issue is definitely the backing up of symlinks. I'll see about getting a patch ready

mkdir Test
cd Test
touch testfile
ln -s testfile testlink
cd ..
mergerfs.dup Test/
trapexit commented 5 years ago

Yeah. I've been working on a fix. The problem is that symlinks can't have xattrs and I forgot that when I wrote the tool and I guess never tested it with a path that had them. I should have a fix soon.

trapexit commented 5 years ago

Could you try https://raw.githubusercontent.com/trapexit/mergerfs-tools/dup-symlinks/src/mergerfs.dup ?

anthonyryan1 commented 5 years ago

That does seem to be working now!