Bockeman
commented
4 years ago @xhernandez Thanks for your detailed response.
In your particular case I would recommend to disable gfid2path feature. You also seem to be using quota. Quota works on a per-directory basis, but given you have multiple hardlinks, I'm not sure if it makes sense (to which directory the quota should be accounted for ?). If not strictly necessary, I would also disable quota.
Please could you tell me how to disable the gfid2path feature, I cannot find it in the documentation.
I disabled quota.
I deleted all 809 files that were causing the [Argument list too long] that self-healing could not handle.
files that exist on more than one brickYou are using a replica. It's expected to have the same file in more than one brick.
I mean, files that exist on more than one brick for a given server. Cases like this result in
ls -l /srv/gluvol1/vault/bobw/files/home/.mozilla/firefox/6qw6a8eb.cockpit \
2>&1 | awk '/?/{print " " $0}'; date +\ \ %F\ %T
-????????? ? ? ? ? ? broadcast-listeners.json
-????????? ? ? ? ? ? prefs.js
2020-10-05 16:17:28dangling gfid files, where the named file has been deleted directly from a brick, but not the corresponding gfid fileYou should never do this. It can cause more troubles.
Obviously, I do not intentionally delete files directly from bricks, but I have found this is often the only way to resolve certain issues (like split-brain), but it is always possible with manual intervention like this that I could make a mistake.
I will attempt to getfattr and hardlink count for all files on all bricks, but I will need to be careful how I do that. There's no point on getfattr for each file, it only needs to be done once per inode. Given a "find" on a 11TB subset of all the data takes over 5 hours, this could take days.
mohit84
xhernandez
Description of problem: One brick on one server is offline and all attempts to bring it back online have failed. The corresponding brick on the other (of a replica 2) server is ok. Other bricks are ok.
The following do not clear the problem:
The problem appears to be similar to https://github.com/gluster/glusterfs/issues/1531 but the cause is different, and the number of volumes and bricks is different. (I note the observation comment regarding "replica 2" and split-brain, but the cost (time/effort) to recover from split-brain is manageable and usually due to external causes, such as a power cut.)
My urgent need is to find a way out of the current situation and bring back online brick06 on the second server. Not so urgent is the need for gluster to handle this condition in a graceful way and report to the user/admin what is the real cause of the problem and how to fix it (if it cannot be fixed automatically).
The exact command to reproduce the issue: Not sure what actually caused this situation to arise, but activity at the time was: Multiple clients, all active, but with minimal activity. Intense activity from one client (actually one of the two gluster servers), scripted "chown" on over a million files which had been running for over 5 hours and was 83% complete. edit or "sed -i" on a 500MB script file (but should not have tipped over the 22GB Mem + 8 GB Swap)
The full output of the command that failed:
Expected results: Some way to bring that brick back online.
- The output of the
gluster volume infocommand:- The operating system / glusterfs version: Fedora F32 Linux veriicon 5.8.15-201.fc32.x86_64 #1 SMP Thu Oct 15 15:56:44 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Linux verijolt 5.8.15-201.fc32.x86_64 #1 SMP Thu Oct 15 15:56:44 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux glusterfs 7.8
Additional info:
snippet from /var/log/messages
snippet from /var/log/glusterfs/bricks/srv-brick06.log