xhernandez
commented
2 years ago Hi @dalwise. I completely missed your previous update. I'm sorry.
Can you run this script on hostnam20 ?
#!/bin/bash
set -eEu
BRICK="${1:?You must pass the path to the root of the brick}"
BRICK="$(realpath "${BRICK}")"
if [[ ! -d "${BRICK}/.glusterfs" ]]; then
echo "'${BRICK}' doesn't seem to contain a brick" >&2
exit 1
fi
declare -A GFIDS
function resolve() {
local gfid="${1}"
local link ref
link="${GFIDS[${gfid}]-}"
if [[ -z "${link}" ]]; then
echo "${gfid} doesn't exist" >&2
GFIDS[${gfid}]="<missing>/"
elif [[ "${link}" == "../../.." ]]; then
GFIDS[${gfid}]="/"
elif [[ "${link:0:6}" == "../../" ]]; then
ref="${link:12:36}"
resolve "${ref}"
GFIDS[${gfid}]="${GFIDS[${ref}]}${link:49}/"
fi
}
while read gfid link; do
GFIDS[${gfid}]="${link}"
done < <(find "${BRICK}/.glusterfs" -type l -links 1 -printf "%f %l\n")
for gfid in "${!GFIDS[@]}"; do
resolve "${gfid}"
done
len="${#BRICK}"
while read gfid path; do
gfid="${gfid:0:8}-${gfid:8:4}-${gfid:12:4}-${gfid:16:4}-${gfid:20}"
path="${path:${len}}"
if [[ -z "${GFIDS[${gfid}]-}" ]]; then
echo "Directory without GFID (${gfid}): '${path}'" >&2
else
if [[ "${GFIDS[${gfid}]}" != "${path}" ]]; then
echo "Mismatching directory (${gfid}): '${path}' <-> '${GFIDS[${gfid}]}'" >&2
fi
unset GFIDS[${gfid}]
fi
done < <(find "${BRICK}" -path "${BRICK}/.glusterfs" -prune -o -type d -exec getfattr -e hex -n trusted.gfid --absolute-names {} \; |
sed -n '/^#\s*file\s*:/{N;s/^#\s*file\s*:\s*\(.*\)\ntrusted\.gfid\s*=\s*0x\(.*\)/\2 \1\//p}')
for gfid in "${!GFIDS[@]}"; do
echo "Orphan GFID (${gfid}): '${GFIDS[${gfid}]}'" >&2
doneTo run it, just pass the root directory of the brick. It will check if all directories are correctly defined. The assertion could be caused by a directory without its corresponding gfid.
dalwise
stale[bot]
Description of problem: Heal count keeps increasing. Current status after 13 days of increases is:
The exact command to reproduce the issue: The issue happened after trying to recover from an inaccessible directory (
/shared/vol_name/logs) within the volume mounted at/shared/vol_name. Trying to access this directory would return "Transport endpoint not connected" on all clients. Other directories in the mounted volume were not affected.gluster volume healshowed GFIDs in need of healing and the problematic directory in split brain. We were not able to get the files healed by using the gluster heal commands. We were then able to resolve problems with most GFIDs by removing the corresponding files in the bricks. However one GFID remained in need of healing on host_name20 and we could not determine what file it corresponded to. Since host_name20 just had the arbiter brick we tried removing it:That allowed us to access the directory that we could not see earlier. We then attempted to rejoin the arbiter with a clean brick:
The directory is still accessible, but the number of files in need of healing has been increasing for the last 13 days.
We can likely recover by simply backing up the files, destroying the current volume and then moving the files onto a newly created volume. However we are at a loss as to:
The full output of the command that failed: (heal count command above)
Expected results: Heal count to return to 0.
Mandatory info: - The output of the
gluster volume infocommand:- The output of the
gluster volume statuscommand:- The output of the
gluster volume healcommand: (Sharedheal countabove ashealcontains tens of millions of entries)- Provide logs present on following locations of client and server nodes - /var/log/glusterfs/
https://www.dropbox.com/s/p93wyyztj5bmzk8/logs.tgz
This compressed log file contains the logs for all three server nodes. The server nodes have the volume mounted and so are acting as clients also.
The volume name is the name of an internal project and has been changed to "vol_name" in command outputs and logs. The hostnames are also internal and have been changed to host_name20, host_name21 & host_name22.
**- Is there any crash ? Provide the backtrace and coredump No crash
Additional info:
- The operating system / glusterfs version:
Note: Please hide any confidential data which you don't want to share in public like IP address, file name, hostname or any other configuration