mohit84
commented
2 years ago Can you share the below info from the system(server and client both)
1) brick logs and client logs
2) cat /proc/cpuinfo
3) free -m
4) ps -aef | grep gluster
5) Run below script almost 3 to 5 min while the performance is slow
while [ 1 ]
do
high_cpu=100.0
for pid in `pgrep gluster`
do
cpu=`top -b -n 1 -p $pid | tail -1 | awk '{print $9}'`
if (( $(echo "$cpu > $high_cpu" |bc -l) ));
then
echo "Capture pstack for next 30 attempt date for pid $pid cpu usage is $cpu"
for i in {1..30}
do
echo "Capture cpu usage for attempt $i"
pstack $pid
sleep 3
done
fi
done
sleep 20
done
6) Generate statedump for brick and client process, to generate a statdump run below command
kill -SIGUSR1 <gluster_process>, the statedump will be save in /var/run/gluster
jkroonza
xhernandez
Description of problem:
We run a maildir structure off of a glusterfs filesystem (to allow multiple servers for redundancy, 2 in this cse). This filesystem is laid out as a 2x2 replica-distribute. The underlying bricks resides on dedicated mounts, running off a PERC H700 RAID controller, with 2 x 4TB NL-SAS drives. The two primary servers handles ~150 courier-imapd processes at any point in time, the two backups ~15 each (normally none).
We are also wondering if global threading pools will help in any way. There are also considerations of too many threads can cause more lock contention than required (which global threading claims to reduce). Is there thread profiling available to determine and feedback as to options here?
From time to time IO operations via FUSE start taking excessively long (>50s), frequently resulting in cases where we need to kill all accessing (aften blocking) processes (they die with umount -fl assuming that a kill has been sent prior). There seems to be a timing correlation between different clients, but not always. The code used to measure this latency (max 50s before we force-remount currently, roughly corresponds with - logging portion only):
Reason for the wait complexity and not using timeout is because timeout often blocks for far longer than 50s due to the touch going into uninterruptable wait. We've tried not doing the remount at all, and things just gets worse from there (load averages starts going through the roof and clients starts complaining verbosely, so it really is the lesser of two extreme evils).
This allows us to calculate an average and standard deviation for these touch times:
This gives on the four servers running clients:
The latter two servers are primarily used for inbound filtering, and is set to handle 1/11th of the overall load between them (haproxy, primary two servers set to weight 10, the two normally backup-only, set to weight 1 - as a debug to try and determine if it's FUSE or the bricks). We're currently copying in ~1.3TB worth of data via FUSE on the last server above (using rsync).
As can be plainly seen, the averages on the latter two servers are MUCH lower than on the primary two, somewhat expected since they don't access the filesystem as much. The problem has been with us for a while, but since we replaced the drives with NL-SAS compared to SATA it has definitely gotten better. It's not solved, however, it's currently worse than ever with the data being copied in.
Counting entries taking over 5 seconds only (arbitrary value), we get 99 and 157 entries on the two primary servers, and only one on the backup/standby servers. Please refer long_times.csv - fields are seconds from epoch, date/time, time taken for touch, server - in my opinion it's clear that there are clusters here in terms of time, and frequently involves multiple servers:
These servers has ~150 courier-imapd processes each at any given point in time. Numerous pop3d processes gets started and stopped (majority of users are using POP3 with a fair number using IMAP, currently holiday period in SA, so not normal load even).
Filesystem usage stats for the two filesystems on the bricks:
The inode discrepancy is the subject of another bug (#3196).
As you can plainly see the average file size is ~240KB/file. There are certain folders with more files than others.
The filesystems are ext4, with the following features set:
NOTE: No dir_index as this has been found to actually worsen readdir() performance instead of improving it. We're happy to switch this back on if there is some confidence that it will improve things overall, we'd like some way of preferably confirming this before making the change (It's significant filesystem downtime to enable/disable this feature, and we're going to have a fair number of gfid split brains as a result due to the way maildir functions).
Worst files/folder counts:
For uriel:
Plainly there are a handful of terrible folders, the worst seems to be a deleted messages which I'll also clean out momentarily (>540k files if summing the mail-a and mail-b sides together). Running time ls on these four folders on the bricks consistently gives 0.5s times on recurring ls runs, and between 10 and 30 seconds on the initial "cold" run (which is also indicative that disk caches etc is still playing a role here).
FUSE mount point (I can't think of any FLOCKs on this filesystem, so perhaps NFS is a potential alternative? Even though my benchmarks a few years back indicated that NFS performed worse than FUSE, especially with relation to IO latency), mounted from fstab, the resulting command:
/usr/sbin/glusterfs --reader-thread-count=32 --fuse-mountopts=noatime,nosuid,noexec,nodev --process-name fuse --volfile-server=localhost --volfile-id=gv_mail --fuse-mountopts=noatime,nosuid,noexec,nodev /var/spool/mailIncreasing the various thread counts helped, but it didn't solve the problem. Similar for the performance.cache-size
Unfortunately not all the documentation is clear as to where (client/server) options are being effected, and whether or not remounts/brick restarts are required. We've restarted all bricks since changing the last settings, and we've also remounted all clients.
We actioned similar mechanisms for touch directly on the underlying filesystems and are not seeing the same effects.
The exact command to reproduce the issue:
No specific simple way we could isolate and we've been at this since last week already.
The full output of the command that failed:
No command that's failing, things are just unacceptably slow. gluster volume performance info peek attached: performance.txt
Expected results:
We don't expect to see excessive latency spikes (we're not so sensitive that we need us range consistently, but we cannot afford for one blocking transaction to completely back-log the system).
Mandatory info: - The output of the
gluster volume infocommand:- The output of the
gluster volume statuscommand:- The output of the
gluster volume healcommand:**- Provide logs present on following locations of client and server nodes - /var/log/glusterfs/
**- Is there any crash ? Provide the backtrace and coredump
FUSE mount gets to a point where we have to force remount.
Additional info:
- The operating system / glusterfs version:
Gentoo Linux, glusterfs 10.1