Slow performance

[GoogleCodeExporter]

FreeBSD 7.2 + lusca 14371

Found that in my test environment (single-user setup), when transferring a
cached file (say 10MB) without any other proxy traffic, transfer begins by
sending anywhere from 0b up to 600KB, usually 300-600KB in web browsers,
and 0-200KB with wget. Here, it gets stuck, and the only way to get the
transfer moving is by giving squid some load, just spam refresh and that
would do the trick. (In later tests, I found it happens even for non-cached
files). Nonetheless, performance is impaired (a lot quicker in FreeBSD 8).
Using the same configuration on FreeBSD 8.0, there were no issues of such,
but actually some other ones which I won't discuss here.

http://www.cockcrotch.com/dump/squid

A sample (bad) video of the issue + my config

Original issue reported on code.google.com by biat...@gmail.com on 19 Jan 2010 at 6:51

[GoogleCodeExporter]

Can you please paste the config into the issue, or attach it? That way there's 
a permanent record.

Are the configurations being used the same on both the 7.2 FreeBSD/pfsense box 
and the 8.0 test box?

Original comment by adrian.c...@gmail.com on 19 Jan 2010 at 7:00

• Changed state: Accepted
• Added labels: Priority-High, Version-Head
• Removed labels: Priority-Medium, Version-1.0

[GoogleCodeExporter]

Original comment by biat...@gmail.com on 19 Jan 2010 at 7:07

Attachments:

• testsquidcfg.gz
• loader.conf
• sysctl.conf

[GoogleCodeExporter]

100% exact same configuration on vanilla 7.2, pfsense 1.2.3, freebsd 8.0

Original comment by biat...@gmail.com on 19 Jan 2010 at 7:08

[GoogleCodeExporter]

ok. so.

FreeBSD-7.2 vanilla has the same problems as pfsense-1.2.3, but freebsd-8.0 is 
fine?

Original comment by adrian.c...@gmail.com on 19 Jan 2010 at 7:13

[GoogleCodeExporter]

FreeBSD-7.2 Vanilla = Problem
pfsense 1.2.3 Final (freebsd 7.2) = Problem
FreeBSD-8.0 Vanilla = No problem (Actually there are other problems, if you'd 
like me
to report)

Original comment by biat...@gmail.com on 19 Jan 2010 at 7:20

[GoogleCodeExporter]

Create different issue's for the problems yuo're seeing in freebsd-8.0.

Original comment by adrian.c...@gmail.com on 19 Jan 2010 at 7:24

[GoogleCodeExporter]

I should add:

./configure '--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin'
'--datadir=/usr/local/etc/squid' '--libexecdir=/usr/local/libexec/squid'
'--localstatedir=/usr/local/squid' '--sysconfdir=/usr/local/etc/squid'
'--enable-removal-policies=heap' '--disable-linux-netfilter' 
'--disable-linux-tproxy'
'--disable-epoll' '--with-pthreads' '--enable-storeio=aufs,coss,null'
'--enable-delay-pools' '--enable-arp-acl' '--enable-pf-transparent'
'--enable-err-languages=English' '--enable-default-err-language=English'
'--prefix=/usr/local' '--disable-ident-lookups' 
'--enable-follow-x-forwarded-for'
'--with-large-files' '--enable-large-cache-files' 
'--enable-err-languages=English'
'--enable-default-err-language=English' '--mandir=/usr/local/man'
'--infodir=/usr/local/info/' '--enable-snmp' 'CFLAGS= -O2 -pipe 
-fno-strict-aliasing
-funroll-loops'

Original comment by biat...@gmail.com on 19 Jan 2010 at 9:36

[GoogleCodeExporter]

Ok, and to save me downloading the video. :) can you describe what you're doing 
to test the throughput?

Original comment by adrian.c...@gmail.com on 19 Jan 2010 at 9:43

[GoogleCodeExporter]

It's just a 4MB video. :)

1. Make sure proxy is unused. Configure firefox or whatever browser and wget to 
use
the proxy.
2. Use whichever to download any file, I used maplesea.com (games) and some 
linux rpm
files. A 4-10MB file would do for me, since my max download speed is 160KB/sec. 
If
your isp can transfer at 10MB/sec then you probably should try a larger file.
3. Now that the file is in your squid cache, just try to download the file 
using wget
or whatever you like. Make sure there's no other traffic. Transfer should stall.
Sometimes even problems starting with wget.
4. Start loading some websites in firefox.. spam refresh if you must, you'll 
find
that doing so would push the wget transfer.

I know it's weird, but what can I say?

Original comment by biat...@gmail.com on 19 Jan 2010 at 10:11

[GoogleCodeExporter]

Well, I've seen the occasional issue creep up where connections "hang" and 
"stall" on a busy proxy, but I never 
really sat down to figure out if its the proxy or the client or the server.

So, hm. Can you give me remote access to a box where this is happening (ie, a 
7.2-rel) box so I can do a bit of 
debugging?

Original comment by adrian.c...@gmail.com on 19 Jan 2010 at 10:42

[GoogleCodeExporter]

No problem. My test environment is at home on a pfsense 1.2.3 router (where i 
first
learned of this issue). So, this box meant to have lusca, and still has lusca 
up and
running, i setup another test box for vanilla freebsd. Right now, i've erased 
all of
the bsds and am installing archlinux to test lusca. To save your effort and 
time, I
am going to get 2-3 boxes up and running for you to test. one vanilla 
freebsd-7.2,
one freebsd-8.0 and maybe linux as well. You name it.

For this we best talk on IRC. I'll let you know when I'm ready as I would have 
to set
them up again. I have a couple of pcs for testing here, not to worry. What 
time's
good for you? I'm +0800 GMT

Original comment by biat...@gmail.com on 19 Jan 2010 at 10:56

[GoogleCodeExporter]

Tested in gentoo-linux, without sysctl.conf/loader.conf tweaks... working great 
as well.

Offtopic: I've read people say squid in linux isn't as good as in freebsd. do 
you
think so too? If so, how?

Original comment by biat...@gmail.com on 20 Jan 2010 at 6:45

[GoogleCodeExporter]

First weird thing:

In freebsd-7.2, with debug set to ALL,1, the comm loop timeout is always being 
hit. ie:

2010/01/21 12:26:20| comm_select: timeout 1000
2010/01/21 12:26:20| do_comm_select: 0 fds ready
2010/01/21 12:26:20| comm_select: time out
2010/01/21 12:26:20| comm_select: timeout 990
2010/01/21 12:26:20| do_comm_select: 0 fds ready
2010/01/21 12:26:20| comm_select: time out
2010/01/21 12:26:20| comm_select: timeout 979
2010/01/21 12:26:20| do_comm_select: 0 fds ready
2010/01/21 12:26:20| comm_select: time out
2010/01/21 12:26:20| comm_select: timeout 968
2010/01/21 12:26:20| do_comm_select: 0 fds ready
2010/01/21 12:26:20| comm_select: time out
2010/01/21 12:26:20| comm_select: timeout 957

That's likely not the cause of the issue, but it may be part of the issue.

Original comment by adrian.c...@gmail.com on 21 Jan 2010 at 4:35

[GoogleCodeExporter]

Yup, it's irrelevant. the "fast" poll method was enabled by aufs which sets the 
max poll time to 10msec (which 
isn't needed anymore, but I digress!) but the above timeout log entry was done 
before the max poll time was 
enforced. I should fix that.

Original comment by adrian.c...@gmail.com on 21 Jan 2010 at 4:48

[GoogleCodeExporter]

Ok, if I had to take a big guess, I'd say that the AUFS->main process 
notification method via that pipe FD is 
somehow broken and we're missing events. When the load is low, a missed event 
means everything stalls. When 
the load isn't low, a missed event isn't as big a deal because some 
subsequently queued and completed IO re-
tickles the completion queue to run.

Next step - replicating this locally on my FreeBSD-7.x box..

Original comment by adrian.c...@gmail.com on 21 Jan 2010 at 4:55

[GoogleCodeExporter]

Right. I can't replicate it but it is the core problem. I think I'm going to 
have to
figure out a useful, scalable replacement for inter-thread notification that 
doesn't
involve writing a byte for every completed event.

Leave it with me. I may re-enable the workaround for the time being.

Original comment by adrian.c...@gmail.com on 26 Mar 2010 at 1:44