Open GoogleCodeExporter opened 9 years ago
ok
i added
#include "../libcore/debug.h"
to file libiapp/comm_ips_tproxy4.c
now another error gets in which is
gcc -DHAVE_CONFIG_H -DDEFAULT_CONFIG_FILE=\"/etc/squid/squid.conf\" -I.
-I../include
-I. -I. -I../include -I../include -m64 -Wall -g -O2 -D_REENTRANT -MT tools.o
-MD
-MP -MF .deps/tools.Tpo -c -o tools.o tools.c
In file included from tools.c:40:
../libiapp/comm_ips_tproxy4.c: In function ‘comm_ips_restoreCapabilities’:
../libiapp/comm_ips_tproxy4.c:76: warning: implicit declaration of function
‘xcalloc’
../libiapp/comm_ips_tproxy4.c:76: warning: cast to pointer from integer of
different size
../libiapp/comm_ips_tproxy4.c:77: warning: cast to pointer from integer of
different size
../libiapp/comm_ips_tproxy4.c:103: warning: implicit declaration of function
‘xfree’
In file included from /usr/include/limits.h:153,
from /usr/lib/gcc/x86_64-linux-gnu/4.3.2/include-fixed/limits.h:122,
from /usr/lib/gcc/x86_64-linux-gnu/4.3.2/include-fixed/syslimits.h:7,
from /usr/lib/gcc/x86_64-linux-gnu/4.3.2/include-fixed/limits.h:11,
from /usr/include/sys/param.h:26,
from squid.h:170,
from tools.c:41:
/usr/include/bits/xopen_lim.h:95:6: error: missing binary operator before token
"("
/usr/include/bits/xopen_lim.h:98:7: error: missing binary operator before token
"("
In file included from squid.h:446,
from tools.c:41:
../include/util.h: At top level:
../include/util.h:72: error: conflicting types for ‘xcalloc’
../libiapp/comm_ips_tproxy4.c:76: error: previous implicit declaration of
‘xcalloc’
was here
../include/util.h:76: warning: conflicting types for ‘xfree’
../libiapp/comm_ips_tproxy4.c:103: warning: previous implicit declaration of
‘xfree’
was here
tools.c: In function ‘mail_warranty’:
tools.c:137: warning: ignoring return value of ‘system’, declared with
attribute
warn_unused_result
tools.c: In function ‘restoreCapabilities’:
tools.c:1132: error: too few arguments to function
‘comm_ips_restoreCapabilities’
make[3]: *** [tools.o] Error 1
make[3]: Leaving directory `/var/tmp/lusca-1.0-r13795/src'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/var/tmp/lusca-1.0-r13795/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/var/tmp/lusca-1.0-r13795/src'
make: *** [all-recursive] Error 1
regards
Original comment by degreane@gmail.com
on 17 Feb 2009 at 1:01
You can try LUSCA_HEAD and see if works, here are working... (LUSCA_HEAD r13810)
Original comment by ajcor...@gmail.com
on 19 Feb 2009 at 4:29
[deleted comment]
Ok i used the LUSCA_HEAD ... I have to admit
compilation was successful yet.......
i didn't touch anything of the tproxy 5 rules found at balbit for iptables and
iproute2
ip rule add fwmark 0x1/0x1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -m physdev --physdev-in eth1
-j DIVERT
# DIVERT chain: mark packets and accept
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
iptables -t mangle -A PREROUTING -p tcp --dport 80 -m physdev --physdev-in
eth1 -j
TPROXY --tproxy-mark 0x1/0x1 --on-port 33128
and as stated and was in squid.conf
http_port 33128 tproxy
but alas it was a failure
i switched back to squid-3.1 and here it was no changes and everything running
fine ....
can someone test the configuration and say how it works for if you verify it
works
then it has another method other than stated at balbit website -;
Regards
Original comment by degreane@gmail.com
on 19 Feb 2009 at 1:58
Try "http_port 3128 transparent tproxy" .
Also, what is logged in cache.log ?
Original comment by adrian.c...@gmail.com
on 19 Feb 2009 at 2:49
Ok here'r my ./configure
squid -v
Squid Cache: Version LUSCA_HEAD
configure options: '--prefix=/usr' '--exec_prefix=/usr' '--bindir=/usr/sbin'
'--sbindir=/usr/sbin' '--libexecdir=/usr/lib/squid' '--sysconfdir=/etc/squid'
'--localstatedir=/var/spool/squid' '--datadir=/usr/share/squid'
'--enable-async-io'
'--with-pthreads' '--enable-storeio=ufs,aufs,coss,diskd,null'
'--enable-linux-netfilter' '--enable-arp-acl' '--enable-epoll'
'--enable-removal-policies=lru,heap' '--enable-snmp' '--enable-delay-pools'
'--enable-htcp' '--enable-cache-digests' '--enable-underscores'
'--enable-referer-log' '--enable-useragent-log'
'--enable-auth=basic,digest,ntlm,negotiate'
'--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-carp'
'--enable-follow-x-forwarded-for' '--with-large-files' '--with-maxfd=65536'
'amd64-debian-linux' '--enable-linux-tproxy4' 'build_alias=amd64-debian-linux'
'host_alias=amd64-debian-linux' 'target_alias=amd64-debian-linux' 'CFLAGS=-Wall
-g
-O2' 'LDFLAGS=-Wl,-Bsymbolic-functions' 'CPPFLAGS='
and here's my cache.log
Starting Squid Cache version LUSCA_HEAD for x86_64-debian-linux-gnu...
2009/02/19 19:01:53| Process ID 26164
2009/02/19 19:01:53| With 1024 file descriptors available
2009/02/19 19:01:53| Using epoll for the IO loop
2009/02/19 19:01:53| Adding nameserver 207.226.54.2 from /etc/resolv.conf
2009/02/19 19:01:53| DNS Socket created at 0.0.0.0, port 36604, FD 7
2009/02/19 19:01:53| User-Agent logging is disabled.
2009/02/19 19:01:53| Referer logging is disabled.
2009/02/19 19:01:53| logfileOpen: opening log /var/log/squid/access.log
2009/02/19 19:01:53| Unlinkd pipe opened on FD 11
2009/02/19 19:01:53| Swap maxSize 102400 + 8192 KB, estimated 8507 objects
2009/02/19 19:01:53| Target number of buckets: 425
2009/02/19 19:01:53| Using 8192 Store buckets
2009/02/19 19:01:53| Max Mem size: 8192 KB
2009/02/19 19:01:53| Max Swap size: 102400 KB
2009/02/19 19:01:53| Local cache digest enabled; rebuild/rewrite every
3600/3600 sec
2009/02/19 19:01:53| Store logging disabled
2009/02/19 19:01:53| Rebuilding storage in /var/spool/squid/cache (CLEAN)
2009/02/19 19:01:53| Using Least Load store dir selection
2009/02/19 19:01:53| Set Current Directory to /var/spool/squid/cache
2009/02/19 19:01:53| Loaded Icons.
2009/02/19 19:01:54| Accepting transparently proxied HTTP connections at
0.0.0.0,
port 33128, FD 12.
2009/02/19 19:01:54| Accepting ICP messages at 0.0.0.0, port 3130, FD 13.
2009/02/19 19:01:54| Accepting HTCP messages on port 4827, FD 14.
2009/02/19 19:01:54| Accepting SNMP messages on port 3401, FD 15.
2009/02/19 19:01:54| WCCP Disabled.
2009/02/19 19:01:54| Ready to serve requests.
2009/02/19 19:01:54| Done reading /var/spool/squid/cache swaplog (419 entries)
2009/02/19 19:01:54| Finished rebuilding storage from disk.
2009/02/19 19:01:54| 419 Entries scanned
2009/02/19 19:01:54| 0 Invalid entries.
2009/02/19 19:01:54| 0 With invalid flags.
2009/02/19 19:01:54| 419 Objects loaded.
2009/02/19 19:01:54| 0 Objects expired.
2009/02/19 19:01:54| 0 Objects cancelled.
2009/02/19 19:01:54| 0 Duplicate URLs purged.
2009/02/19 19:01:54| 0 Swapfile clashes avoided.
2009/02/19 19:01:54| Took 0.3 seconds (1587.2 objects/sec).
2009/02/19 19:01:54| Beginning Validation Procedure
2009/02/19 19:01:54| Completed Validation Procedure
2009/02/19 19:01:54| Validated 419 Entries
2009/02/19 19:01:54| store_swap_size = 2916k
2009/02/19 19:01:54| storeLateRelease: released 0 objects
and here's my squid.conf
acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src 207.226.54.0/24
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow manager all
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow all
http_access deny all
icp_access allow localnet
icp_access deny all
http_port 33128 transparent tproxy
access_log /var/log/squid/access.log squid
cache_log /var/log/squid/cache.log
cache_store_log none
pid_filename /var/run/squid.pid
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
via on
acl apache rep_header Server ^Apache
cache_effective_user proxy
visible_hostname squid-test
forwarded_for off
coredump_dir /var/spool/squid/cache
and here's my startup network file
#!/bin/sh
## General variables defenition of the system
BRCTL='/usr/sbin/brctl'
IPT='/sbin/iptables'
IP='/sbin/ip'
IPT_SAVE='/sbin/iptables-save'
IPT_RESTORE='/sbin/iptables-restore'
TC='/sbin/tc'
## End of General Variable defenition
## Interface configuration
## Bring Up the system ip interfaces and the bridge
$BRCTL addbr br0
$BRCTL addif br0 eth0
$BRCTL addif br0 eth1
$IP link set eth0 up
$IP link set eth1 up
$IP link set br0 up
$IP add add 207.226.54.226/24 dev br0
$IP ro replace default via 207.226.54.1
## End of Interface Configuration
and here's my tproxy iptables/iproute2 file
#!/bin/bash
ip ru del fwmark 1 lookup 100
ip ro del local 0.0.0.0/0 dev lo table 100
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
iptables -F -t nat
iptables -F -t mangle
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -m physdev --physdev-in eth1
-j DIVERT
# DIVERT chain: mark packets and accept
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
iptables -t mangle -A PREROUTING -p tcp --dport 80 -m physdev --physdev-in
eth1 -j
TPROXY --tproxy-mark 1 --on-port 33128
and still it doesn't work
when i switch back to squid-3.1 i only change
http_port 33128 tproxy
and it works fine
regards
Original comment by degreane@gmail.com
on 19 Feb 2009 at 3:04
There is something whacked about this. Why is there no debugging at all?
Are you starting lusca as root? This tproxy4 module of mine does the same Linux
Capabilities related
permissions stuff as tproxy2, even though it -may- not be needed. The Squid-3
code wasn't exactly clear cut on
the correct implementation.
Original comment by adrian.c...@gmail.com
on 19 Feb 2009 at 3:49
Am starting Lusca as
squid -D -f /etc/squid/squid.conf
bear in mind that kernel is 2.6.28 with tproxy4 as it comes in it and iptables
1.4.0-patched from balbit
can you make it and compile it in your own system and check it for yourself ????
if not then send me exact conf files or compilation steps
if you need my compilation steps i'll be glad to post it .....
Regards
Original comment by degreane@gmail.com
on 19 Feb 2009 at 3:54
does squid needs to be run as explicit user root ???? for it to work !????
Original comment by degreane@gmail.com
on 19 Feb 2009 at 4:26
just try it as root. If it works, i'll disable the checks in the tproxy4 code
so it doesn't need to.
Original comment by adrian.c...@gmail.com
on 19 Feb 2009 at 4:59
Original comment by adrian.c...@gmail.com
on 19 Feb 2009 at 5:00
no it doesn't run as root ...:(
Original comment by degreane@gmail.com
on 19 Feb 2009 at 5:16
I am going to setup tproxy2
await my results
:)
and tell me if anything specific needs to be done
regards
Original comment by degreane@gmail.com
on 19 Feb 2009 at 5:17
Hm, I really need tproxy4 tested though. I'll go over the Squid-3 code again to
see what is being done differently
and let you know when I've updated LUSCA_HEAD with some stuff for you to try.
Thanks!
Original comment by adrian.c...@gmail.com
on 19 Feb 2009 at 5:22
I am installing CentOS 5.2 x86_64 and will get to setup LUSC_HEAD with tproxy2
as it is kernel 2.6.18 with the patches from balbit
hope i shall get lucky with this
much regards
Adrian are there any special notes or tweaks for x86_64 or just the usual
compilation
????
Stay tuned for next episode
Original comment by degreane@gmail.com
on 20 Feb 2009 at 11:43
Lusca is still far from achieving any good result with T-Proxyv4
i wish one would work on this soon
Original comment by degreane@gmail.com
on 31 Mar 2009 at 9:47
I'll work on it next week when I am back in Perth.
Original comment by adrian.c...@gmail.com
on 1 Apr 2009 at 5:10
One Question
I use now Lusca hoping and anxiously waiting tproxy4 to work on it as it should
be
ported the same as in 3.1 as Balabit says ... never the less i have certain
issues
with using Lusca now the cvs and as i notice these in my cache.log
2009/04/01 20:21:19| neighborsHtcpClear: clear reason: 1
2009/04/01 20:21:19| ctx: exit level 0
2009/04/01 20:21:19| ctx: enter level 0:
'http://by139w.bay139.mail.live.com/mail/mail.fpp?cnmn=Microsoft.Msn.Hotmail.Ui.
Fpp.MailBox.GetInboxData&a=Y9HA4g76H5Jpr9Nvlfc%252b%252bg%253d%253d&au=216602005
9&ptid=0'
2009/04/01 20:21:19| neighborsHtcpClear: clear reason: 1
2009/04/01 20:21:19| ctx: exit level 0
2009/04/01 20:21:19| ctx: enter level 0:
'http://app.sweetim.com/sweetim/dispatcher'
2009/04/01 20:21:19| neighborsHtcpClear: clear reason: 1
2009/04/01 20:21:20| ctx: exit level 0
2009/04/01 20:21:20| ctx: enter level 0:
'http://www.facebook.com/ajax/chat/buddy_list.php'
2009/04/01 20:21:20| neighborsHtcpClear: clear reason: 1
2009/04/01 20:21:20| ctx: exit level 0
2009/04/01 20:21:20| ctx: enter level 0:
'http://www.facebook.com/ajax/chat/settings.php'
2009/04/01 20:21:20| neighborsHtcpClear: clear reason: 1
2009/04/01 20:21:20| ctx: exit level 0
2009/04/01 20:21:20| ctx: enter level 0:
'http://www.facebook.com/ajax/presence/notifications_read.php'
2009/04/01 20:21:20| neighborsHtcpClear: clear reason: 1
2009/04/01 20:21:20| ctx: exit level 0
2009/04/01 20:21:20| ctx: enter level 0: 'http://frankfurt.ebuddy.com/dispatch'
2009/04/01 20:21:20| neighborsHtcpClear: clear reason: 1
2009/04/01 20:21:20| ctx: exit level 0
2009/04/01 20:21:20| ctx: enter level 0:
'http://flashservice.xvideos.com/flashservices/gateway.php'
2009/04/01 20:21:20| neighborsHtcpClear: clear reason: 1
2009/04/01 20:21:20| ctx: exit level 0
2009/04/01 20:21:20| ctx: enter level 0: 'http://users.conduit.com/iis2ebs.asp'
2009/04/01 20:21:20| neighborsHtcpClear: clear reason: 1
2009/04/01 20:21:20| ctx: exit level 0
2009/04/01 20:21:20| ctx: enter level 0:
'http://www.facebook.com/ajax/chat/buddy_list.php'
2009/04/01 20:21:20| neighborsHtcpClear: clear reason: 1
2009/04/01 20:21:21| ctx: exit level 0
2009/04/01 20:21:21| ctx: enter level 0:
'http://www.facebook.com/ajax/chat/buddy_list.php'
2009/04/01 20:21:21| neighborsHtcpClear: clear reason: 1
2009/04/01 20:21:21| ctx: exit level 0
2009/04/01 20:21:21| ctx: enter level 0:
'http://services.mystuff.u-page.com/MyStuffService.asmx/LegacyLogin'
2009/04/01 20:21:21| neighborsHtcpClear: clear reason: 1
2009/04/01 20:21:21| ctx: exit level 0
2009/04/01 20:21:21| ctx: enter level 0:
'http://zcom.zynga.com/zcom/update_user.php'
2009/04/01 20:21:21| neighborsHtcpClear: clear reason: 1
2009/04/01 20:21:21| ctx: exit level 0
2009/04/01 20:21:21| ctx: enter level 0:
'http://app.sweetim.com/sweetim/dispatcher'
2009/04/01 20:21:21| neighborsHtcpClear: clear reason: 1
2009/04/01 20:21:21| ctx: exit level 0
2009/04/01 20:21:21| ctx: enter level 0:
'http://207.46.109.48/gateway/gateway.dll?SessionID=1753807654.1513022840'
2009/04/01 20:21:21| neighborsHtcpClear: clear reason: 1
2009/04/01 20:21:21| ctx: exit level 0
2009/04/01 20:21:21| ctx: enter level 0:
'http://sn103w.snt103.mail.live.com/mail/mail.fpp?cnmn=Microsoft.Msn.Hotmail.Ui.
Fpp.MailBox.GetInboxData&a=vAETEZfvAkarzFjDSQDFXA%253d%253d&au=3705948697&ptid=0
'
2009/04/01 20:21:21| neighborsHtcpClear: clear reason: 1
^C
and also check out my squidclient info and check the near hits value its very
high
why !????
HTTP/1.0 200 OK
Server: Lusca/LUSCA_HEAD
Date: Wed, 01 Apr 2009 17:20:38 GMT
Content-Type: text/plain
Expires: Wed, 01 Apr 2009 17:20:38 GMT
X-Cache: MISS from c1.comnet.net.lb
X-Cache-Lookup: MISS from c1.comnet.net.lb:3128
Via: 1.0 c1.comnet.net.lb:3128 (Lusca/LUSCA_HEAD)
Connection: close
Squid Object Cache: Version LUSCA_HEAD
Start Time: Wed, 01 Apr 2009 16:48:55 GMT
Current Time: Wed, 01 Apr 2009 17:20:38 GMT
Connection information for squid:
Number of clients accessing cache: 9
Number of HTTP requests received: 119165
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 3757.4
Average ICP messages per minute since start: 0.0
Select loop called: 1555461 times, 1.223 ms avg
Cache information for squid:
Request Hit Ratios: 5min: 14.6%, 60min: 11.3%
Byte Hit Ratios: 5min: 8.0%, 60min: 6.1%
Request Memory Hit Ratios: 5min: 0.5%, 60min: 8.7%
Request Disk Hit Ratios: 5min: 59.7%, 60min: 58.8%
Storage Swap size: 308884 KB
Storage Mem size: 9392 KB
Mean Object Size: 10.77 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.58309 0.58309
Cache Misses: 0.68577 0.64968
Cache Hits: 0.00000 0.00000
Near Hits: 122.32249 122.32249
Not-Modified Replies: 0.00000 0.00000
DNS Lookups: 0.00000 0.00000
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 1902.868 seconds
CPU Time: 53.231 seconds
CPU Usage: 2.80%
CPU Usage, 5 minute avg: 2.76%
CPU Usage, 60 minute avg: 2.78%
Process Data Segment Size via sbrk(): 38844 KB
Maximum Resident Size: 0 KB
Page faults with physical i/o: 0
Memory usage for squid via mallinfo():
Total space in arena: 38844 KB
Ordinary blocks: 38643 KB 433 blks
Small blocks: 0 KB 0 blks
Holding blocks: 70224 KB 6 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 201 KB
Total in use: 108867 KB 100%
Total free: 201 KB 0%
Total size: 109068 KB
Memory accounted for:
Total accounted: 32674 KB
memPoolAlloc calls: 17393677
memPoolFree calls: 17219372
File descriptor usage for squid:
Maximum number of file descriptors: 65000
Largest file desc currently in use: 3974
Number of file desc currently in use: 3966
Files queued for open: 0
Available number of file descriptors: 61034
Reserved number of file descriptors: 100
Store Disk files open: 188
IO loop method: epoll
Internal Data Structures:
29765 StoreEntries
1240 StoreEntries with MemObjects
27 Hot Object Cache Items
28675 on-disk objects
much regards and hope to be helping in providing a good working program
awaiting your set-up next week
Original comment by degreane@gmail.com
on 1 Apr 2009 at 5:27
I've committed a fix to the htcp message problem in LUSCA_HEAD revision 19320.
How does it run for you with TPROXY4 now?
Original comment by adrian.c...@gmail.com
on 10 Apr 2009 at 4:17
Anyway. To cut a long story short, we figured out what was going on.
To summarise:
* way way too much logging (eg the htcp clear stuff) which I've subsequently
quietened;
* tproxy4 doesn't need netfilter and enabling it (at least in the way I've
setup the
tproxy/interception code) will confuse and break things;
* various other sundry bits and pieces to his network and box config to make
tproxy
fully work.
Anyway. Lusca+tproxy4 should now work just fine. I'm leaving this ticket open
for now
to remind me to write some damned documentation on how it works and how to
configure it.
Original comment by adrian.c...@gmail.com
on 8 Jul 2009 at 4:59
You still need to write the documentation Addrian my friend
we all waiting for the doccumentation for T-Proxy support with lusca and all
FreeBSd kernel + PF stuff or Linux Kernel + iptables stuff
much regards
Original comment by degreane@gmail.com
on 23 Nov 2010 at 7:15
In fact I'm ancious waiting too.
Original comment by int...@gmail.com
on 10 Feb 2011 at 1:09
Original issue reported on code.google.com by
degreane@gmail.com
on 17 Feb 2009 at 12:03