search

google-code-export / lusca-cache

Automatically exported from code.google.com/p/lusca-cache
0 stars 0 forks source link

cache miss after restart server with store_rewrite_list #105

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?

1.lusca with store_rewrite_list

What is the expected output? What do you see instead?

proxy-box4#tail -f /var/log/squid/cache.log
Set-Cookie: pfuid=ClIoJku+jSWshWG1BNn7Ag==; expires=Mon, 06-Apr-20 02:12:53
GMT; domain=.opt.fimserve.com; path=/}
2010/04/09 09:56:00| ctx: exit level  0
2010/04/09 09:56:00| squidaio_queue_request: WARNING - Queue congestion
2010/04/09 14:22:56| httpReadReply: FD 54: skipping 1xx response!
2010/04/09 14:22:59| httpReadReply: FD 54: skipping 1xx response!
2010/04/09 14:23:44| httpReadReply: FD 31: skipping 1xx response!
2010/04/09 14:23:44| httpReadReply: FD 31: skipping 1xx response!

What version of the product are you using? On what operating system?

proxy-box4# squid -v 
Squid Cache: Version  LUSCA_HEAD-r14535
configure options:  '--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin'
'--datadir=/usr/local/etc/squid' '--libexecdir=/usr/local/libexec/squid'
'--localstatedir=/usr/local/squid' '--sysconfdir=/usr/local/etc/squid'
'--enable-removal-policies=heap' '--disable-linux-netfilter'
'--disable-linux-tproxy' '--disable-epoll' '--disable-unlinkd'
'--with-coss-membuf-size=33554432' '--disable-wccpv2' '--with-pthreads'
'--enable-storeio=aufs coss' '--enable-delay-pools' '--enable-snmp'
'--disable-carp' '--disable-wccp' '--disable-ident-lookups'
'--enable-arp-acl' '--enable-pf-transparent' '--with-large-files'
'--enable-large-cache-files' '--enable-err-languages=English'
'--enable-default-err-language=English' '--prefix=/usr/local'
'--mandir=/usr/local/man' '--infodir=/usr/local/info/'
'--build=i386-portbld-freebsd7.2' 'build_alias=i386-portbld-freebsd7.2'
'CC=cc' 'CFLAGS=-O2 -pipe  -fno-strict-aliasing' 'LDFLAGS=' 'CPPFLAGS='

with OS freebsd 7.2-STABLE

Please provide any additional information below.

good cache hit  dynamic content (youtube, google map,etc)  with
store_rewrite_list (storeurl.pl) but cache miss after restart server

Original issue reported on code.google.com by tembokg...@gmail.com on 9 Apr 2010 at 10:25

GoogleCodeExporter commented 9 years ago 
Ok. I've had a few people report this.

What I need to fix this:

* Your config file, minus comments
* A specific set of example commands to reproduce the problem - eg, fetch this, 
fetch
this - should be a hit, fetch this - should be a hit, restart, fetch this - 
it's in
cache but won't be a hit.

Please do this and I'll be able to sort it out!

Original comment by adrian.c...@gmail.com on 9 Apr 2010 at 11:18

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 

##start of config
http_port 10.10.10.1:3128 transparent
http_port 127.0.0.1:3128
icp_port 3130
server_http11 on

pid_filename /usr/local/squid/logs/squid.pid

cache_effective_user proxy
cache_effective_group proxy

error_directory /usr/local/etc/squid/errors/English
icon_directory /usr/local/etc/squid/icons
visible_hostname lusca.net
cache_mgr kweteng@localnet
access_log /var/log/squid/access.log
cache_log /var/log/squid/cache.log
cache_store_log none
log_http_violations off
logfile_rotate 1
shutdown_lifetime 10 seconds
forwarded_for off

acl localnet src 10.10.10.0/27
acl localnet src 11.11.11.0/25

uri_whitespace strip
dns_nameservers 10.10.10.1
dns_testnames 127.0.0.1
cache_mem 256 MB
maximum_object_size_in_memory 64 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA

cache_dir aufs /cache1 32768 64 256
cache_dir aufs /cache2 32768 64 256
cache_dir aufs /cache3 32768 64 256

minimum_object_size 512 bytes
maximum_object_size 256000 KB
offline_mode off
cache_swap_low 98
cache_swap_high 99

# redirector configured

storeurl_rewrite_program /usr/local/etc/squid/storeurl.pl
storeurl_rewrite_children 1
storeurl_rewrite_concurrency 100
storeurl_bypass on 

# Setup some default acls
acl all src 0.0.0.0/0
acl localhost src 127.0.0.1/255.255.255.255
acl safeports port 21 70 80 210 280 443 488 563 591 631 667 777 901 81 3128 
1025-65535
acl sslports port 443 563 81
acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECT
#acl dynamic urlpath_regex cgi-bin \?

http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

# Always allow localhost connections
http_access allow localhost

# Allow local network(s) on interface(s)
http_access allow localnet

# Default block all to be sure
http_access deny all

acl store_rewrite_list urlpath_regex
\/(get_video|videoplayback\?id|videoplayback.*id)
\.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|wmv|3gp|mp(4|3)|exe|msi|zip|on2|mar|swf
)\?
acl store_rewrite_list url_regex -i \.youtube\.com\/get_video\?
acl store_rewrite_list url_regex -i \.youtube\.com\/videoplayback
\.youtube\.com\/videoplay \.youtube\.com\/get_video\?
acl store_rewrite_list url_regex -i \.youtube\.[a-z][a-z]\/videoplayback
\.youtube\.[a-z][a-z]\/videoplay \.youtube\.[a-z][a-z]\/get_video\?
acl store_rewrite_list url_regex -i \.googlevideo\.com\/videoplayback
\.googlevideo\.com\/videoplay \.googlevideo\.com\/get_video\?
acl store_rewrite_list url_regex -i \.google\.com\/videoplayback
\.google\.com\/videoplay \.google\.com\/get_video\? 
acl store_rewrite_list_domain url_regex
^http:\/\/([a-zA-Z-]+[0-9-]+)\.[A-Za-z]*\.[A-Za-z]*
acl store_rewrite_list_domain url_regex
(([a-z]{1,2}[0-9]{1,3})|([0-9]{1,3}[a-z]{1,2}))\.[a-z]*[0-9]?\.[a-z]{3}
acl store_rewrite_list_path urlpath_regex
\.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|avc|zip|mp3|3gp|rar|on2|mar|exe)$
acl store_rewrite_list_domain_CDN url_regex 
\.rapidshare\.com.*\/[0-9]*\/.*\/[^\/]*
^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*) \.doubleclick\.net.*
^http:\/\/[.a-z0-9]*\.photobucket\.com.*\.[a-z]{3}$ quantserve\.com

acl dontrewrite url_regex redbot\.org \.php
(get_video|videoplayback\?id|videoplayback.*id).*begin\=
acl getmethod method GET

acl apache rep_header Server ^Apache
broken_vary_encoding allow apache

storeurl_access deny dontrewrite
storeurl_access deny !getmethod
storeurl_access allow store_rewrite_list_domain_CDN
storeurl_access allow store_rewrite_list
storeurl_access allow store_rewrite_list_domain
storeurl_access allow store_rewrite_list_path
storeurl_access deny all

acl snmp_host src 127.0.0.1
snmp_port 3401
acl snmppublic snmp_community public
snmp_access allow snmppublic snmp_host
snmp_access deny all

# 1 year = 525600 mins, 1 month = 43800 mins
refresh_pattern imeem.*\.flv        0 0% 0  override-lastmod override-expire
refresh_pattern ^ftp:   40320   20% 40320   override-expire reload-into-ims 
store-stale
refresh_pattern ^gopher: 1440   0%  1440 

#ads
refresh_pattern
^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\
.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com
|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pi
xel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.c
om|syndication\.com|media.fastclick.net).*
129600 20% 129600 ignore-no-cache ignore-no-store ignore-private override-expire
ignore-reload ignore-auth ignore-must-revalidate store-stale negative-ttl=40320
max-stale=1440
#specific sites
refresh_pattern \.rapidshare.*\/[0-9]*\/.*\/[^\/]* 161280   90% 161280 
ignore-reload
store-stale
refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?)    129600
99999999% 129600 override-expire ignore-reload store-stale
refresh_pattern \.(ico|video-stats) 129600 999999% 129600   override-expire
ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
override-lastmod ignore-must-revalidate negative-ttl=10080 store-stale
refresh_pattern \.etology\?         129600 999999% 129600   override-expire 
ignore-reload
ignore-no-cache store-stale
refresh_pattern galleries\.video(\?|sz)         129600 999999% 129600   override-expire
ignore-reload ignore-no-cache store-stale
refresh_pattern brazzers\?          129600 999999% 129600   override-expire 
ignore-reload
ignore-no-cache store-stale
refresh_pattern \.adtology\?        129600 999999% 129600   override-expire 
ignore-reload
ignore-no-cache store-stale
refresh_pattern ^.*safebrowsing.*google         129600 999999% 129600   override-expire
ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
ignore-must-revalidate negative-ttl=10080 store-stale
refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk)    129600
999999% 129600  override-expire ignore-reload store-stale ignore-private
negative-ttl=10080
refresh_pattern ytimg\.com.*\.(jpg|png) 129600 999999% 129600   override-expire
ignore-reload store-stale
refresh_pattern (avgate|avira).*\.(idx|gz)$     1440 90% 1440 ignore-reload
ignore-no-cache ignore-no-store store-stale ignore-must-revalidate 
refresh_pattern images\.friendster\.com.*\.(png|gif)        129600 999999% 129600
override-expire ignore-reload store-stale
refresh_pattern ((facebook.com)|(85.131.151.39)).*\.(png|gif) 129600 999999% 
129600
override-expire ignore-reload store-stale
refresh_pattern garena\.com 129600 999999% 129600   override-expire 
reload-into-ims
store-stale
refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 129600 999999% 
129600
override-expire ignore-reload store-stale
refresh_pattern vid\.akm\.dailymotion\.com.*\.on2\?         129600 999999% 129600
ignore-no-cache override-expire override-lastmod store-stale
refresh_pattern .fbcdn.net.*\.(jpg|gif|png)     129600 999999% 
129600  ignore-no-cache
override-expire ignore-reload store-stale negative-ttl=5
refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\.         129600 999999% 129600
ignore-no-cache ignore-no-store ignore-reload override-expire store-stale
refresh_pattern ^http:\/\/www.onemanga.com.*\/  129600 999999% 
129600  reload-into-ims
override-expire store-stale
refresh_pattern mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 129600
999999% 129600  reload-into-ims override-expire ignore-private store-stale

# ANTI VIRUS
refresh_pattern guru.avg.com/.*\.(bin)                  43200 999999% 43200 ignore-no-cache
ignore-no-store ignore-reload  reload-into-ims store-stale
refresh_pattern (avgate|avira).*(idx|gz)$                           43200 999999% 
43200
ignore-no-cache ignore-no-store ignore-reload  reload-into-ims store-stale
refresh_pattern update.nai.com/.*\.(gem|zip|mcs)                    43200 999999% 
43200
ignore-no-cache ignore-no-store ignore-reload  reload-into-ims store-stale
refresh_pattern ^http:\/\/liveupdate.symantecliveupdate.com.*\(zip)     43200 
999999%
43200 ignore-no-cache ignore-no-store ignore-reload  reload-into-ims store-stale
refresh_pattern kaspersky.*\.avc$       129600 999999% 129600   ignore-reload 
store-stale
refresh_pattern kaspersky 1440 50% 161280 ignore-no-cache store-stale

refresh_pattern windowsupdate.com/.*\.(cab|exe)                 43200  999999%  129600
ignore-no-cache ignore-no-store ignore-reload  reload-into-ims store-stale
refresh_pattern update.microsoft.com/.*\.(cab|exe)          43200  999999%  129600
ignore-no-cache ignore-no-store ignore-reload  reload-into-ims store-stale
refresh_pattern download.microsoft.com/.*\.(cab|exe)            43200  999999%  129600
ignore-no-cache ignore-no-store ignore-reload  reload-into-ims store-stale

#images facebook
refresh_pattern ((facebook.com)|(85.131.151.39)).*\.(jpg|png|gif)   129600 
999999%
129600 ignore-reload  override-expire ignore-no-cache ignore-no-store 
store-stale
#refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3)                129600 
999999%
129600 ignore-reload  override-expire ignore-no-cache ignore-no-store 
store-stale
refresh_pattern .fbcdn.net.*\.(jpg|gif|png|swf|mp3)                 40320   20%     
40320 
ignore-reload store-stale negative-ttl=5
refresh_pattern  static\.ak\.fbcdn\.net*\.(jpg|gif|png)                 129600 
999999%
129600 ignore-reload  override-expire ignore-no-cache ignore-no-store 
store-stale
refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png|flv|avi)  
    129600
999999% 129600 ignore-reload  override-expire ignore-no-cache ignore-no-store 
store-stale
refresh_pattern ^http:\/\/\.*\.zynga\.com\/(.*)             129600 999999% 129600
reload-into-ims ignore-no-cache ignore-no-store ignore-reload override-expire 
store-stale

#general
refresh_pattern \.(jp(e?g|e|2)|tiff?|bmp|gif|png) 129600 999999% 129600
ignore-no-cache ignore-no-store reload-into-ims override-expire store-stale
refresh_pattern \.(z(ip|[0-9]{2})|r(ar|[0-9]{2})|jar|bz2|gz|tar|rpm|vpu) 129600
999999% 129600  override-expire reload-into-ims
refresh_pattern \.(mp3|wav|og(g|a)|flac|midi?|rm|aac|wma|mka|ape)       129600 
999999%
129600  override-expire reload-into-ims ignore-reload
refresh_pattern \.(exe|msi|dmg|bin|xpi|iso|swf|mar|psf|cab) 129600 999999% 
129600
override-expire reload-into-ims ignore-no-cache
refresh_pattern
\.(mpeg|ra?m|avi|mp(g|e|4)|mov|divx|asf|wmv|m\dv|rv|vob|asx|ogm|flv|3gp|on2|swf)
129600 9999999% 129600 override-expire reload-into-ims

refresh_pattern -i (cgi-bin)        0   0%  0
refresh_pattern \.(php|jsp|cgi|asx)\?   0   0%  0
refresh_pattern .           0   50%     161280 store-stale

zph_mode tos
zph_local 0x30
zph_parent 0
zph_option 136

global_internal_static off
max_stale 10 years
retry_on_error on
buffered_logs on
read_ahead_gap 32 KB

header_access Accept-Encoding deny  all
client_persistent_connections on
server_persistent_connections on 
half_closed_clients off
strip_query_terms off
quick_abort_min 0 KB
quick_abort_max 0 KB
quick_abort_pct 100
vary_ignore_expire on
reload_into_ims on
pipeline_prefetch on
#range_offset_limit 50 KB
read_timeout 30 minutes
client_lifetime 6 hours
negative_ttl 30 seconds
positive_dns_ttl 6 hours
negative_dns_ttl 60 seconds
pconn_timeout 15 seconds
request_timeout 1 minute
store_avg_object_size 13 KB
log_icp_queries off
ipcache_size 16384
ipcache_low 98
ipcache_high 99
log_fqdn off
fqdncache_size 2048
memory_pools on
memory_pools_limit 1024 MB
#forwarded_for on

#cachemgr_passwd none info
cachemgr_passwd none all
client_db on
n_aiops_threads 32
load_check_stopen on
load_check_stcreate on
download_fastest_client_speed on

########### end config squid.conf ############

###start config storeurl.pl#####

#!/usr/local/bin/perl
# $Rev$
# by chudy_fernandez@yahoo.com
# Updates at 
http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube/Discussion
$|=1;
while (<>) {
    @X = split;
#   $X[1] =~ s/&sig=.*//;
    $x = $X[0] . " ";
    $_ = $X[1];
    $u = $X[1];
    $url = $X[0];

if ($url =~ m@^http://([^\.]*\.)?bing\.[^\/]*/[^?]*\?.*@i) {

    # Replace any previous safe directives
    $url =~ s@(adlt=[^&]*&?)@@ig;

    # Add safe search directive
    $url .= '&adlt=strict&cc=au';

} elsif ($url =~ m@^http://([^\.]*\.)?google\.[^\/]*/[^?]*\?.*@i) {

    # Replace any previous safe directives
    $url =~ s@(safe=[^&]*&?)@@ig;

    # Add safe search directive
    $url .= '&safe=active';
print "$url\n";

            # compatibility for old cached get_video?video_id
} elsif
(m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\
.com).*?(videoplayback\?id=.*?|video_id=.*?)\&(.*?)/)
{
    $z = $2; $z =~ s/video_id=/get_video?video_id=/;
    print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $z . "\n";

            # youtube HD itag=22
} elsif
(m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\
.com).*?\&(itag=22).*?\&(id=[a-zA-Z0-9]*)/)
{
    print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $2 . "&" . $3 . "\n";

            # youtube Normal screen always HD itag 35, Normal screen never HD itag 34, itag=18
<--normal?
} elsif
(m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\
.com).*?\&(itag=[0-9]*).*?\&(id=[a-zA-Z0-9]*)/)
{
    print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $3 . "\n";

} elsif (m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) {
    print $x . "http://www.google-analytics.com/__utm.gif\n";

            #Cache High Latency Ads
} elsif
(m/^http:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|\.googlesyndicati
on\.com|yieldmanager|cpxinteractive)(.*)/)
{
    $y = $3;$z = $2;
    for ($y) {
    s/pixel;.*/pixel/;
    s/activity;.*/activity/;
    s/(imgad[^&]*).*/\1/;
    s/;ord=[?0-9]*//;
    s/;×tamp=[0-9]*//;
    s/[&?]correlator=[0-9]*//;
    s/&cookie=[^&]*//;
    s/&ga_hid=[^&]*//;
    s/&ga_vid=[^&]*//;
    s/&ga_sid=[^&]*//;
    # s/&prev_slotnames=[^&]*//
    # s/&u_his=[^&]*//;
    s/&dt=[^&]*//;
    s/&dtd=[^&]*//;
    s/&lmt=[^&]*//;
    s/(&alternate_ad_url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
    s/(&url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
    s/(&ref=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
    s/(&cookie=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
    s/[;&?]ord=[?0-9]*//;
    s/[;&]mpvid=[^&;]*//;
    s/&xpc=[^&]*//;
    # yieldmanager
    s/\?clickTag=[^&]*//;
    s/&u=[^&]*//;
    s/&slotname=[^&]*//;
    s/&page_slots=[^&]*//;
    }
    print $x . "http://" . $1 . $2 . $y . "\n";

            #cache high latency ads
} elsif (m/^http:\/\/(.*?)\/(ads)\?(.*?)/) {
    print $x . "http://" . $1 . "/" . $2  . "\n";

            # spicific servers starts here....
} elsif (m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {
    print $x . "http://" . $1 . "\n";

            #cdn, varialble 1st path
} elsif (($u =~ /filehippo/) &&
(m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
    @y = ($1,$2,$4,$5);
    $y[0] =~ s/[a-z0-9]{2,5}/cdn./;
    print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

            #rapidshare
} elsif (($u =~ /rapidshare/) &&
(m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/(
[^\/\?\&]{4,})$/))
{
    print $x . "http://cdn." . $3 . "/SQUIDINTERNAL/" . $5 . "\n";

} elsif (($u =~ /maxporn/) && 
(m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) {
    print $x . "http://" . $1 . "/SQUIDINTERNAL/" . $3 . "\n";

            #like porn hub variables url and center part of the path, filename etention 3 or 4
with or without ? at the end
} elsif (($u =~ /tube8|pornhub|xvideos/) &&
(m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[
a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/))
{
    print $x . "http://cdn." . $3 . $5 . "\n";
            #...spicific servers end here.

            #photos-X.ak.fbcdn.net where X a-z
} elsif (m/^http:\/\/photos-[a-z].ak.fbcdn.net\/(.*)/) {
    print $x . "http://photos.ak.fbcdn.net/" . $1  . "\n";

            #general purpose for cdn servers. add above your specific servers.
} elsif (m/^http:\/\/([0-9.]*?)\/\/(.*?)\.(.*)\?(.*?)/) {
    print $x . "http://squid-cdn-url//" . $2  . "." . $3 . "\n";

            #for yimg.com doubled
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) {
    print $x . "http://cdn.yimg.com/"  . $3 . "\n";

            #for yimg.com with &sig=
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*)/) {
    @y = ($1,$2);
    $y[0] =~ s/[a-z]+[0-9]+/cdn/;
    $y[1] =~ s/&sig=.*//;
    print $x . "http://" . $y[0] . ".yimg.com/"  . $y[1] . "\n";

            #generic http://variable.domain.com/path/filename."ext" or "exte" with or withour "?"
} elsif (m/^http:\/\/(.*)([^\.\-]*?\..*?)\/(.*)\.([^\/\?\&]{3,4})(\?.*)?$/) {
    @y = ($1,$2,$3,$4);
    $y[0] =~ s/(([a-zA-A-]+[0-9-]+)|(.*cdn.*)|(.*cache.*))/cdn/;
    print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

            # generic http://variable.domain.com/...
} elsif 
(m/^http:\/\/(([A-Za-z]+[0-9-]+)*?|.*cdn.*|.*cache.*)\.(.*?)\.(.*?)\/(.*)$/) {
    print $x . "http://cdn." . $3 . "." . $4 . "/" . $5 .  "\n";

            # spicific extention that ends with ?
} elsif 
(m/^http:\/\/(.*?)\/(.*?)\.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|on2)\?(.*)/) {
    print $x . "http://" . $1 . "/" . $2  . "." . $3 . "\n";

            # all that ends with ;
} elsif (m/^http:\/\/(.*?)\/(.*?)\;(.*)/) {
    print $x . "http://" . $1 . "/" . $2  . "\n";

} else {
    print $x . $_ . "\n";
}
}

###end config storeurl.pl#####

Now we try with the above config
* browse youtube and other site (the first always miss)
* after a few minutes, clear cache browser, try again with the same content 
(always
hits) 
* reboot server, and try again with the same content (always miss)

whether this is a bug ???, so be poisoning the cache with dynamic content after
reboot server ???

Original comment by tembokg...@gmail.com on 9 Apr 2010 at 4:14

GoogleCodeExporter commented 9 years ago 
Howdy!

Can you please provide example urls I can feed into Lusca via wget, 
squidclient, etc?

Specifically - what I may see when I go to Youtube will be different to what 
you may
see. So I do really need exact urls I can directly feed to Lusca rather than via
browsing a website.

Original comment by adrian.c...@gmail.com on 9 Apr 2010 at 5:34

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
sir, 
never mind I found the bug !!

It's that FireFox with download helper addon I disabled the addon and it's 
working !

Thanks a lot !! 

theris example my log :

# url browser
http://www.youtube.com/watch?v=bimECz25r08&feature=related

# access.log
*firts time
1270844264.311   3256 10.10.10.120  TCP_MISS/200 78760 GET
http://www.youtube.com/watch?v=bimECz25r08&feature=related - 
DIRECT/74.125.71.113
text/html
1270843594.981   80760 10.10.10.120 TCP_MISS/200 1679507 GET
http://v7.lscache5.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2C
ip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor%2Coc%3AU0dWRVhRVl9FSkNNNl9QS1pF&
fexp=904518%2C906901&algorithm=throttle-factor&itag=34&ipbits=0&burst=40&sver=3&
expire=1270868400&key=yt1&signature=1B10F43CD26F27A6ED1EEB75E261779644755A7D.063
DEB00B8888B95E64FB0E9EF0C805240A79C37&factor=1.25&id=6e29840b3db9af4f&
- DIRECT/208.117.252.34 video/x-flv

*after a few minutes (with clear cache browser)
1270843938.439  67815 10.10.10.120 TCP_HIT/200 1679516 GET
http://v7.lscache5.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2C
ip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor%2Coc%3AU0dWRVhRVl9FSkNNNl9QS1pF&
fexp=904518%2C906901&algorithm=throttle-factor&itag=34&ipbits=0&burst=40&sver=3&
expire=1270868400&key=yt1&signature=1B10F43CD26F27A6ED1EEB75E261779644755A7D.063
DEB00B8888B95E64FB0E9EF0C805240A79C37&factor=1.25&id=6e29840b3db9af4f&
- NONE/- video/x-flv

*after reboot with download helper addon firefox (with clear cache browser)

1270843938.439  67815 10.10.10.120 TCP_MISS/200 1679516 GET
http://v7.lscache5.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2C
ip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor%2Coc%3AU0dWRVhRVl9FSkNNNl9QS1pF&
fexp=904518%2C906901&algorithm=throttle-factor&itag=34&ipbits=0&burst=40&sver=3&
expire=1270868400&key=yt1&signature=1B10F43CD26F27A6ED1EEB75E261779644755A7D.063
DEB00B8888B95E64FB0E9EF0C805240A79C37&factor=1.25&id=6e29840b3db9af4f&
-  DIRECT/208.117.252.34 video/x-flv

*after reboot and disable download helper addon firefox (with clear cache 
browser)

1270844815.092  67757 0.10.10.120 TCP_HIT/200 1679517 GET
http://v7.lscache5.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2C
ip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor%2Coc%3AU0dWRVhRVl9FSkNNNl9QS1pF&
fexp=904518%2C906901&algorithm=throttle-factor&itag=34&ipbits=0&burst=40&sver=3&
expire=1270868400&key=yt1&signature=1B10F43CD26F27A6ED1EEB75E261779644755A7D.063
DEB00B8888B95E64FB0E9EF0C805240A79C37&factor=1.25&id=6e29840b3db9af4f&
- NONE/- video/x-flv

Original comment by tembokg...@gmail.com on 9 Apr 2010 at 8:35