search

dyne / dowse

The Awareness Hub for the Internet of Things
http://dowse.eu
159 stars 17 forks source link

squid-httpseverywhere module not working #1

Open jaromil opened 10 years ago

jaromil commented 10 years ago

The new module can be enabled with a symlink in modules/enabled

libxml-libxml-perl must be installed and update-rules run in the same dir.

Then running Dowse start it will generate a perl file with all paths correct to the installation. Squid3 will be launched, but then quit. Here the log:

2014/01/20 21:23:06| WARNING: redirector #1 (FD 9) exited
2014/01/20 21:23:06| WARNING: redirector #2 (FD 11) exited
2014/01/20 21:23:06| WARNING: redirector #3 (FD 13) exited
2014/01/20 21:23:06| WARNING: redirector #4 (FD 15) exited
2014/01/20 21:23:06| Too few redirector processes are running
2014/01/20 21:23:06| storeDirWriteCleanLogs: Starting...
2014/01/20 21:23:06|   Finished.  Wrote 0 entries.
2014/01/20 21:23:06|   Took 0.00 seconds (  0.00 entries/sec).
FATAL: The redirector helpers are crashing too rapidly, need help!

Squid Cache (Version 3.1.20): Terminated abnormally.
CPU Usage: 0.056 seconds = 0.008 user + 0.048 sys
Maximum Resident Size: 28432 KB
Page faults with physical i/o: 0
Memory usage for squid via mallinfo():
        total space in arena:    2248 KB
        Ordinary blocks:         2186 KB     20 blks
        Small blocks:               0 KB      0 blks
        Holding blocks:          2772 KB      5 blks
        Free Small blocks:          0 KB
        Free Ordinary blocks:      61 KB
        Total in use:            4958 KB 221%
        Total free:                61 KB 3%

The generated perl module for squid is found in $DOWSE/run/squid-httpseverywhere.pm

I'm not sure how to run it standalone to have tests, or check its compatibility with Squid3. Needing some help.

antagonismorg commented 10 years ago

What is your redirector_children set to? Try increasing the limit to see if it resolves the issue.

antagonismorg commented 10 years ago

In 2.x, I set url_rewrite_children 40 to resolve the issue.

jaromil commented 10 years ago

It doesn't solves here (stock Debian 7 components, including squid 2)

FATAL: The redirector helpers are crashing too rapidly, need help!

seems the helpers are too few because they crash on launch.

I'm not sure how to debug deeper their problem by running the perl script in a scaffolded way, without squid.

antagonismorg commented 10 years ago

This thread may be helpful. http://marc.info/?l=squid-users&m=107185287327641&w=2

To test the perl script, simply su to the same user as the proxy and execute perl /path/to/script/squid.pl. This will let you know if there are any issues with executing the script.

jaromil commented 10 years ago

did try that, script starts fine and "runs", basically doing nothing. Not sure how I can feed it urls..

will try squid -NDd1 now

antagonismorg commented 10 years ago

Any luck with running squid in debug?

jaromil commented 10 years ago

made some progresses in today's last commit. (there were some dowse related trivial permission problems to fix)

scaffolding debug works by piping urls into the generater perl handler, which takes some time to startup obviously, but then spits out a proper result.

lowering the children spawned it runs squid, but still when started from squid I cannot see results. started from commandline works.

I believe today's commits have removed all dowse related problems. are you running a working setup of this? which version of squid is used then?

antagonismorg commented 10 years ago

I am running, 2.7.STABLE9-4.1 for packages on Squeeze. I have the latest https-everywhere and perl-HTTPSEverywhere git repos.

jaromil commented 9 years ago

regarding this issue, I have a solution proposal that would benefit more than one software: can we make that perl script input the patterns into a key/value store like redis? if yes, the thing can really fly. it would fill it at start and then even more than an app can query it.

jaromil commented 9 years ago

hi P. would you be interested in revamping this discussion? I see EFF still regards https-everywhere as a priority. I'm quite sure it wouldn't take much to refactor the perl parser in order to have it input values into a fast k/v storage like Redis, something that would make the mechanism much faster. I'm willing to work on this for Dowse and it would be much more fun to do it together with you.

antagonismorg commented 9 years ago

I would be delighted to work with you on this. We might also want to include Mike Cardwell the original author of perl-https-everywhere as well. Drop me an email at marlowe@antagonism.org and we can get started.

parazyd commented 6 years ago

@jaromil Shall we close this?