search

SpiderClub / haipproxy

:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
https://spiderclub.github.io/haipproxy/
MIT License
5.44k stars 912 forks source link

Docker. Squid is already running #19

Closed rivmar closed 6 years ago

rivmar commented 6 years ago

Hi. I am trying to use haipproxy with docker:

~/$ docker version
Client:
 Version:   18.02.0-ce
 API version:   1.36
 Go version:    go1.9.3
 Git commit:    fc4de44
 Built: Wed Feb  7 21:16:33 2018
 OS/Arch:   linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:  18.02.0-ce
  API version:  1.36 (minimum version 1.12)
  Go version:   go1.9.3
  Git commit:   fc4de44
  Built:    Wed Feb  7 21:15:05 2018
  OS/Arch:  linux/amd64
  Experimental: false

on Ubuntu 16.04.3 LTS

And I am constantly getting error:

haipproxy_1 | 2018/03/12 14:46:39| Squid is already running! Process ID 12

I added to docker-compose:

    tty: true
    restart: always

but without too much success:

haipproxy_1 | 2018/03/12 14:53:01| Set Current Directory to /var/spool/squid haipproxy_1 | 2018/03/12 14:53:01| Starting Squid Cache version 3.5.12 for x86_64-pc-linux-gnu... haipproxy_1 | 2018/03/12 14:53:01| Service Name: squid haipproxy_1 | 2018/03/12 14:53:01| Process ID 12 haipproxy_1 | 2018/03/12 14:53:01| Process Roles: master worker haipproxy_1 | 2018/03/12 14:53:01| With 1048576 file descriptors available haipproxy_1 | 2018/03/12 14:53:01| Initializing IP Cache... haipproxy_1 | 2018/03/12 14:53:01| DNS Socket created at [::], FD 8 haipproxy_1 | 2018/03/12 14:53:01| DNS Socket created at 0.0.0.0, FD 9 haipproxy_1 | 2018/03/12 14:53:01| Adding nameserver 127.0.0.11 from /etc/resolv.conf haipproxy_1 | 2018/03/12 14:53:01| Adding ndots 1 from /etc/resolv.conf haipproxy_1 | 2018/03/12 14:53:01| Logfile: opening log daemon:/var/log/squid/access.log haipproxy_1 | 2018/03/12 14:53:01| Logfile Daemon: opening log /var/log/squid/access.log haipproxy_1 | 2018/03/12 14:53:01| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec haipproxy_1 | 2018/03/12 14:53:01| Store logging disabled haipproxy_1 | 2018/03/12 14:53:01| Swap maxSize 0 + 262144 KB, estimated 20164 objects haipproxy_1 | 2018/03/12 14:53:01| Target number of buckets: 1008 haipproxy_1 | 2018/03/12 14:53:01| Using 8192 Store buckets haipproxy_1 | 2018/03/12 14:53:01| Max Mem size: 262144 KB haipproxy_1 | 2018/03/12 14:53:01| Max Swap size: 0 KB haipproxy_1 | 2018/03/12 14:53:01| Using Least Load store dir selection haipproxy_1 | 2018/03/12 14:53:01| Set Current Directory to /var/spool/squid haipproxy_1 | 2018/03/12 14:53:01| Finished loading MIME types and icons. haipproxy_1 | 2018/03/12 14:53:01| HTCP Disabled. haipproxy_1 | 2018/03/12 14:53:01| commBind: Cannot bind socket FD 13 to [::1]: (99) Cannot assign requested address haipproxy_1 | 2018/03/12 14:53:01| commBind: Cannot bind socket FD 14 to [::1]: (99) Cannot assign requested address haipproxy_1 | 2018/03/12 14:53:01| ERROR: Failed to create helper child read FD: UDP[::1] haipproxy_1 | 2018/03/12 14:53:01| Squid plugin modules loaded: 0 haipproxy_1 | 2018/03/12 14:53:01| Adaptation support is off. haipproxy_1 | 2018/03/12 14:53:01| Accepting HTTP Socket connections at local=[::]:3128 remote=[::] FD 12 flags=9 haipproxy_1 | 2018/03/12 14:53:02| storeLateRelease: released 0 objects haipproxy_1 | 2018/03/12 14:53:04| Reconfiguring Squid Cache (version 3.5.12)... haipproxy_1 | 2018/03/12 14:53:04| Closing HTTP port [::]:3128 haipproxy_1 | 2018/03/12 14:53:04| Logfile: closing log daemon:/var/log/squid/access.log haipproxy_1 | 2018/03/12 14:53:04| Logfile Daemon: closing log daemon:/var/log/squid/access.log haipproxy_1 | 2018/03/12 14:53:04| Startup: Initializing Authentication Schemes ... haipproxy_1 | 2018/03/12 14:53:04| Startup: Initialized Authentication Scheme 'basic' haipproxy_1 | 2018/03/12 14:53:04| Startup: Initialized Authentication Scheme 'digest' haipproxy_1 | 2018/03/12 14:53:04| Startup: Initialized Authentication Scheme 'negotiate' haipproxy_1 | 2018/03/12 14:53:04| Startup: Initialized Authentication Scheme 'ntlm' haipproxy_1 | 2018/03/12 14:53:04| Startup: Initialized Authentication. haipproxy_1 | 2018/03/12 14:53:04| Processing Configuration File: /etc/squid/squid.conf (depth 0) haipproxy_1 | 2018/03/12 14:53:04| Logfile: opening log daemon:/var/log/squid/access.log haipproxy_1 | 2018/03/12 14:53:04| Logfile Daemon: opening log /var/log/squid/access.log haipproxy_1 | 2018/03/12 14:53:04| Squid plugin modules loaded: 0 haipproxy_1 | 2018/03/12 14:53:04| Adaptation support is off. haipproxy_1 | 2018/03/12 14:53:04| Store logging disabled haipproxy_1 | 2018/03/12 14:53:04| DNS Socket created at [::], FD 10 haipproxy_1 | 2018/03/12 14:53:04| DNS Socket created at 0.0.0.0, FD 11 haipproxy_1 | 2018/03/12 14:53:04| Adding nameserver 127.0.0.11 from /etc/resolv.conf haipproxy_1 | 2018/03/12 14:53:04| Adding ndots 1 from /etc/resolv.conf haipproxy_1 | 2018/03/12 14:53:04| HTCP Disabled. haipproxy_1 | 2018/03/12 14:53:04| commBind: Cannot bind socket FD 13 to [::1]: (99) Cannot assign requested address haipproxy_1 | 2018/03/12 14:53:04| commBind: Cannot bind socket FD 14 to [::1]: (99) Cannot assign requested address haipproxy_1 | 2018/03/12 14:53:04| ERROR: Failed to create helper child read FD: UDP[::1] haipproxy_1 | 2018/03/12 14:53:04| Finished loading MIME types and icons. haipproxy_1 | 2018/03/12 14:53:04| Accepting HTTP Socket connections at local=[::]:3128 remote=[::] FD 12 flags=9 haipproxy_1 | 2018/03/12 14:56:04| Reconfiguring Squid Cache (version 3.5.12)... haipproxy_1 | 2018/03/12 14:56:04| Closing HTTP port [::]:3128 haipproxy_1 | 2018/03/12 14:56:04| Logfile: closing log daemon:/var/log/squid/access.log haipproxy_1 | 2018/03/12 14:56:04| Logfile Daemon: closing log daemon:/var/log/squid/access.log haipproxy_1 | 2018/03/12 14:56:04| Startup: Initializing Authentication Schemes ... haipproxy_1 | 2018/03/12 14:56:04| Startup: Initialized Authentication Scheme 'basic' haipproxy_1 | 2018/03/12 14:56:04| Startup: Initialized Authentication Scheme 'digest' haipproxy_1 | 2018/03/12 14:56:04| Startup: Initialized Authentication Scheme 'negotiate' haipproxy_1 | 2018/03/12 14:56:04| Startup: Initialized Authentication Scheme 'ntlm' haipproxy_1 | 2018/03/12 14:56:04| Startup: Initialized Authentication. haipproxy_1 | 2018/03/12 14:56:04| Processing Configuration File: /etc/squid/squid.conf (depth 0) haipproxy_1 | 2018/03/12 14:56:05| Logfile: opening log daemon:/var/log/squid/access.log haipproxy_1 | 2018/03/12 14:56:05| Logfile Daemon: opening log /var/log/squid/access.log haipproxy_1 | 2018/03/12 14:56:05| Squid plugin modules loaded: 0 haipproxy_1 | 2018/03/12 14:56:05| Adaptation support is off. haipproxy_1 | 2018/03/12 14:56:05| Store logging disabled haipproxy_1 | 2018/03/12 14:56:05| DNS Socket created at [::], FD 10 haipproxy_1 | 2018/03/12 14:56:05| DNS Socket created at 0.0.0.0, FD 11 haipproxy_1 | 2018/03/12 14:56:05| Adding nameserver 127.0.0.11 from /etc/resolv.conf haipproxy_1 | 2018/03/12 14:56:05| Adding ndots 1 from /etc/resolv.conf haipproxy_1 | 2018/03/12 14:56:05| HTCP Disabled. haipproxy_1 | 2018/03/12 14:56:05| commBind: Cannot bind socket FD 13 to [::1]: (99) Cannot assign requested address haipproxy_1 | 2018/03/12 14:56:05| commBind: Cannot bind socket FD 14 to [::1]: (99) Cannot assign requested address haipproxy_1 | 2018/03/12 14:56:05| ERROR: Failed to create helper child read FD: UDP[::1] haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 89.236.17.108/3128/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 61.220.26.97/80/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 159.89.195.153/8118/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 35.196.26.166/3128/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 34.231.147.235/3128/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 216.177.233.181/8080/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 103.74.246.124/65205/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 153.126.201.100/3128/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 121.129.127.209/80/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 109.105.40.39/53281/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 179.106.37.39/8080/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 144.76.176.72/8080/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 181.49.24.126/8080/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 191.210.166.52/8080/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 137.74.254.242/3128/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 192.116.142.153/8080/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 137.74.168.174/8080/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 117.6.161.118/53281/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 128.199.138.67/8118/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 186.227.8.21/3128/0 haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 82.196.12.94/3128/0 haipproxy_1 | 2018/03/12 14:56:05| Finished loading MIME types and icons. haipproxy_1 | 2018/03/12 14:56:05| Accepting HTTP Socket connections at local=[::]:3128 remote=[::] FD 12 flags=9 haipproxy_1 | 2018/03/12 14:56:31| netdbExchangeHandleReply: corrupt data, aborting haipproxy_1 | 2018/03/12 14:56:39| netdbExchangeHandleReply: corrupt data, aborting haipproxy_1 | 2018/03/12 15:02:03| Squid is already running! Process ID 12 haipproxy_1 | 2018/03/12 15:02:04| Squid is already running! Process ID 12 haipproxy_1 | 2018/03/12 15:02:05| Squid is already running! Process ID 12 haipproxy_1 | 2018/03/12 15:02:06| Squid is already running! Process ID 12 haipproxy_1 | 2018/03/12 15:02:07| Squid is already running! Process ID 12 haipproxy_1 | 2018/03/12 15:02:09| Squid is already running! Process ID 12 haipproxy_1 | 2018/03/12 15:02:13| Squid is already running! Process ID 12 haipproxy_1 | 2018/03/12 15:02:20| Squid is already running! Process ID 12 haipproxy_1 | 2018/03/12 15:02:33| Squid is already running! Process ID 12 haipproxy_haipproxy_1 exited with code 0

Is it normal? Or maybe I should change some other settings?

ResolveWang commented 6 years ago 
haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 191.210.166.52/8080/0
haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 137.74.254.242/3128/0
haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 192.116.142.153/8080/0
haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 137.74.168.174/8080/0
haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 117.6.161.118/53281/0
haipproxy_1 | 2018/03/12 14:56:05| Configuring Parent 128.199.138.67/8118/0

It seems that you have just sent requests with squid successfully.If you can send requests with squid, just leave the messages alone.They are logs of squid.

rivmar commented 6 years ago

The problem is that when error "Squid is already running! " appears I can not sent request anymore... Here is another log:

haipproxy_1 | 2018/03/13 07:27:58| Configuring Parent 181.30.101.242/3128/0 haipproxy_1 | 2018/03/13 07:27:58| Configuring Parent 118.178.227.171/80/0 haipproxy_1 | 2018/03/13 07:27:58| Configuring Parent 27.123.1.46/65103/0 haipproxy_1 | 2018/03/13 07:27:58| Configuring Parent 52.207.79.142/80/0 haipproxy_1 | 2018/03/13 07:27:58| Configuring Parent 190.24.131.250/3128/0 haipproxy_1 | 2018/03/13 07:27:58| Configuring Parent 185.119.56.8/53281/0 haipproxy_1 | 2018/03/13 07:27:58| Configuring Parent 137.74.254.242/3128/0 haipproxy_1 | 2018/03/13 07:27:58| Finished loading MIME types and icons. haipproxy_1 | 2018/03/13 07:27:58| Accepting HTTP Socket connections at local=[::]:3128 remote=[::] FD 16 flags=9 haipproxy_1 | 2018/03/13 07:27:58| Closing HTTP port [::]:3128 haipproxy_1 | 2018/03/13 07:27:58| storeDirWriteCleanLogs: Starting... haipproxy_1 | 2018/03/13 07:27:58| Finished. Wrote 0 entries. haipproxy_1 | 2018/03/13 07:27:58| Took 0.00 seconds ( 0.00 entries/sec). haipproxy_1 | Aborted (core dumped) haipproxy_1 | 2018/03/13 07:28:00| Squid is already running! Process ID 12 haipproxy_1 | 2018/03/13 07:28:01| Squid is already running! Process ID 12 haipproxy_1 | 2018/03/13 07:28:02| Squid is already running! Process ID 12

I have to make docker stop and docker rm to delete container and up it again to make it work.

ResolveWang commented 6 years ago

I'm sorry that i can't reproduce this problem, may be this answer is useful to you.You can try to deploy haipproxy without docker, and see whether this problem will happen

BTW, if you are familiar with python, you can use py_cli

teadrinker2015 commented 1 year ago

This information may be useful to you: I was encountered the same error message, because I set the port for Redis on host to 6380, due to conflict with another existing Redis on 6379. The haipproxy container does not immediately fail, instead had been running normally for several hours, and after a computer sleep-wake cycle, completely refuse to restart.

The problem docker-compose:

version: '2.0'
services:
  redis:
    image: redis
    ports:
      - '6380:6379' # I fixed 6380 to 6379 later
    command: >
      --requirepass 123456
  splash:
    image: scrapinghub/splash
    ports:
      - '8050:8050'
  haipproxy:
    build: .
    command: sh run.sh
    volumes:
      - .:/haipproxy
    ports:
      - '3128:3128'
    links:
      - redis
      - splash

I also modified the dockerfile: ( because the original one cannot build properly the time I was using )

FROM ubuntu:18.04

MAINTAINER ResolveWang <resolvewang@foxmail.com>

ENV LC_ALL C.UTF-8
ENV LANG C.UTF-8
RUN apt update
RUN apt install squid -yq
RUN sed -i 's/http_access deny all/http_access allow all/g' /etc/squid/squid.conf
RUN cp /etc/squid/squid.conf /etc/squid/squid.conf.backup
RUN apt install python3 python3-pip -yq
RUN which python3|xargs -i ln -s {} /usr/bin/python
RUN which pip3|xargs -i ln -s {} /usr/bin/pip
COPY . /haipproxy
WORKDIR /haipproxy
RUN python -m pip install --upgrade pip
RUN pip install -i https://pypi.douban.com/simple/ -r requirements.txt
CMD ['python', 'crawler_booter.py', '--usage', 'crawler', 'common']
teadrinker2015 commented 1 year ago

I was wrong, the haipproxy container still refuse to start due to Squid is already running after a host reboot (perhaps include a docker compose stop and start cycle). While it can continue longer than before.

And I tried the solution in answer:

may be this answer is useful to you.

The pid_file option seems to be the default in my current version of squid.conf. I will report after next reboot.