openaustralia / yinyo

A wonderfully simple API driven service to reliably execute many long running scrapers in a super scaleable way
https://yinyo.io
Apache License 2.0
6 stars 1 forks source link

Note that localhost:8080 is required. #61

Closed jamezpolley closed 4 years ago

jamezpolley commented 4 years ago

Expected behaviour:

Following the README, running client.sh should connect to the API server on localhost:8080.

Observed behaviour: If there's already something listening on localhost:8080, no error messages are given, but requests will go to that service.

mlandauer commented 4 years ago

I think it would be better from a user-experience point of view if things errored if there was already something running on port 8080. I think it's a bug that it doesn't error currently. I think we should only resort to adding a note in the documentation if we can't find a way to make the thing error sensibly.

jamezpolley commented 4 years ago

In my case, the thing listing on 8080 was Traefik, running inside Docker Swarm. I'm not sure what could be done to detect this. In summary:

Info below is just my notes from my current setup.

dockerd only seems to bind to tcp6:

tcp6       4      0 :::8080                 :::*                    LISTEN      1658/dockerd

The kubectl process binds to both tcp and tcp6 sockets, but on a different address to docker:

1d [james@bully:~/src/oaf/morph-ng/morph-ng] master(+2/-2)* ± sudo netstat -ntlp | grep 8080
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      21946/kubectl
tcp6       0      0 ::1:8080                :::*                    LISTEN      21946/kubectl

Docker also sets up iptables rules to catch inbound traffic and redirect it:

1d [james@bully:~/src/oaf/morph-ng/morph-ng] master(+2/-2)* ± sudo iptables-save | grep 8080
-A DOCKER-INGRESS -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 8080 -j ACCEPT
-A DOCKER-INGRESS -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.18.0.2:8080

curl output:

1d [james@bully:~/src/oaf/morph-ng/morph-ng] master(+2/-2)* ± curl -s -G -X POST http://localhost:8080/runs -d name_prefix=test/scrapers/test-python -vvv
*   Trying ::1...
* TCP_NODELAY set
* Expire in 150000 ms for 3 (transfer 0x56189bdfd5c0)
* Expire in 200 ms for 4 (transfer 0x56189bdfd5c0)
* Connected to localhost (::1) port 8080 (#0)
> POST /runs?name_prefix=test/scrapers/test-python HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.0
> Accept: */*
>
mlandauer commented 4 years ago

That all sounds a bit painful @jamezpolley. Probably best to merge your PR then!