toddproject / todd

A distributed network assurance platform
Apache License 2.0
237 stars 30 forks source link

Nil address in agent advertisements #132

Closed Mierdin closed 7 years ago

Mierdin commented 7 years ago

(Relocated from https://github.com/toddproject/todd/issues/91, where @hanya1203 reported the below)

@Mierdin @slydetech Thank you for your advise. When I try to take logs I struck new error. And server didn't recognize agents Would you have any idea?

todd command log

[root@localhost todd]# docker exec toddserver todd agents No agents found.

Agent log

INFO[0520] AGENTADV -- 2017-03-10 07:21:46.560882068 +0000 UTC DEBU[0520] Agent task received: {"type":"DownloadAsset","assets":["http://\u003cnil\u003e:8090/factcollectors/get_hostname","http://\u003cnil\u003e:8090/factcollectors/get_addresses","http://\u003cnil\u003e:8090/testlets/http","http://\u003cnil\u003e:8090/testlets/iperf"]} INFO[0520] Downloading http://:8090/factcollectors/get_hostname to /opt/todd/agent/assets/factcollectors/get_hostname ERRO[0520] Error while downloading 'http://:8090/factcollectors/get_hostname': Get http://%3Cnil%3E:8090/factcollectors/get_hostname: dial tcp: lookup : invalid domain name ERRO[0520] Get http://%3Cnil%3E:8090/factcollectors/get_hostname: dial tcp: lookup : invalid domain name WARN[0520] The KeyValue task failed to initialize DEBU[0522] Retrieving value of key - unackedGroup DEBU[0524] Retrieving value of key - unackedGroup DEBU[0526] Retrieving value of key - unackedGroup DEBU[0528] Retrieving value of key - unackedGroup DEBU[0530] Retrieving value of key - unackedGroup DEBU[0530] Asset found locally: get_addresses (with hash e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) DEBU[0530] Asset found locally: get_hostname (with hash e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) DEBU[0530] Asset found locally: http (with hash e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) ERRO[0530] Problem running fact-gathering collector

Serverlog

INFO[0060] Beginning group calculation INFO[0060] Getting /todd/agents' key value WARN[0060] Agent list empty when queried INFO[0060] Accessing objects at/todd/objects/group/ WARN[0060] ToDD object store empty when queried DEBU[0060] Setting '/todd/groupmap' key INFO[0060] Updated group map in etcd: {} DEBU[0065] Loaded assets: map[factcollectors:map[get_addresses:630e6d795ee363e3741e0eba74350645c8ea1896bb1a4cb1c7f18c71fdee6462 get_hostname:e3dd6bd5888c29046bafcf013c258bbc32e4ff3c502a45b1ea5a125894db98a2] testlets:map[http:9bbb9448e6dfa8f0f09c70871f73a1176a3f2f95a3aeae940c3cd0079cbd5426 iperf:ff8ddea702fcf07bbabf3f6e78943d8be339626573931e5dcc5ef1bd8521d134]] DEBU[0067] Agent advertisement recieved: {"Uuid":"cd11bcc3d24094564d52716bfcfef64778eab35d0f3208b9b7c5e95261e2f5ad","DefaultAddr":"172.17.0.4","Expires":0,"LocalTime":"2017-03-10T07:22:16.593628356Z","Facts":{},"FactCollectors":{"get_addresses":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","get_hostname":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"},"Testlets":{"http":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}} WARN[0067] Agent cd11bcc3d24094564d52716bfcfef64778eab35d0f3208b9b7c5e95261e2f5ad did not have the required asset files. This advertisement is ignored. DEBU[0067] Sent task to cd11bcc3d24094564d52716bfcfef64778eab35d0f3208b9b7c5e95261e2f5ad: {"type":"DownloadAsset","assets":["http://\u003cnil\u003e:8090/factcollectors/get_addresses","http://\u003cnil\u003e:8090/factcollectors/get_hostname","http://\u003cnil\u003e:8090/testlets/http","http://\u003cnil\u003e:8090/testlets/iperf"]} DEBU[0070] Loaded assets: map[factcollectors:map[get_addresses:630e6d795ee363e3741e0eba74350645c8ea1896bb1a4cb1c7f18c71fdee6462 get_hostname:e3dd6bd5888c29046bafcf013c258bbc32e4ff3c502a45b1ea5a125894db98a2] testlets:map[http:9bbb9448e6dfa8f0f09c70871f73a1176a3f2f95a3aeae940c3cd0079cbd5426 iperf:ff8ddea702fcf07bbabf3f6e78943d8be339626573931e5dcc5ef1bd8521d134]]

Mierdin commented 7 years ago

@hanya1203 Please read http://todd.readthedocs.io/en/latest/installconfig/configure.html - you're seeing all of those nil values because neither the agent or the server have a correctly configured DefaultInterface - and looking at your configs, it looks like this is because you have set the DefaultInterface to use a loopback:

DefaultInterface = lo

This needs to be set to an accessible interface on both the agent and the server so the agent knows which IP address to download assets from, and so the server knows how to orchestrate tests.

Mierdin commented 7 years ago

I opened https://github.com/toddproject/todd/issues/133 aimed at preventing either the agent or server from starting when this happens, so it's a little clearer what's wrong. That should close this issue when merged.

hanya1203 commented 7 years ago

@Mierdin Thanks a lot. I can solve the error. In server.cfg, there was erroneous code. DefaultInterface = lo I changed the config so that Todd server look for suitable interface. Then Todd server recognize agent.

Mierdin commented 7 years ago

No problem. To be clear, you'll need to make the same change in the agent configuration, or you'll run into problems at test time.

On Mar 13, 2017 2:37 AM, "hanya1203" notifications@github.com wrote:

@Mierdin https://github.com/Mierdin Thanks a lot. I can solve the error. In server.cfg, there was erroneous code. DefaultInterface = lo I changed the config so that Todd server look for suitable interface. Then Todd server recognize agent.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/toddproject/todd/issues/132#issuecomment-286057280, or mute the thread https://github.com/notifications/unsubscribe-auth/AECM-9CG6QZVWkOC_VCFDg-XSKFX21wDks5rlQ5CgaJpZM4MaC6L .

Mierdin commented 7 years ago

133 should make it more obvious when this misconfiguration is taking place. You should definitely ensure that the DefaultInterface points to your intended "default interface" but #133 will make is so that you can't even run the agent or server unless this results in a valid IP address