search

discoproject / disco

a Map/Reduce framework for distributed computing
http://discoproject.org
BSD 3-Clause "New" or "Revised" License
1.63k stars 241 forks source link

assert master url scheme and netloc #643

Closed ghost closed 7 years ago

ghost commented 7 years ago

Given the following case, where disco-master has been set in /etc/hosts and the defaults have been used, except DISCO_MASTER:

(venv_xk) jenkins@jenkins-4139:~/workspace/count_keys$ printenv | grep DISCO
DISCO_HOME=
DISCO_MASTER=disco-master
DISCO_MASTER_HOST=disco-master

Requests to disco-master hang, timeout after a few minutes:

Traceback (most recent call last):
  File "wtf.py", line 103, in <module>
    job.run(input=[(str(i), str(i), b) for i, b in enumerate(batch)], params=params)
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/job.py", line 145, in run
    self.worker.jobdict(self, **jobargs),
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/worker/pipeline/worker.py", line 189, in jobdict
    settings=job.settings)
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/util.py", line 283, in inputlist
    for input in inputs) if inp]
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/util.py", line 42, in chainify
    return list(chain(*iterable))
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/util.py", line 283, in <genexpr>
    for input in inputs) if inp]
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/util.py", line 275, in inputexpand
    return [inputlist(input, label=label, settings=settings)]
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/util.py", line 283, in inputlist
    for input in inputs) if inp]
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/util.py", line 42, in chainify
    return list(chain(*iterable))
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/util.py", line 283, in <genexpr>
    for input in inputs) if inp]
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/util.py", line 278, in inputexpand
    return chainify(blobs for name, tags, blobs in ddfs.findtags(input))
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/util.py", line 42, in chainify
    return list(chain(*iterable))
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/util.py", line 278, in <genexpr>
    return chainify(blobs for name, tags, blobs in ddfs.findtags(input))
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/ddfs.py", line 191, in findtags
    urls        = self.get(tag, token=token).get('urls', [])
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/ddfs.py", line 206, in get
    return self._download(canonizetag(tag), token=token)
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/ddfs.py", line 422, in _download
    token=self._token(url, token, method))
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/comm.py", line 96, in download
    headers=headers, sleep=sleep).read()
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/comm.py", line 78, in request
    return request(method, url, data=data, headers=headers, sleep=sleep + 1)
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/comm.py", line 78, in request
    return request(method, url, data=data, headers=headers, sleep=sleep + 1)
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/comm.py", line 78, in request
    return request(method, url, data=data, headers=headers, sleep=sleep + 1)
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/comm.py", line 78, in request
    return request(method, url, data=data, headers=headers, sleep=sleep + 1)
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/comm.py", line 78, in request
    return request(method, url, data=data, headers=headers, sleep=sleep + 1)
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/comm.py", line 78, in request
    return request(method, url, data=data, headers=headers, sleep=sleep + 1)
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/comm.py", line 78, in request
    return request(method, url, data=data, headers=headers, sleep=sleep + 1)
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/comm.py", line 78, in request
    return request(method, url, data=data, headers=headers, sleep=sleep + 1)
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/comm.py", line 78, in request
    return request(method, url, data=data, headers=headers, sleep=sleep + 1)
  File "/var/lib/jenkins/workspace/count_keys/venv_xk/local/lib/python2.7/site-packages/disco/comm.py", line 76, in request
    raise CommError(errmsg, url, status)
disco.error.CommError: Unable to access resource (://disco-master/ddfs/tag/0): [Errno 111] Connection refused

It took me a while to figure out that urlresolve wasn't doing what I had expected, also I didn't see any test for it, so I tried to add the test case for what I faced.

pooya commented 7 years ago

Thank! Cleaned up the commits and applied!