flatironinstitute / spikeforest_old

SpikeForest -- spike sorting analysis for website
Apache License 2.0
16 stars 8 forks source link

error when downloading test data set #59

Open jenniferColonell opened 5 years ago

jenniferColonell commented 5 years ago

I'm getting an error when the running example code supplied for downloading a test dataset using SFMdaRecordingExtractor.

The traceback from the error is pasted below. Verbose options in mountainremoteclient.py (where the error occurs) have been turned ON. Looks like response to request.urlopen(url) is invalid? Is this a permissions issue? Let me know if it might be a problem on the requesting side.

Thanks!

jennifer

loading recording... _http_get_json::: http://pairio.org/get/spikeforest/61c9b2b17db77a27841bbeeabff923448b0f6388 url open response: <http.client.HTTPResponse object at 0x7f656dbb5630> Traceback (most recent call last): File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainremoteclient.py", line 175, in _http_get_json ret = json.load(req) File "/usr/lib/python3.5/json/init.py", line 268, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/lib/python3.5/json/init.py", line 312, in loads s.class.name)) TypeError: the JSON object must be str, not 'bytes'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "download.py", line 12, in recording = SFMdaRecordingExtractor(dataset_directory=recdir, download=True) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/spikeforest/extractors/sfmdaextractors/sfmdaextractors.py", line 33, in init self._dataset_params = read_dataset_params(dataset_directory, params_fname) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/spikeforest/extractors/sfmdaextractors/sfmdaextractors.py", line 224, in read_dataset_params fname2 = ca.realizeFile(path=fname1) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(*args, kwargs) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainclient.py", line 545, in realizeFile return self._realize_file(path=path, dest_path=dest_path, show_progress=show_progress, download_from=download_from, local_only=local_only, remote_only=remote_only) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainclient.py", line 1175, in _realize_file path=path, local_only=local_only, resolve_locally=resolve_locally, dest_path=dest_path, show_progress=show_progress) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(*args, *kwargs) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainclient.py", line 1526, in realizeFile sha1 = self.computeFileSha1(path=path) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(args, kwargs) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainclient.py", line 1586, in computeFileSha1 dd = self._parent.loadObject(path='sha1://' + sha1) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(*args, kwargs) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainclient.py", line 653, in loadObject subkey=subkey, local_first=local_first, collection=collection, download_from=download_from, local_only=local_only, remote_only=remote_only) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(*args, *kwargs) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainclient.py", line 871, in loadText key=key, path=path, subkey=subkey, local_first=local_first, collection=collection, download_from=download_from, local_only=local_only, remote_only=remote_only) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(args, kwargs) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainclient.py", line 545, in realizeFile return self._realize_file(path=path, dest_path=dest_path, show_progress=show_progress, download_from=download_from, local_only=local_only, remote_only=remote_only) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainclient.py", line 1195, in _realize_file url, size = self._find_on_kachery_or_kbucket(download_from=df0, sha1=sha1) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainclient.py", line 1271, in _find_on_kachery_or_kbucket kachery_url = self._resolve_kachery_url(download_from) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainclient.py", line 1230, in _resolve_kachery_url return self._get_value_from_alias(name) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainclient.py", line 1145, in _get_value_from_alias ret = self.getValue(key=vals[1], collection=vals[0]) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(*args, *kwargs) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainclient.py", line 393, in getValue collection=collection, local_first=local_first, check_alt=check_alt) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainclient.py", line 1134, in _get_value key=key, subkey=subkey, collection=collection, url=self._pairio_url) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainremoteclient.py", line 42, in getValue obj = _http_get_json(url0) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(args, **kwargs) File "/opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainremoteclient.py", line 177, in _http_get_json raise Exception('Unable to load json from url: ' + url) Exception: Unable to load json from url: http://pairio.org/get/spikeforest/61c9b2b17db77a27841bbeeabff923448b0f6388

alexmorley commented 5 years ago

@jenniferColonell thanks for the detailed bug report :birthday:

I'm 95% sure this is a python 3.5 (what you have) vs python 3.6/7 (which is what most of us are using) issue.

Could you test something for me?

If you edit:
line 174 /opt/spikeforest/SF_venv/lib/python3.5/site-packages/mountainclient/mountainremoteclient.py and change ret = json.load(req) to ret = json.load(req.json())

I'm pretty sure that should fix it. Let me know either way :)

jenniferColonell commented 5 years ago

@alexmorley Thanks for that prompt reply, and sorry for the delay getting back to you.

My Linux workstation was having issues (unrelated), so I tried the test code on a different system (janelia compute cluster) where Python 3.7 is installed.

I ran into a different error, still in mountainremoteclient.py. However, the version of moutainremoteclient is different from that which got installed under python 3.5 -- so the fix you suggested won't apply. The code appears to be from the static_typing branch rather than the master branch.

I'm not sure if it's a useful clue, but if I try to go to the url manually, and then follow the link to the json file (in Firefox), the link is blocked by local IT policy. I don't see any signs of that happening when accessing through your code....it looks like a timeout instead. I tried bumping the time up to 5 seconds, with no change. Still, permissions on my side may be a problem.

The traceback is pasted below. Verbose was turned on in moutainremoteclient.py.

loading recording... _http_get_json::: http://pairio.org/get/spikeforest/61c9b2b17db77a27841bbeeabff923448b0f6388 Elapsed time for _http_get_json: 0.1043250560760498 http://pairio.org/get/spikeforest/61c9b2b17db77a27841bbeeabff923448b0f6388 _http_get_json::: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 Retrying http request in 0.2 sec: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 _http_get_json::: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 Retrying http request in 0.5 sec: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 _http_get_json::: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 Retrying http request in 5.0 sec: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 _http_get_json::: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 Traceback (most recent call last): File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 131, in _http_get_json req = request.urlopen(url) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 531, in open response = meth(req, response) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 569, in error return self._call_chain(args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 503: Service Unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 131, in _http_get_json req = request.urlopen(url) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 531, in open response = meth(req, response) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 569, in error return self._call_chain(args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 503: Service Unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 131, in _http_get_json req = request.urlopen(url) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 531, in open response = meth(req, response) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 569, in error return self._call_chain(args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 503: Service Unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 131, in _http_get_json req = request.urlopen(url) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 531, in open response = meth(req, response) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 569, in error return self._call_chain(args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 503: Service Unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainclient.py", line 1195, in _find_on_kachery_or_kbucket obj = _http_get_json(check_url, verbose=self._verbose) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(*args, kwargs) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 137, in _http_get_json return _http_get_json(url, verbose=verbose, retry_delays=retry_delays[1:]) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(*args, *kwargs) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 137, in _http_get_json return _http_get_json(url, verbose=verbose, retry_delays=retry_delays[1:]) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(args, kwargs) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 137, in _http_get_json return _http_get_json(url, verbose=verbose, retry_delays=retry_delays[1:]) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(*args, *kwargs) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 139, in _http_get_json raise Exception('Unable to open url: ' + url) Exception: Unable to open url: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 WARNING: failed in check to kachery spikeforest.public: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 _http_get_json::: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 Retrying http request in 0.2 sec: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 _http_get_json::: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 Retrying http request in 0.5 sec: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 _http_get_json::: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 Retrying http request in 5.0 sec: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 _http_get_json::: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 Traceback (most recent call last): File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 131, in _http_get_json req = request.urlopen(url) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 531, in open response = meth(req, response) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 569, in error return self._call_chain(args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(*args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 503: Service Unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 131, in _http_get_json req = request.urlopen(url) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 531, in open response = meth(req, response) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 569, in error return self._call_chain(args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 503: Service Unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 131, in _http_get_json req = request.urlopen(url) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 531, in open response = meth(req, response) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 569, in error return self._call_chain(args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 503: Service Unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 131, in _http_get_json req = request.urlopen(url) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 531, in open response = meth(req, response) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 569, in error return self._call_chain(args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(args) File "/misc/local/python-3.7.0/lib/python3.7/urllib/request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 503: Service Unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainclient.py", line 1195, in _find_on_kachery_or_kbucket obj = _http_get_json(check_url, verbose=self._verbose) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(*args, kwargs) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 137, in _http_get_json return _http_get_json(url, verbose=verbose, retry_delays=retry_delays[1:]) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(*args, *kwargs) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 137, in _http_get_json return _http_get_json(url, verbose=verbose, retry_delays=retry_delays[1:]) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(args, kwargs) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 137, in _http_get_json return _http_get_json(url, verbose=verbose, retry_delays=retry_delays[1:]) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mtlogging/mtlogging.py", line 26, in wrapper ret = function(*args, **kwargs) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/mountainclient/mountainremoteclient.py", line 139, in _http_get_json raise Exception('Unable to open url: ' + url) Exception: Unable to open url: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 WARNING: failed in check to kachery spikeforest.public: http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20 Traceback (most recent call last): File "download_data.py", line 12, in recording = SFMdaRecordingExtractor(dataset_directory=recdir, download=True) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/spikeforest/extractors/sfmdaextractors/sfmdaextractors.py", line 33, in init self._dataset_params = read_dataset_params(dataset_directory, params_fname) File "/groups/apig/home/colonellj/sfenv/sf_env/lib/python3.7/site-packages/spikeforest/extractors/sfmdaextractors/sfmdaextractors.py", line 226, in read_dataset_params raise Exception('Unable to find file: ' + fname1) Exception: Unable to find file: sha1dir://fb52d510d2543634e247e0d2d1d4390be9ed9e20.synth_magland/datasets_noise10_K10_C4/001_synth/params.json

alexmorley commented 5 years ago

Yeah this is definitely something different now.

Firefox), the link is blocked by local IT policy. I don't see any signs of that happening when accessing through your code....it looks like a timeout instead. I tried bumping the time up to 5 seconds, with no change. Still, permissions on my side may be a problem.

If you can't access the file from a browser I think that's it (it will timeout if it is blocked from accessing a service). Weird that it would be restricted though but maybe your local IT policy doesn't allow visiting sites that aren't registered via DNS? If that's the case that's super annoying... @magland does that sound right to you?

magland commented 5 years ago

What @alexmorley is saying sounds right to me. Probably the solution is for us to get a domain name for the public spikeforest kachery. I should be able to do that on Friday.

jenniferColonell commented 5 years ago

@alexmorley @magland -- I talked to folks in our IT group, and the issue is likely that our network won't allow connecting to that port number. So, if I try to connect to the IP address + port number: http://132.249.245.246:24341 I'm blocked. But if I just go to the IP address (http://132.249.245.246), I'm not blocked -- but there's nothing there when viewed in a browser (it times out, eventually).

So, to get this to work for Janelia users, the port number would probably need to be changed, We should have better luck with one of the standard http port numbers: 80, 8008, 8080, and 593.

This could easily be an issue at other institutions with firewalls.

alexmorley commented 5 years ago

Eek that's annoying. Thanks for checking, I'm sure it is an issue elsewhere so we'll find a fix :)

magland commented 5 years ago

@jenniferColonell @alexmorley. It's not straightforward to change the port on our end, but there are various workarounds we could explore, including making a proxy server. But let's first try the following easier solution:

Please verify that this does NOT work within Janelia:

curl http://132.249.245.246:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20

Then see if this works, with your fingers crossed:

curl http://spikeforestpublic.org:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20

If it works, it should return:

{
    "success": true,
    "found": true,
    "size": 46092
}

(Note that the above URLs were edited)

Then for completeness, please also try the following:

curl http://kbucket.flatironinstitute.org/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20

Thanks!

jenniferColonell commented 5 years ago

@magland @alexmorley

confirmed that the first ip+port does NOT work from w/in Janelia.

Unfortunately, fingers crossed or not, the second version with the domain name is also blocked. The output is pasted below, just showing the html for the firewall blocking message.

The kbucket url is OK, and returns the output you expected.

I'll send a note to our IT department to see if it's possible to make exceptions for specific IP+port combinations.

----output from failed access of the spikeforestpublic.org IP

[colonellj@c11u24 ~]$ curl http://spikeforestpublic.org:24341/check/sha1/fb52d510d2543634e247e0d2d1d4390be9ed9e20

Application Blocked

Application Blocked

Access to the application you were trying to use has been blocked in accordance with HHMI policy IT-105.
Please contact HHMI Technical Support at ext. 4030 if you believe this is in error.

User: 10.36.11.34

Application: web-browsing

magland commented 5 years ago

Okay. And I'll also check to see what can be done on our end.

jenniferColonell commented 4 years ago

@magland , @alexmorley Finally got resolution to this problem on our end -- the Janelia IT department made an exception specifically for port 24341. I don't have any idea how frequently this will be an issue with other institutions, but I'm set for now. Thanks! If it's useful to have me test new addresses/ports, etc. just let me know.

magland commented 4 years ago

Thanks for the update and glad that worked.

On Mon, Jul 8, 2019 at 8:19 PM jenniferColonell notifications@github.com wrote:

@magland https://github.com/magland , @alexmorley https://github.com/alexmorley Finally got resolution to this problem on our end -- the Janelia IT department made an exception specifically for port 24341. I don't have any idea how frequently this will be an issue with other institutions, but I'm set for now. Thanks! If it's useful to have me test new addresses/ports, etc.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/flatironinstitute/spikeforest/issues/59?email_source=notifications&email_token=AA4CIQGNMZ6LSKNL2ZN62CTP6PKRHA5CNFSM4HYP5X3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZOW2RY#issuecomment-509439303, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4CIQH6IPHLOVAYQB6YFJ3P6PKRHANCNFSM4HYP5X3A .