Unable to connect to kachery nodes from network with restricted web access

[shashwatsridhar]

Hey,

I'm unable to download datasets to a system connected to a network with restricted web access (located at the Janelia campus). The kachery-p2p-start-daemon command issued from the system fails to initiate the connection required to access datasets. The output is as follows:

(neuropixels-2020) [sridhars@c11u24 kachery]$ kachery-p2p-start-daemon --channel flatiron1
npx: installed 13 in 2.289s
node: 14.10.0
+ export KACHERY_P2P_API_PORT=20431
+ KACHERY_P2P_API_PORT=20431
+ export KACHERY_P2P_CONFIG_DIR=/groups/pachitariu/home/sridhars/.kachery-p2p
+ KACHERY_P2P_CONFIG_DIR=/groups/pachitariu/home/sridhars/.kachery-p2p
+ exec npx kachery-p2p-daemon@0.4.28 start --channel flatiron1 --verbose 0 --dverbose 0 --port 0 --file-server-port 0
npx: installed 100 in 5.846s
____________________________________
DAEMON VERSION: kachery-p2p-0.4.28
PROTOCOL VERSION: kachery-p2p-0.4.18

CHANNEL: flatiron1 (1 node)
self b6ebc5

OTHER
self b6ebc5
____________________________________

[info] API server is running {"protocol":"http","port":"20431"}
Unable to connect to bootstrap node 45.33.92.31:45002: Unexpected server response: 503
Unable to connect to bootstrap node 45.33.92.33:45002: Unexpected server response: 503

I'm trying to download the datasets using the script provided at scripts/download_recordings.py, which obviously fails with the following error:

(neuropixels-2020) [sridhars@c11u24 datasets]$ python ~/repositories/neuropixels-data-sep-2020/scripts/download_recordings.py 
Creating (1 of 1): /groups/pachitariu/home/sridhars/working/datasets/cortexlab-drift-dataset1.dat
Traceback (most recent call last):
  File "/groups/pachitariu/home/sridhars/repositories/neuropixels-data-sep-2020/neuropixels_data_sep_2020/recordings.py", line 16, in load_known_recordings_dict
    x = kp.load_object(KNOWN_RECORDINGS_URI)
  File "/groups/pachitariu/home/sridhars/miniconda3/envs/neuropixels-2020/lib/python3.8/site-packages/kachery_p2p/core.py", line 278, in load_object
    local_path = load_file(uri, p2p=p2p, from_node=from_node, from_channel=from_channel)
  File "/groups/pachitariu/home/sridhars/miniconda3/envs/neuropixels-2020/lib/python3.8/site-packages/kachery_p2p/core.py", line 129, in load_file
    raise LoadFileError(f'Error loading file: {r["error"]}: {uri}')
kachery_p2p.exceptions.LoadFileError: Error loading file: File not found.: sha1://c0eef852db375ba9de2fe0871da3264b7435ee64/known_recordings.json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/groups/pachitariu/home/sridhars/repositories/neuropixels-data-sep-2020/scripts/download_recordings.py", line 22, in <module>
    recording = nd.load_recording(recording_id, download=True)
  File "/groups/pachitariu/home/sridhars/repositories/neuropixels-data-sep-2020/neuropixels_data_sep_2020/recordings.py", line 30, in load_recording
    valid_recordings: List[Any] = get_valid_recordings()
  File "/groups/pachitariu/home/sridhars/repositories/neuropixels-data-sep-2020/neuropixels_data_sep_2020/recordings.py", line 22, in get_valid_recordings
    x = load_known_recordings_dict()
  File "/groups/pachitariu/home/sridhars/repositories/neuropixels-data-sep-2020/neuropixels_data_sep_2020/recordings.py", line 18, in load_known_recordings_dict
    raise Exception('Problem loading recordings dict. Perhaps you are not running the kachery-p2p daemon?')
Exception: Problem loading recordings dict. Perhaps you are not running the kachery-p2p daemon?

The conda environment used was created using the instructions given in the readme for this repo. No additional packages were installed. Thanks!

[shashwatsridhar]

@alexmorley anything else I should add here?

[magland]

Hi @shashwatsridhar, you mention you have "restricted web access". Maybe you are not able to establish an outgoing connection to that IP address? Are you able to use a different computer or a different network?

[shashwatsridhar]

I am able to use a different computer (my own) which is not in the Janelia network, and I am able to download datasets. Moreover, when I route my traffic through the Janelia VPN, I cannot access the dataset descriptions on laboratorybox.org (for eg. this). The page loads, but only the text "Loading..." is shown. I am wondering if the two issues are related, and maybe provide more insight into what might be wrong?

I could download the datasets to a system outside the network (say, to my PC), and re-upload them to the remote system via ssh, but that's a pretty inefficient method to access the data. So I thought I'd post an issue and see if there are any solutions you might have (like choosing the right port, or so).

[magland]

Sounds to me like the Janelia network is blocking websocket connections (which would affect both the website and the kachery daemon). I don't have a solution at the moment, unfortunately.

[alexmorley]

@shashwatsridhar can you open the "network" tab of developer tools and click reload and post a screenshot while on labbox? It might be because the websockets don't use ssl (they are ws:// as opposed to wss://) which would be fixable.

[shashwatsridhar]

Here it is:

I don't see anything indicating whether SSL is being used or not. Hope you spot something useful!

Edit: ah, I think I do. The greyed-out lock symbol in front of the urls, I guess?

[magland]

I can confirm it is using ws:// rather than wss://

@alexmorley, perhaps we can chat about how to convert this to wss. I am deploying using kubernetes (and linode). Maybe there is an easy way to adjust this. I only have one reference to ws:// in the source code. So maybe we just add an "s" and update something on the kube config?