Unable to get list of datasets

[tschaffter]

I'm running a local instance of the data node available on localhost:8080. The data node has two datasets that I have created using Swagger UI.

I'm trying to get the list of datasets in this local instance:

from nlpsandboxclient import client

data_node_host = "http://localhost:8080/api/v1"

datasets = client.list_datasets(host=data_node_host)

for dataset in datasets:
  print("Dataset name: " + dataset.name)

The output does not show any dataset names or error.

Tagging @yy6linda

[thomasyu888]

@tschaffter, I'm not sure this will work but can you try 0.0.0.0 instead of localhost?

[tschaffter]

@thomasyu888 Same behavior when using 0.0.0.0 instead of localhost

[tschaffter]

I placed the code in a Python script and I get a different behavior.

The code below does not show anything in the terminal using python list_dataset.py:

from nlpsandboxclient import client

data_node_host = "http://localhost:8080/api/v1"

datasets = client.list_datasets(host=data_node_host)

Now when adding the iteration over datasets:

from nlpsandboxclient import client

data_node_host = "http://localhost:8080/api/v1"

datasets = client.list_datasets(host=data_node_host)

for dataset in datasets:
  print("Dataset name: " + dataset.name)

Error:

# python list_datasets.py
Traceback (most recent call last):
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/http/client.py", line 1253, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/http/client.py", line 1299, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/http/client.py", line 1248, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/http/client.py", line 1008, in _send_output
    self.send(msg)
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/http/client.py", line 948, in send
    self.connect()
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f9d45531160>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rstudio/nlpsandbox/python/list_datasets.py", line 7, in <module>
    for dataset in datasets:
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/nlpsandboxclient/client.py", line 576, in list_datasets
    datasets = api_instance.list_datasets(offset=offset, limit=limit)
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/nlpsandbox/api_client.py", line 772, in __call__
    return self.callable(self, *args, **kwargs)
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/nlpsandbox/api/dataset_api.py", line 465, in __list_datasets
    return self.call_with_http_info(**kwargs)
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/nlpsandbox/api_client.py", line 834, in call_with_http_info
    return self.api_client.call_api(
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/nlpsandbox/api_client.py", line 409, in call_api
    return self.__call_api(resource_path, method,
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/nlpsandbox/api_client.py", line 196, in __call_api
    response_data = self.request(
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/nlpsandbox/api_client.py", line 435, in request
    return self.rest_client.GET(url,
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/nlpsandbox/rest.py", line 232, in GET
    return self.request("GET", url,
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/nlpsandbox/rest.py", line 198, in request
    r = self.pool_manager.request(method, url,
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/opt/miniconda/envs/nlpsandbox/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /api/v1/datasets?offset=0&limit=10 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9d45531160>: Failed to establish a new connection: [Errno 111] Connection refused'))

Same behavior with 0.0.0.0.

@thomasyu888 @yy6linda While I don't know why the connection fails, I'm surprised that I don't get this error on the line datasets = ....

[tschaffter]

The data node is up and running locally:

$ curl -X GET "http://localhost:8080/api/v1/datasets?limit=10&offset=0" -H  "accept: application/json"
{
  "datasets": [
    {
      "name": "datasets/awesome-dataset"
    },
    {
      "name": "datasets/awesome-dataset-2"
    }
  ],
  "limit": 10,
  "links": {
    "next": ""
  },
  "offset": 0,
  "totalResults": 2
}

[thomasyu888]

@tschaffter I can't reproduce the problem:

from nlpsandboxclient import client
data_node_host = "http://localhost:8080/api/v1"
datasets = client.list_datasets(host=data_node_host)
for dataset in datasets:
     print(dataset)
{'name': 'datasets/awesome-dataset'}
{'name': 'datasets/2014-i2b2-20210603'}
{'name': 'datasets/test-dataset'}
{'name': 'datasets/i2b2-phi-20210603'}
{'name': 'datasets/i2b2-phi-20210606'}

You shouldn't get any error when running client.list_datasets() because it returns a generator, so until you iterate through the generator, no code is actually run.

Are you running the code inside a docker container? If you are, you'll have to add --network host for your code to have access to the localhost.

[tschaffter]

Are you running the code inside a docker container? If you are, you'll have to add --network host for your code to have access to the localhost.

You're right, I forgot about this. Thanks @thomasyu888 ! I will close this ticket when I can confirm the fix.

[tschaffter]

@thomasyu888 @yy6linda I have finalized the initial version of the notebook that show how to interact with a data node. The notebook successfully connect to a local instance of a data node run locally using Docker. This is achieve by giving the Docker container that run RStudio access to the Docker network of the data node instance. See link below for more information on the setup.

https://github.com/nlpsandbox/nlpsandbox-notebooks/blob/main/notebooks/data-node.Rmd

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.