Connection error when creating a dataset

[jitendrawbd]

What happened?

I am using the emulator with Python. I am using Docker container to run the emulator. I am trying to run a very simple test script which creates a dataset. When running the code, I get a connection timeout error

What did you expect to happen?

Expected the mock dataset to be created on running the code

How can we reproduce it (as minimally and precisely as possible)?

Create a test file with the below code

from google.cloud import bigquery
from google.auth.credentials import AnonymousCredentials
from google.api_core.client_options import ClientOptions

def test_bigquery_client():
   client_options = ClientOptions(api_endpoint="http://0.0.0.0:9050")
   client = bigquery.Client(
    "local",
    client_options=client_options,
    credentials=AnonymousCredentials(),
   )
  dataset = bigquery.Dataset('local.test_dataset')
  dataset = client.create_dataset(dataset)
  print(dataset)

Run the pytest, you will see the error

P.S.: I have also tried to use client.create_dataset("test_dataset"). Getting the same result

Anything else we need to know?

No response

[ohaibbq]

Can you post the docker run configuration you are using when starting the emulator as well as any logs from the container? Are you mapping a host port to the container port?

We are using a similar setup and can create datasets just fine.

python
Python 3.11.7 (main, Dec  4 2023, 18:10:11) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.cloud import bigquery
from google.auth.credentials import AnonymousCredentials
from google.api_core.client_options import ClientOptions

client_options = ClientOptions(api_endpoint="http://0.0.0.0:9050")
client = bigquery.Client(
"local",
client_options=client_options,
credentials=AnonymousCredentials(),
)
dataset = bigquery.Dataset('local.test_dataset')
dataset = client.create_dataset(dataset)
print(dataset)
>>> from google.auth.credentials import AnonymousCredentials
>>> from google.api_core.client_options import ClientOptions
>>>
>>>
>>> client_options = ClientOptions(api_endpoint="http://0.0.0.0:9050")
>>> client = bigquery.Client(
... "local",
... client_options=client_options,
... credentials=AnonymousCredentials(),
... )
>>> dataset = bigquery.Dataset('local.test_dataset')
>>> dataset = client.create_dataset(dataset)
>>> print(dataset)
Dataset(DatasetReference('local', 'test_dataset'))

docker run -p 9050:9050 --rm ghcr.io/goccy/bigquery-emulator:latest /bin/bigquery-emulator --project=local --log-level=info
[bigquery-emulator] REST server listening at 0.0.0.0:9050
[bigquery-emulator] gRPC server listening at 0.0.0.0:9060
2024-03-28T17:41:34.742Z    INFO    server/middleware.go:53 POST /bigquery/v2/projects/local/datasets   {"query": "prettyPrint=false"}

[jitendrawbd]

Thanks for the response. I am using the below docker run command docker run -it ghcr.io/goccy/bigquery-emulator:latest --project=local --log-level=info

The only log I see in the container is below

[bigquery-emulator] REST server listening at 0.0.0.0:9050
[bigquery-emulator] gRPC server listening at 0.0.0.0:9060

The error I get is below

[ohaibbq]

You'll need to map a Docker host port to the emulator container port via docker run -p 9050:9050 when running the emulator in order for the client to access it on the local 0.0.0.0 network.

[jitendrawbd]

Thanks for the response! I did try with port mapping too. Getting the same error docker run -p 9050:9050 ghcr.io/goccy/bigquery-emulator:latest --project=local --log-level=info

P.S.: I am using python 3.12.2 version

[ohaibbq]

Are you running your pytest command locally or in a separate Docker container? If you are running it in another docker container you may need to run it in the same Docker network.

I'd recommend going over this page and have a look at the example they lay out with redis-server and redis-cli.

https://docs.docker.com/network/#container-networks

[jitendrawbd]

I am running the pytest locally

[ohaibbq]

The reproduction case you provided does not fully encompass what might be going wrong as it works just fine for me.

Are you able to use port mapping successfully with other containers? Perhaps Docker doesn't have permission to mount ports? Does the bq tool work when the emulator is running?

bq --api "http://0.0.0.0:9050" query --nouse_legacy_sql --format prettyjson --project_id local 'SELECT TRUE;'

[jitendrawbd]

Not sure why, but I had to use http://localhost:9050 instead of http://0.0.0.0:9050 to get this working. I used all the above docker commands with connection to localhost to get it working. Thanks for the help!