Running google datastore emulator with docker

[m1m6]

i'm trying to create a datastore emulator with docker , and followed the instructions here https://cloud.google.com/datastore/docs/tools/datastore-emulator

also i used the cloud-sdk https://hub.docker.com/r/google/cloud-sdk/

I was able to create the emulator and authenticate with it, but when i attempt to access it throght localhost:8000/datastore it gives me "Not found" response,

How could i access the datastore data?

the command i used to create it is: docker run -p 8000:8000 google/cloud-sdk gcloud beta emulators datastore start --project=pname --host-port localhost:8000 --no-store-on-disk

[ahmetb]

Can you try --host-port 0.0.0.0:8000? using localhost restrict the traffic only to the inside of container. using 0.0.0.0 listens on all interfaces which makes it accessible from outside the container (i.e. from your laptop). Similar discussion at #104.

[m1m6]

@ahmetb still the same when accessing http://0.0.0.0:8000/datastore it gives not found also tried to bind with host port

docker run -h gdatastore -p 8000:8000 google/cloud-sdk gcloud beta emulators datastore start --project=pname --host-port gdatastore:8000 --no-store-on-disk

but still the same

[ahmetb]

@m1m6 I see your --host-port gdatastore:8000 is not --host-port 0.0.0.0:8000. Can you try with 0.0.0.0?

[m1m6]

Tried and got the same results 😅😅

[ahmetb]

can you exec into the container and try wget/curl into the localhost:8000?

[m1m6]

curl -I localhost:8000 HTTP/1.1 404 Not Found content-length: 10

but if i access it from browser localhost:8000 ==> response "OK" localhost:8000/datastore ==> response "Not Found"

[ahmetb]

(I'm assuming you ran the curl inside the container). Please try omitting -I as the implementation might not be supporting the HEAD method.

It sounds like your problem might be with the emulator and not the Docker image.

Can you try installing the gcloud SDK and the emulator directly on your machine?

[m1m6]

Ok when i hit curl 0.0.0.0:8000 it returns OK

[ahmetb]

So is this problem resolved?

Inside the container, curl -v localhost:8000 should return 200 OK, and if you started with --host-port 0.0.0.0:8000. At that point wherever you're mapping (-p 8000:8000) the port, querying port 8000 on that should work too.

[m1m6]

Got another error when running the datastore emulator, i took the environment vars and set them before running the java application container

and i used this to connect to the datastore

Datastore ds = DatastoreOptions.newBuilder().setHost("http://localhost:8282").setProjectId("tradeos-test1").build().getService();

docker command sudo docker run -p 8282:8282 google/cloud-sdk gcloud beta emulators datastore start --project=tradeos-test1 --host-port localhost:8282

env vars:

export DATASTORE_DATASET=tradeos-test1
 export DATASTORE_EMULATOR_HOST=localhost:8282
 export DATASTORE_EMULATOR_HOST_PATH=localhost:8282/datastore
 export DATASTORE_HOST=http://localhost:8282
 export DATASTORE_PROJECT_ID=test1

when hitting an restfull api the connection to the datastore occur, but when attempting to put any data it gives me connection refused

INFO [2017-10-26 09:34:56,749] edu.mvm.hello.business.service.UserService: com.google.cloud.datastore.DatastoreImpl@1387f297 connected successfully ERROR [2017-10-26 09:35:02,791] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: ebeb29c5d546cfa8 ! java.net.ConnectException: Connection refused (Connection refused) ! at java.net.PlainSocketImpl.socketConnect(Native Method) ! at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ! at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ! at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ! at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ! at java.net.Socket.connect(Socket.java:589) ! at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ! at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ! at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ! at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ! at sun.net.www.http.HttpClient.New(HttpClient.java:339) ! at sun.net.www.http.HttpClient.New(HttpClient.java:357) ! at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) ! at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) ! at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) ! at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) ! at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316) ! at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291) ! at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77) ! at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981) ! at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:183) ! ... 72 common frames omitted ! Causing: com.google.datastore.v1.client.DatastoreException: I/O error ! at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:224) ! at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:193) ! at com.google.datastore.v1.client.Datastore.commit(Datastore.java:85) ! at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.commit(HttpDatastoreRpc.java:153) ! ... 70 common frames omitted ! Causing: com.google.cloud.datastore.DatastoreException: I/O error

you can see from the first line of loggers, that the connection happened successfully, but inserting not occurs correctly

[m1m6]

Ok, got it works, after i set the ip address for the container in env vars, and change the url of the datastore connection to the ip address,

 export DATASTORE_DATASET=tradeos-test1
 export DATASTORE_EMULATOR_HOST=172.17.0.2:8282
 export DATASTORE_EMULATOR_HOST_PATH=172.17.0.2:8282/datastore
 export DATASTORE_HOST=http://172.17.0.2:8282
 export DATASTORE_PROJECT_ID=tradeos-test1

Datastore ds = DatastoreOptions.newBuilder().setHost("http://172.17.0.2:8282").setProjectId("tradeos-test1").build().getService();

thanks @ahmetb for ur help :+1:

[ahmetb]

No problem. 👍

[DSh3p4rd]

Im trying to run the datastore emulator in a background process so it doesnt hang... I get a weird error in the cloud build container which I'm guessing has less to do with this image and more to do with how args are processed.

Anyway,

steps:
  - name: 'gcr.io/cloud-builders/gcloud'
    args: ['beta', 'emulators', 'datastore', 'start', '--host-port=localhost:8081', '--no-store-on-disk', '--consistency=1.0', '&']

results in:

Already have image (with digest): gcr.io/cloud-builders/gcloud
ERROR: (gcloud.beta.emulators.datastore.start) unrecognized arguments: &
Usage: gcloud beta emulators datastore start [optional flags]
optional flags may be --consistency | --data-dir | --help | --host-port |
--legacy | --store-on-disk

[philippgille]

@ahmetb : I recently had a similar problem where I tried to connect to the emulator running in Docker without --host-port set. See https://github.com/googleapis/google-cloud-go/issues/1258.

Docker containers are rarely run with --network host, and most services just work when running as Docker container and publishing the port (e.g. Redis, etcd, but also AWS DynamoDB emulator, Azurite Azure Storage emulator, ...), so people are used to that behavior. So I think there are two options:

1. Either listen for connections on 0.0.0.0 by default (not the emulator generally, but specifically when run as Docker container)
2. Add a > Note: For accessing the emulator from outside of the Docker container you need to include `--host-port` in the Docker run command! to the Docker Hub description / repository README

I guess (1) is not an option because of security concerns (in case the container IS run with --host-port for example), but I see no downside of (2).