Invalid JSON on searching Gateway

[genekogan]

Running the Federated_SMS_Spam_prediction [Part 2] - Train a Model tutorial, I get the following error on data = my_grid.search("#X", "#spam", "#dataset"):

ValueError                                Traceback (most recent call last)
<ipython-input-4-d51f7579f673> in <module>
----> 1 data = my_grid.search("#X", "#spam", "#dataset")
      2 target = my_grid.search("#Y", "#spam", "#dataset")

~/Library/Python/3.7/lib/python/site-packages/grid/grid_network.py in search(self, *query)
     39         # Connect with grid nodes that contains the dataset and get their pointers
     40         tensor_set = []
---> 41         for node_id, node_url in match_nodes:
     42             worker = self.__connect_with_node(node_id, node_url)
     43             tensor_set.append(worker.search(query))

ValueError: too many values to unpack (expected 2)

Digging a bit further, trying it out manually, I see that the problem is an invalid JSON error on the query, originating in grid/grid_network.py.

body = json.dumps({"query": list(["#X", "#spam", "#dataset"])})
print(body)

gives {"query": ["#X", "#spam", "#dataset"]} which appears to me as normal json. Then:

response = requests.post("http://localhost:8080/search", data=body)
print(response.content)

The query goes through (response code is 200). But the response content is b'{"message": "Invalid JSON format."}'.

The gateway and two workers appear to be running fine for me. Not sure if the error is on my side or a bug.

[IonesioJunior]

Hello @genekogan, Thank you for your feedback! Did you test it using grid library or syft library?

[genekogan]

It's using Grid (this repo). The error occurs in this example

[IonesioJunior]

Could you test with syft.grid module? Actually grid library is deprecated and should be removed soon. Fortunately, we have the same tutorial using syft.grid module. If the errors persist, please send a message here. I'll be happy to help you.

[genekogan]

I am still getting the same error when I use the above linked tutorial from PySyft. On torch==1.4.0, syft==0.2.2a1.

Something confusing me which may also be relevant is that the tutorial instructs to run gateway.py and websocket_app.py but these files don't exist in PySyft, they appear to be only in PyGrid, so I am running them from there. Additionally, python websocket_app.py --start_local_db --id=alice --port=3001 --gateway_url=http://localhost:5000 must be a typo as --start_local_db is not a parameter of websocket_app.py.

[IonesioJunior]

Yep, you're right! You should run these apps. this repository aims to be the platform that runs pysyft library over the wire! :) We're moving the stuffs to its places to clarify their meaning. Unfortunately it isn't done yet, so its a little bit confusing. About your issue: I'll try to dig deep to understand what's happening. Could you send me your python env (pip freeze), please?

[genekogan]

Python 3.7.4

absl-py==0.8.1
alembic==1.3.3
appdirs==1.4.3
appnope==0.1.0
asn1crypto==1.0.1
aspy.yaml==1.3.0
astor==0.8.0
astroid==2.3.2
asyncio==3.4.3
attrs==19.2.0
backcall==0.1.0
beautifulsoup4==4.8.1
black==19.10b0
bleach==3.1.0
bs4==0.0.1
certifi==2019.11.28
cffi==1.12.3
cfgv==2.0.1
chardet==3.0.4
Click==7.0
coverage==5.0.3
cryptography==2.7
cycler==0.10.0
decorator==4.4.0
defusedxml==0.6.0
dj-database-url==0.5.0
Django==2.2.6
django-heroku==0.3.1
dlib==19.18.0
dnspython==1.16.0
entrypoints==0.3
eventlet==0.25.0
face-recognition==1.2.3
face-recognition-models==0.3.0
Flask==1.0.2
Flask-Cors==3.0.7
Flask-Login==0.4.1
Flask-Migrate==2.5.2
Flask-Session==0.3.1
Flask-SocketIO==4.2.1
Flask-Sockets==0.2.1
Flask-SQLAlchemy==2.4.1
Flask-Testing==0.7.1
freetype-py==2.1.0.post1
gast==0.2.2
gevent==1.4.0
gevent-websocket==0.10.1
glfw==1.8.3
google-pasta==0.1.7
greenlet==0.4.15
grid==0.1.0.0a1.dev437
grpcio==1.24.1
gunicorn==19.9.0
h5py==2.8.0
identify==1.4.7
idna==2.8
imageio==2.6.1
importlib-metadata==0.23
imutils==0.5.3
ipykernel==5.1.2
ipython==7.8.0
ipython-genutils==0.2.0
isort==4.3.21
itsdangerous==1.1.0
jedi==0.15.1
Jinja2==2.10.1
joblib==0.14.0
json5==0.8.5
jsonschema==3.1.1
jupyter-client==5.3.4
jupyter-core==4.6.0
jupyterlab==1.2.5
jupyterlab-server==1.0.6
Keras==2.3.1
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
lazy-object-proxy==1.4.3
lxml==4.4.1
lz4==2.2.1
Mako==1.1.0
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==3.1.1
mccabe==0.6.1
mistune==0.8.4
monotonic==1.5
more-itertools==7.2.0
msgpack==0.6.2
msgpack-numpy==0.4.4.3
nbconvert==5.6.0
nbformat==4.4.0
networkx==2.4
nodeenv==1.3.3
noise==1.2.2
notebook==6.0.1
numpy==1.16.4
opencv-python==4.1.0.25
opt-einsum==3.1.0
p5==0.6.0
packaging==20.1
pandas==0.25.3
pandocfilters==1.4.2
parso==0.5.1
pathspec==0.6.0
pbr==5.4.4
pexpect==4.7.0
phe==1.4.0
pickleshare==0.7.5
Pillow==6.0.0
pluggy==0.13.1
pre-commit==1.20.0
prometheus-client==0.7.1
prompt-toolkit==2.0.10
protobuf==3.11.2
psutil==5.6.3
psycopg2==2.8.3
psycopg2-binary==2.8.4
ptyprocess==0.6.0
py==1.8.1
pycairo==1.18.1
pycparser==2.19
Pygments==2.4.2
PyGObject==3.32.2
PyInquirer==1.0.3
pylint==2.4.3
pyOpenSSL==19.0.0
pyparsing==2.4.2
pyrsistent==0.15.4
pytest==5.3.4
python-dateutil==2.8.0
python-editor==1.0.4
python-engineio==3.9.3
python-osc==1.7.2
python-socketio==4.3.1
pytools==2019.1.1
pytz==2019.2
PyWavelets==1.1.1
PyYAML==5.1.2
pyzmq==18.1.0
redis==3.3.11
regex==2019.8.19
requests==2.22.0
requests-toolbelt==0.9.1
scikit-image==0.16.2
scikit-learn==0.21.3
scipy==1.4.1
selenium==3.141.0
Send2Trash==1.5.0
six==1.11.0
soupsieve==1.9.4
SQLAlchemy==1.3.13
sqlparse==0.3.0
svgpathtools==1.3.3
svgwrite==1.3.1
syft==0.2.2a1
syft-proto==0.1.1a1.post2
tabulate==0.8.5
tblib==1.4.0
tensorboard==1.15.0
tensorflow==1.15.0
tensorflow-estimator==1.15.1
tensorflow-hub==0.5.0
tensorflowjs==1.3.2
tensorpack==0.9.8
termcolor==1.1.0
terminado==0.8.2
testpath==0.4.2
tf-encrypted==0.5.9
toml==0.10.0
torch==1.4.0
torchvision==0.5.0
tornado==6.0.3
tqdm==4.36.1
traitlets==4.3.3
triangle==20190115.3
typed-ast==1.4.0
urllib3==1.25.6
virtualenv==16.7.7
vispy==0.6.1
wcwidth==0.1.7
webencodings==0.5.1
websocket-client==0.56.0
websockets==8.0.2
Werkzeug==0.15.3
whitenoise==4.1.4
wrapt==1.11.2
youtube-dl==2020.1.24
zipp==0.6.0
zstd==1.4.3.2

[fermat97]

Hi @IonesioJunior I cannot also find gateway.py and websocket_app.py scripts. Some of the attributes have also changed or something such as WebsocketGridClient.

[github-actions[bot]]

This issue has been marked stale because it has been open 30 days with no activity. Leave a comment or remove the stale label to unmark it. Otherwise, this will be closed in 7 days.