search

e-mission / e-mission-docs

Repository for docs and issues. If you need help, please file an issue here. Public conversations are better for open source projects than private email.
https://e-mission.readthedocs.io/en/latest
BSD 3-Clause "New" or "Revised" License
15 stars 34 forks source link

cfc_webapp /result/metrics/local_data json_decoding issue #530

Closed njriasan closed 4 years ago

njriasan commented 4 years ago

When running a post request for the metrics the server is unable to return any results do the inability to properly encode numpy values in json. It just seems to require extending the custom json_encoder used for mongo types to also include numpy types as suggested here: https://stackoverflow.com/questions/50916422/python-typeerror-object-of-type-int64-is-not-json-serializable/50916741. I will submit a pull request with the change.

shankari commented 4 years ago

I thought I fixed this in the main repo. The metrics show up just fine in the many UIs. Can you pull from master and see if it fixes the problem?

shankari commented 4 years ago

@njriasan if the pull from the main repo fixed things, can you close this issue?

njriasan commented 4 years ago

I tried this again from a fresh download of the e-mission-server. I still get the issue but if it shows up fine in the many UIs I assume this is an issue with my script used to access the metrics.

shankari commented 4 years ago

can you post a concrete MCVE (https://stackoverflow.com/help/mcve)? Maybe it is in a code path that the UI doesn't touch.

njriasan commented 4 years ago

Ya sure. Let me know if anything I put is unclear.

Setup:

  1. Make sure mongodb is running in a terminal 
    $ mongod
  2. Launch the activate the emission conda environment in two terminals.
  3. If starting from a fresh server clone make sure you have a seed_model.json so you can run the pipeline 
    $ cp emission/tests/data/seed_model_from_test_data.json seed_model.json
  4. Run the first example for running the pipeline in one terminal. 
    $ ./e-mission-py.bash bin/debug/load_timeline_for_day_and_user.py emission/tests/data/real_examples/shankari_2015-07-22 test_july_22
$ ./e-mission-py.bash bin/debug/intake_single_user.py -e test_july_22
  5. Launch the e-mission server 
    $ ./e-mission-py.bash emission/net/api/cfc_webapp.py

    Example Code:

Here is the example code to run in another terminal. For simplicity I've hardcoded anything that might be passed in from terminal so it can be run as a standalone script.

import requests
import json

json_dict = dict()
json_dict['user'] = "test_july_22"
json_dict['start_time'] = { 
                    "year": 2015,
                    "month": 7,
                    "day": 22, 
                    "timezone": "America/Los_Angeles"
                }
json_dict['end_time'] = { 
                    "year": 2015,
                    "month": 7,
                    "day": 23, 
                    "timezone": "America/Los_Angeles"
                }
json_dict['freq'] = "DAILY"
json_dict['metric_list'] = ["distance"]
json_dict['is_return_aggregate'] = False
r = requests.post("http://localhost:8080/result/metrics/local_date", json=json_dict)
print(r.json())

Client Output:

Traceback (most recent call last):
  File "metric-query.py", line 22, in <module>
    print(r.json())
  File "/home/nicholas/.local/lib/python3.6/site-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/home/nicholas/anaconda3/envs/emission/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/home/nicholas/anaconda3/envs/emission/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/nicholas/anaconda3/envs/emission/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 5 (char 5)

Server Output:

START 2020-05-02 20:10:07.947642 POST /result/metrics/local_date
END 2020-05-02 20:10:07.966751 POST /result/metrics/local_date c95491b1-a02e-459d-b785-9d3bbb3d67e9 0.019029855728149414 
Traceback (most recent call last):
  File "/home/nicholas/Documents/Research/sp20/server-test/emission/net/api/bottle.py", line 997, in _handle
    out = route.call(**args)
  File "/home/nicholas/Documents/Research/sp20/server-test/emission/net/api/bottle.py", line 2004, in wrapper
    json_response = dumps(rv)
  File "/home/nicholas/anaconda3/envs/emission/lib/python3.6/site-packages/bson/json_util.py", line 403, in dumps
    return json.dumps(_json_convert(obj, json_options), *args, **kwargs)
  File "/home/nicholas/anaconda3/envs/emission/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/home/nicholas/anaconda3/envs/emission/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/nicholas/anaconda3/envs/emission/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/nicholas/anaconda3/envs/emission/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'int64' is not JSON serializable
/home/nicholas/anaconda3/envs/emission/lib/python3.6/sre_parse.py:864: DeprecationWarning: Flags not at the start of the expression \{\{((?:(?mx)(       (truncated)
  p = _parse_sub(source, pattern, True, False)
shankari commented 4 years ago

Here's what I see when I try to reproduce this:

With no data loaded:

On client

Traceback (most recent call last):
  File "/tmp/not_working.py", line 22, in <module>
    print(r.json())
  File "/Users/shankari/miniconda/envs/emission/lib/python3.6/site-packages/requests/models.py", line 885, in json
    return complexjson.loads(self.text, **kwargs)
  File "/Users/shankari/miniconda/envs/emission/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/Users/shankari/miniconda/envs/emission/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/shankari/miniconda/envs/emission/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 5 (char 5)

On server (no error)

START 2020-05-03 15:35:10.546745 POST /result/metrics/local_date
END 2020-05-03 15:35:10.552173 POST /result/metrics/local_date None 0.00532984733581543
/Users/shankari/miniconda/envs/emission/lib/python3.6/sre_parse.py:864: DeprecationWarning: Flags not at the start of the expression \{\{((?:(?mx)(       (truncated)
  p = _parse_sub(source, pattern, True, False)

With data loaded, pipeline not run

On client (no error)

{'user_metrics': [[]]}

On server (no error)

START 2020-05-03 15:43:42.613099 POST /result/metrics/local_date
END 2020-05-03 15:43:42.639936 POST /result/metrics/local_date 3eb75e82-8701-4c8a-8afa-47fb1fce65e3 0.026766061782836914

Aha! With data loaded and after pipeline run

On client (error)

    ...
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 5 (char 5)

On server (error)

END 2020-05-03 16:01:13.215529 POST /result/metrics/local_date 3eb75e82-8701-4c8a-8afa-47fb1fce65e3 0.03864717483520508
Traceback (most recent call last):
  File "/Users/shankari/e-mission/e-mission-server/emission/net/api/bottle.py", line 997, in _handle
    out = route.call(**args)
  File "/Users/shankari/e-mission/e-mission-server/emission/net/api/bottle.py", line 2004, in wrapper
    json_response = dumps(rv)
  File "/Users/shankari/miniconda/envs/emission/lib/python3.6/site-packages/bson/json_util.py", line 403, in dumps
    return json.dumps(_json_convert(obj, json_options), *args, **kwargs)
  File "/Users/shankari/miniconda/envs/emission/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/Users/shankari/miniconda/envs/emission/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/shankari/miniconda/envs/emission/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/Users/shankari/miniconda/envs/emission/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'int64' is not JSON serializable

So this is definitely reproducible.

shankari commented 4 years ago

@njriasan this appears to be related to the local date code path ONLY. The metrics screen uses the timestamp call by default, which is why we haven't run into this in the UI.

If I use the following MVCE

import requests
import json
import arrow

json_dict = dict()
json_dict['user'] = "test_july_22"
json_dict['start_time'] = arrow.get(2015, 7, 22, tzinfo="America/Los_Angeles").timestamp
json_dict['end_time'] = arrow.get(2015, 7, 23, tzinfo="America/Los_Angeles").timestamp
print(json_dict)
json_dict['freq'] = "D"
json_dict['metric_list'] = ["distance"]
json_dict['is_return_aggregate'] = False
r = requests.post("http://localhost:8080/result/metrics/timestamp", json=json_dict)
print(r.json())

I get the following output

{'user_metrics': [[{'ts': 1437523200, 'local_dt': {'year': 2015, 'month': 7, 'day': 22, 'hour': 0, 'minute': 0, 'second': 0, 'weekday': 2, 'timezone': 'UTC'}, 'fmt_time': '2015-07-22T00:00:00+00:00', 'nUsers': 1, 'WALKING': 6295.9773474392405, 'BICYCLING': 11584.339997579958, 'CAR': 33749.95118568352}]]}
shankari commented 4 years ago

ah wait, somebody did test the local date option. I just didn't focus on it until now. This is a duplicate of https://github.com/e-mission/e-mission-docs/issues/431

shankari commented 4 years ago

Ah, here's the prior fix for the timestamp case https://github.com/e-mission/e-mission-server/commit/88fef81582d7706d22525d0beefcba4149441f0a

now we need to see why that doesn't work for local date.