After a few minutes, fetcher.py crashes

[tovabbi]

After producing output for a few minutes, fetcher.py crashes with this error:

Traceback (most recent call last):
  File "fetcher.py", line 175, in <module>
    f.start_request()
  File "fetcher.py", line 124, in start_request
    user_data.append(json.dumps(item["overlay"][uid]["p"]))
KeyError: 'p'

[lkwatson]

I've experienced this as well. Changing the line

user_data.append(json.dumps(item["overlay"][uid]["p"]))

to

user_data.append(json.dumps(item["overlay"][uid]))

Works fine, at least from what I can see.

[tovabbi]

fetcher.py is not crashing this way indeed, but graph.py does later

Traceback (most recent call last):
  File "graph.py", line 64, in <module>
    g.generate_all_csvs(start_time=now - 3 * ONE_DAY_SECONDS, end_time=now)
  File "graph.py", line 56, in generate_all_csvs
    self.to_csv(uid, start_time, end_time)
  File "graph.py", line 27, in to_csv
    status_history = history.StatusHistory(uid)
  File "/opt/zzzzz/history.py", line 21, in __init__
    self.activity = self.parse_status(map(str.strip, f.readlines()))
  File "/opt/zzzzz/history.py", line 45, in parse_status
    status_obj = status.Status(int(float(time)), fields)
  File "/opt/zzzzz/status.py", line 41, in __init__
    self._status[status] = self.value_map[fields[status]]
KeyError: 'status'

[bartmin]

I noticed this issue too, and added a line: if "p" in item["overlay"][uid]: before: user_data.append(json.dumps(item["overlay"][uid]["p"]))

(and of course indents are added where necessary - it's Python :) )

EDIT: I don't know yet if it crashes graph.py later.

[tovabbi]

@bartmin So far, this seems good. Thanks, I keep testing it.

[lkwatson]

I tried this, but upon running graph.py I got the following error

Traceback (most recent call last):
  File "graph.py", line 64, in <module>
    g.generate_all_csvs(start_time=now - 3 * ONE_DAY_SECONDS, end_time=now)
  File "graph.py", line 56, in generate_all_csvs
    self.to_csv(uid, start_time, end_time)
  File "graph.py", line 27, in to_csv
    status_history = history.StatusHistory(uid)
  File "/home/chip/zzzzz/history.py", line 19, in __init__
    self.activity = self.parse_status(map(str.strip, f.readlines()))
  File "/home/chip/zzzzz/history.py", line 35, in parse_status
    time, fields = line.split("|")
ValueError: need more than 1 value to unpack

Which seems to indicate a log file was created for an id, but no data was put in it. It appears that in the code block:

with open("log/{uid}.txt".format(uid=uid), "a") as f:
     user_data = []
     user_data.append(str(time.time()))
     if "p" in item["overlay"][uid]:
         user_data.append(json.dumps(item["overlay"][uid]["p"]))
     f.write("|".join(user_data))
     f.write("\n")

The file would be saved, but since "p" didn't exists, nothing got written. Moving that if statement above the first line of this block solved this for me (so far, at least). Like so:

if "p" in item["overlay"][uid]:
    with open("log/{uid}.txt".format(uid=uid), "a") as f:
        user_data = []
        user_data.append(str(time.time()))
        user_data.append(json.dumps(item["overlay"][uid]["p"]))
        f.write("|".join(user_data))
        f.write("\n")

[bartmin]

@lkwatson you're right, this "if" should switch on and off the whole block.