habit history condenses current day's data

[Alys]

I'm not sure if this is a bug or a feature that is not well understood. If the latter, tell me and I'll document it on the wiki.

For habits, the exported history data (in userdata.xml and userdata.json and history.csv) contains only one value per day, rather than a full record of every time the plus or minus button was clicked. I know that the code has the preenHistory function to condense older history entries, but maybe it's condensing the current day's entries too? Is it meant to do that? An uncondensed history of recent habit clicks would help people remember whether they'd correctly recorded their habits over the past few hours.

Below is some test data that shows multiple clicks being done over two days (today and yesterday) for one habit, but only two history entries being reported (one for each day). I've recorded the time of each click in my local timezone, GMT+10. After each click I downloaded the json file and extracted the data for the habit and for the user's stats. While I was doing these tests, I did nothing else with this account, so all the stats changes are from the habit. The task's value and the XP and HP stats changed as expected after each click, so the clicks were definitely being received by the server. The account's userid is 3e595299-3d8a-4a10-bfe0-88f555e4aa0c (my work account).

---

14-03-15 13:23:20

create habit called "habit test"

{"text":"habit test","challenge":{},"attribute":"str","priority":1,"value":0,"tags":{},"notes":"","dateCreated":"2014-03-15T03:23:08.347Z","id":"aab707ee-a0e0-426a-b571-9a5daa5913f7","down":true,"up":true,"history":[],"type":"habit"} "stats":{"training":{"con":0,"str":0,"per":0,"int":0},"buffs":{"snowball":false,"streaks":false,"stealth":0,"con":0,"per":0,"int":0,"str":0},"per":5,"int":53,"con":0,"str":0,"points":0,"class":"wizard","lvl":58,"gp":840.7309718617495,"exp":54,"mp":126.91000000000007,"hp":46,"toNextLevel":1560,"maxHealth":50,"maxMP":193}

---

14-03-15 13:26:00

plus

{"text":"habit test","challenge":{},"attribute":"str","priority":1,"value":1,"tags":{},"notes":"","dateCreated":"2014-03-15T03:23:08.347Z","id":"aab707ee-a0e0-426a-b571-9a5daa5913f7","down":true,"up":true,"history":[{"value":1,"date":1394853962626}],"type":"habit"} "stats":{"training":{"con":0,"str":0,"per":0,"int":0},"buffs":{"snowball":false,"streaks":false,"stealth":0,"con":0,"per":0,"int":0,"str":0},"per":5,"int":53,"con":0,"str":0,"points":0,"class":"wizard","lvl":58,"gp":843.5609718617495,"exp":72,"mp":126.91000000000007,"hp":46,"toNextLevel":1560,"maxHealth":50,"maxMP":193}

---

14-03-15 13:28:02

plus

{"text":"habit test","challenge":{},"attribute":"str","priority":1,"value":1.9747,"tags":{},"notes":"","dateCreated":"2014-03-15T03:23:08.347Z","id":"aab707ee-a0e0-426a-b571-9a5daa5913f7","down":true,"up":true,"history":[{"value":1.9747,"date":1394853962626}],"type":"habit"} "stats":{"training":{"con":0,"str":0,"per":0,"int":0},"buffs":{"snowball":false,"streaks":false,"stealth":0,"con":0,"per":0,"int":0,"str":0},"per":5,"int":53,"con":0,"str":0,"points":0,"class":"wizard","lvl":58,"gp":846.3193728617496,"exp":90,"mp":126.91000000000007,"hp":46,"toNextLevel":1560,"maxHealth":50,"maxMP":193}

---

14-03-15 13:29:40

minus

{"text":"habit test","challenge":{},"attribute":"str","priority":1,"value":1.024043774264157,"tags":{},"notes":"","dateCreated":"2014-03-15T03:23:08.347Z","id":"aab707ee-a0e0-426a-b571-9a5daa5913f7","down":true,"up":true,"history":[{"value":1.024043774264157,"date":1394853962626}],"type":"habit"} "stats":{"training":{"con":0,"str":0,"per":0,"int":0},"buffs":{"snowball":false,"streaks":false,"stealth":0,"con":0,"per":0,"int":0,"str":0},"per":5,"int":53,"con":0,"str":0,"points":0,"class":"wizard","lvl":58,"gp":846.3193728617496,"exp":90,"mp":126.91000000000007,"hp":44.4,"toNextLevel":1560,"maxHealth":50,"maxMP":193}

---

14-03-15 13:47:23

plus

{"text":"habit test","challenge":{},"attribute":"str","priority":1,"value":1.998143412564751,"tags":{},"notes":"","dateCreated":"2014-03-15T03:23:08.347Z","id":"aab707ee-a0e0-426a-b571-9a5daa5913f7","down":true,"up":true,"history":[{"value":1.998143412564751,"date":1394853962626}],"type":"habit"} "stats":{"training":{"con":0,"str":0,"per":0,"int":0},"buffs":{"snowball":false,"streaks":false,"stealth":0,"con":0,"per":0,"int":0,"str":0},"per":5,"int":53,"con":0,"str":0,"points":0,"class":"wizard","lvl":58,"gp":849.0760748381402,"exp":108,"mp":126.91000000000007,"hp":44.4,"toNextLevel":1560,"maxHealth":50,"maxMP":193}

---

14-03-15 14:05:27

plus

{"text":"habit test","challenge":{},"attribute":"str","priority":1,"value":2.9482287028135716,"tags":{},"notes":"","dateCreated":"2014-03-15T03:23:08.347Z","id":"aab707ee-a0e0-426a-b571-9a5daa5913f7","down":true,"up":true,"history":[{"value":2.9482287028135716,"date":1394853962626}],"type":"habit"} "stats":{"training":{"con":0,"str":0,"per":0,"int":0},"buffs":{"snowball":false,"streaks":false,"stealth":0,"con":0,"per":0,"int":0,"str":0},"per":5,"int":53,"con":0,"str":0,"points":0,"class":"wizard","lvl":58,"gp":851.7648162095444,"exp":125,"mp":126.91000000000007,"hp":44.4,"toNextLevel":1560,"maxHealth":50,"maxMP":193}

---

14-03-15 15:51:52

minus

{"text":"habit test","challenge":{},"attribute":"str","priority":1,"value":2.020995311995297,"tags":{},"notes":"","dateCreated":"2014-03-15T03:23:08.347Z","id":"aab707ee-a0e0-426a-b571-9a5daa5913f7","down":true,"up":true,"history":[{"value":2.020995311995297,"date":1394853962626}],"type":"habit"} "stats":{"training":{"con":0,"str":0,"per":0,"int":0},"buffs":{"snowball":false,"streaks":false,"stealth":0,"con":0,"per":0,"int":0,"str":0},"per":5,"int":53,"con":0,"str":0,"points":0,"class":"wizard","lvl":58,"gp":851.7648162095444,"exp":125,"mp":126.91000000000007,"hp":42.8,"toNextLevel":1560,"maxHealth":50,"maxMP":193}

---

after cron (this account received rewards from a quest, but I used "fix character values" to subtract the rewarded xp and gp)

{"text":"habit test","challenge":{},"attribute":"str","priority":1,"value":2.020995311995297,"tags":{},"notes":"","dateCreated":"2014-03-15T03:23:08.347Z","id":"aab707ee-a0e0-426a-b571-9a5daa5913f7","down":true,"up":true,"history":[{"value":2.020995311995297,"date":1394853962626}],"type":"habit"} "stats":{"training":{"con":0,"str":0,"per":0,"int":0},"buffs":{"snowball":false,"streaks":false,"stealth":0,"con":0,"per":0,"int":0,"str":0},"per":5,"int":53,"con":0,"str":0,"points":0,"class":"wizard","lvl":58,"gp":851.7648162095444,"exp":125,"mp":146.21000000000006,"hp":32.76950447525947,"toNextLevel":1560,"maxHealth":50,"maxMP":193}

---

14-03-16 11:34:38

plus

{"text":"habit test","challenge":{},"attribute":"str","priority":1,"value":2.9705244023522885,"tags":{},"notes":"","dateCreated":"2014-03-15T03:23:08.347Z","id":"aab707ee-a0e0-426a-b571-9a5daa5913f7","down":true,"up":true,"history":[{"value":2.020995311995297,"date":1394853962626},{"value":2.9705244023522885,"date":1394933687085}],"type":"habit"} "stats":{"training":{"con":0,"str":0,"per":0,"int":0},"buffs":{"snowball":false,"streaks":false,"stealth":0,"con":0,"per":0,"int":0,"str":0},"per":5,"int":53,"con":0,"str":0,"points":0,"class":"wizard","lvl":58,"gp":854.4519835352547,"exp":142,"mp":146.21000000000006,"hp":32.76950447525947,"toNextLevel":1560,"maxHealth":50,"maxMP":193}

---

14-03-16 11:40:24

minus

{"text":"habit test","challenge":{},"attribute":"str","priority":1,"value":2.0438206252967634,"tags":{},"notes":"","dateCreated":"2014-03-15T03:23:08.347Z","id":"aab707ee-a0e0-426a-b571-9a5daa5913f7","down":true,"up":true,"history":[{"value":2.020995311995297,"date":1394853962626},{"value":2.0438206252967634,"date":1394933687085}],"type":"habit"} "stats":{"training":{"con":0,"str":0,"per":0,"int":0},"buffs":{"snowball":false,"streaks":false,"stealth":0,"con":0,"per":0,"int":0,"str":0},"per":5,"int":53,"con":0,"str":0,"points":0,"class":"wizard","lvl":58,"gp":854.4519835352547,"exp":142,"mp":146.21000000000006,"hp":31.169504475259465,"toNextLevel":1560,"maxHealth":50,"maxMP":193}

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[goldfndr]

By design? https://github.com/HabitRPG/habitrpg-shared/blob/e788872932640310427f468d2ec6afa19d08d842/script/index.coffee#L979 (through L982) compares the last habit tick date (if any) to "today", and if so then only the value is changed.

[jakechen]

In relation to the issue your data display tool (which links here), the issue stems from timezones. Preening combines dates based on GMT, whereas most displays will show local. This explains why your data display tool will occasionally display 2 records for the same day. For example from your data display tool:

vs from data export:

As you can see, my activity on the 17th at 6pm CST was used for the first record in the history on the 18th. This means that the preening function in general, while definitely necessary, may need to be revisited since it will incorrectly record activities in the local timezone. Not sure what priority this is though, and could be potentially quite large.

[Alys]

I would like to see the preening function not merge any datapoints that have been made in the previous couple of days. That would allow people to see the times of all of their clicks for today and yesterday, which is a common request (e.g., to see if they remembered to record a particular instance of a Habit). The downside is that more database space would be required.

If preening can be modified to not merge datapoints for today and yesterday, then the effect of timezone on preening becomes less important - perhaps unimportant enough that we can ignore it.

[jakechen]

It seems to me that task.value is never tracked more than once a day, instead the task.value is incremented every time score() is called:

if th[th.length-1] and moment(th[th.length-1].date).isSame(new Date, 'day')
          th[th.length-1].value = task.value

(https://github.com/HabitRPG/habitrpg-shared/blob/develop/script/index.coffee#L1078)

As such I think the core issue lies outside of preenHistory().

In order to track a habit's daily occurrences then the functionality of score() needs to be modified not to increment, but to add a new timestamp. Then preenHistory() may also have to be modified since the resolution of times will be different (i.e. history of events, not days).

[paglias]

is this still an issue?

[Alys]

Ah, I haven't used Habits for weeks, so I have no data to look at.

As I understand it from comments others have made, EVERY click you make on a Habit today is stored. There's no preening of any kind until cron runs - is that right? If that's how it works now, then this issue is certainly resolved.

[markwk]

From what I can tell the data exports and user data json file doesn't record the actual click time. Instead the export merges all of these into a server datetime that must relate to a cron update.

This seems like massive manipulation of potentially very useful user data. For example, I'd love to run a report on when I check-in my various habits. Obviously the current data exports and timestamps doesn't allow for this.

Am I missing something or is this by design? Are the click times removed entirely from the database? This seems like an area where we should at least keep some of this data for a certain period as mentioned in Alys's Comment on Jan 20, 2015.

[Alys]

For anyone interested in the questions @markwk asked, they were answered in the Newbies guild where markwk posted similar questions. Copying the messages here for posterity:

markwk: How can I get the status of a task (daily or habit, for example) in the task history export? It only exports five values: Task Name, Task ID, Task Type, Date, Value. Where can I see an exportable log of completed tasks like my dailies?

Alys: From the website, Data > Data Display Tool will give you some history information. You can get all the same data in a machine-friendly format from Data > Export Data. What do you mean by status? Whether it's completed or not? That's in the Data > Export Data > "Export User Data" option.

markwk: I mean that the task history export doesn't record if you completed the task or not each day. Maybe I'm missing something though. All I see in my export is a line like this: ?Floss 3d63854c-f316-45f0-9c26-1254a552287e daily 2016-09-24 02:23:34 25.450464222323028 I was able to look at the json file in the user data export and it does record the current day's task completion status. But what I'm getting at is that the habit history export does not record this on a day by day basis. It seems like the habit history export represents the backend task value and not whether you checked a habit off or not.

Alys: The back-end task value is controlled by whether you mark the task completed or not: completing a task increase the value, not completing a Daily or clicking minus on a Habit decreases it (however if you cast the Searing Brightness or Brutal Smash skills on a task, it will modify the value dramatically). The Data Display Tool uses this to display the historical statuses of the tasks as best it can. If you don't cast those skills on the tasks you care about, you can use the history values to determine whether you used tasks in a positive or negative way.

markwk: Thanks. As an alternative is there an export of the raw history, i.e. past day or so's on-site actions, clicks on tasks, etc.? It seems like what I'm seeking is the raw history.

Alys: The raw history is in the "Export User Data" option.

markwk: I see. That json file does contain a lot info about today's tasks but it doesn't appears to contain any additional data beyond the current day. Is that right? Also in the history part, it doesn't appear to store the actual day's update timestamp. Instead it is the cron time. Right? As such, it seems like there is no way to get data on when you completed tasks in Habitica.

Alys: No, the history goes back longer than the current day. The dates do include a time portion. Clicks for Habits and completion for To-Dos are recorded as the exact times when you took those actions. The timestamps for Dailies are the cron times, but you mark Dailies complete at most once a day, so the exact time generally isn't relevant - if you'd like exact times for Dailies to be recorded, please ask about it at Help > Request a Feature. All of that data is stored in the "Export User Data" option. The Data Display Tool shows the same data but in a format that might be easier to understand.