provide better streak-describing data in goals data from API

[rjbs]

(Or, maybe: explain to @rjbs how the heck to get this data.)

Each goal in the goals GET API looks like this:

   "goals" : [
      {
         "active" : "1",
         "category_id" : "1718",
         "color" : "#21882B",
         "cooldown" : "5",
         "description" : "",
         "foreground" : "#000000",
         "frequency" : "1",
         "id" : "333",
         "name" : "read unread mail",
         "note_prompt" : null,
         "position" : "0",
         "public" : "1",
         "quantity" : "1",
         "quantity_needed" : 0,
         "reminders" : [],
         "require_note" : "1",
         "rgb" : [ 33, 136, 43 ],
         "start_date" : "2013-08-12T00:00:00",
         "trend" : [
            {
               "cooldown" : "5",
               "date" : "2013-08-12",
               "frequency" : "1",
               "id" : "286653",
               "needed" : "1",
               "notes" : [
                  "new count: 1"
               ],
               "quantity" : 5,
               "streak" : "16"
            },
           ....

When I look at my goals, the things of overwhelming concern are:

• is it currently safe?
• when did the streak really start?
• when will the streak end if I do nothing?

I'm not sure I can answer any of those questions from the above data!

The "start_date" doesn't tell me when the streak started. It's just the earliest date I'm seeing in the datastream. Can I rely on the "streak" value as the number of days ago that the streak began, or does it mean something else? (Especially if the thing has to be done more than once in a period, etc.)

I think I can determine whether it's safe by looking through the dates in "trend" to find today and seeing whether there is a non-zero "streak" value, but it's not clear. Also, what's "quantity" mean? I see that for days after today, it is a steadily decreasing number. Is it actually quantity of days safe? If so, can I look at today, add "quantity" and know when the streak ends?

Basically: more documentation would, of course, be great, but from what I can figure out, I wonder whether the API isn't giving badly-optimized data to begin with. So, maybe really quick, sketchy v1 API documentation is all that's needed so you can work on API v2? :) :)

[jshirley]

I'd like to document it but I keep getting caught up sword sharpening, and going, "Hmm, maybe I shouldn't use Swagger!" I really need to just document it all!

Basically, everything that gets dumped into the trend is to support the UI, in some capacity, which the quantity sort of represents.

Here's the short of it:

Quantity is the number of items impacting that day, so if you have a goal "Once every 3 days" and do it two times in a row, it will accumulate and add the number of times done for the duration of the streak, resulting in what should be a trend of quantities that decreases (as you've seen). If you don't "stack" and overlap, the quantity should be maintained through the entire streak.

In #4 I am working on basically a denormalized, pre/post-calculated record that will save these stats in a better way... not sure if that will really be API v2 or just another endpoint.

[rjbs]

What's the simple way, given the API result for a goal, to know when it expires? I assume you find the quantity on today, and then... I'm not sure. Something with the cooldown and goal quantity?

[jshirley]

The simplest way is to iterate over the trend, find the last ID, then add the number of days that the cooldown is on the goal.

This should do the trick, looks like it works for my purposes anyway: https://gist.github.com/jshirley/6312764

(Though you may need to tweak time zones)

[jshirley]

@rjbs does the above work for you until I can get #4 completed?

[rjbs]

Sorry, I haven't had a chance to test it. I will try to get to it today to make sure it matches my expectations. If so, yes. Thanks.

[rjbs]

This does not work.

I have tasks that only need to be done once per fortnight, for example. They are totally safe at the moment, so none of their trend entries has a defined id key. It seems better if I replace the defined $_->{id} check with $_->{quantity}, but I'm not sure if this is cromulent.

[rjbs]

This is no good. It counts down to cooldown days from the final safe day in the calendar. For example, one of my goals was last done ... I think August 9th. It's got a 21 day cooldown. August 9th doesn't appear anywhere in the dump of the goal. The dump I'm looking at is from Aug 18th, so I'd alternately expect to see a number like 12 in the goal. (21 day cooldown - 9 days already passed) That also does not appear.

I don't think the API is giving me enough information to answer this question.

[jshirley]

Well, it's a good thing I have all this documented :/

Add the ?range= parameter, days back/days forward.

So /v1/goals?range=31,7 for example, that will go 31 days back and that should show you what you want, just keep increasing the range until it does?

On Aug 26, 2013, at 3:19 PM, Ricardo Signes notifications@github.com wrote:

This is no good. It counts down to cooldown days from the final safe day in the calendar. For example, one of my goals was last done ... I think August 9th. It's got a 21 day cooldown. August 9th doesn't appear anywhere in the dump of the goal. The dump I'm looking at is from Aug 18th, so I'd alternately expect to see a number like 12 in the goal. (21 day cooldown - 9 days already passed) That also does not appear.

I don't think the API is giving me enough information to answer this question.

— Reply to this email directly or view it on GitHub.

[rjbs]

I will try this tomorrow. My guess is that it means I will be doing two-pass queries. The first will just GET. Then I'll find the maximum cooldown in all active goals and re-get with ?range=X,7, where X is that cooldown. This will guarantee that if a goal is active, I will have its start point. (Or maybe I need X+1? I'll sort that out in the morning.)

[jshirley]

Ah! Yes, I see that.

I just changed the range parameter to also accept "max"

http://tdp.me/v1/goals?range=max,7

The max will look at your maximum cooldown value, add one and use that. Hopefully that does the trick for you with a single query.

On Tue, Aug 27, 2013 at 8:39 PM, Ricardo Signes notifications@github.comwrote:

I will try this tomorrow. My guess is that it means I will be doing two-pass queries. The first will just GET. Then I'll find the maximum cooldown in all active goals and re-get with ?range=X,7, where X is that cooldown. This will guarantee that if a goal is active, I will have its start point. (Or maybe I need X+1? I'll sort that out in the morning.)

— Reply to this email directly or view it on GitHubhttps://github.com/jshirley/tdp/issues/9#issuecomment-23387834 .

J. Shirley :: jshirley@gmail.com :: Killing two stones with one bird. http://our.coldhardcode.com/jshirley/

[rjbs]

Either it isn't working or you didn't deploy it yet:

46670 is no good
  cooldown is: 21
  got n in trend: 8

[jshirley]

Well, I'm not very bright.

It has to be the max cooldown * 2, if that is 21 days back and it only returns the last 21 days of data, it won't include the day it was actually done. Did that, deployed, should show you what you want now.

The serialization method to generate the trend data already looks back that far to populate the trend data, but doesn't include the "trend start" in the initial payload.

I do think this is a good feature, but will eventually be a pretty costly query without having the rolling denormalization objects constructed (which can store the "chain start" date, times since, etc.)

On Aug 28, 2013, at 4:45 AM, Ricardo Signes notifications@github.com wrote:

Either it isn't working or you didn't deploy it yet:

46670 is no good cooldown is: 21 got n in trend: 8 — Reply to this email directly or view it on GitHub.

[rjbs]

46670 is no good
  cooldown is: 21
  got n in trend: 8

Does it really have to be cooldown *2?

Cooldown=21 means that a goal is safe the day it's done and for the next 20 days.. Cooldown=1 means the goal is safe only the day you do it.

So if you do a Cooldown=1 goal on the 1st, you have to do it next on the 2nd, 1st + 1.

If you do a Cooldown=21 goal on the 1st, you have to do it again on the 22nd.

So if a goal is currently alive, it must have an occurance within the last Cooldown days. If it's the 22nd and my Cooldown=21 streak is active, the longest ago it could've been done last is 22nd-21, the first.

So: either I am wrong, or you or wrong, or something else is going on (i.e., we're both wrong).

I think the problem is this:

If I fetch with ?range=max,7, I get this:

46670 is no good
  cooldown is: 21
  got n in trend: 8

If I fetch with ?range=21,7, I get this:

46670 goes pear shaped on 2013-09-22T00:00:00
  cooldown is 21
  got n in trend: 29