Add autorefresh functionality

[davidbauer]

After data has been loaded, we keep checking in the background if new links are available and display them in a similar way that Twitter says "x new tweets" in its timeline.

The process would be:

• Once a request has been executed
• Check every x seconds if there are new links available
• If so, display a message to the user: "n new links" at the top of the list and "(n)" in the browser tab
• When user clicks the message, new teasers are added

[akristofcak]

Hi David. There are 2 general ways I know of this can be implemented. The first one, which I have personally implemented on reading.ly, is detailed below. It is not as robust as the more sophisticated approach but it is easy to conceptualize and implement. The second one is using something called Comet. I personally haven't implemented this because at the time when I needed this (for Readingly) it was way over my head. But if you go on stack overflow and just google around, there are many articles and sites about it (http://en.wikipedia.org/wiki/Comet_(programming))

So back to my approach:

• in your javascript, create a time function (setInterval()) which runs every X seconds. The more often this runs, the more taxing it is on the browser and your server, so I would be careful about doing something like "every second"
• in the time function, create a call to a script on the server (in jQuery this is a simple $.get or $.post method...if you don't use jQuery, it more clunky but doable - basically you need to do an AJAX call)
• your script on the server then does the same stuff your server does for the site (i'm assuming you use the twitter API to retrieve the feed which you then parse and organize into a neat HTML) with the exception being that you want to only run this since tweet ID XYZ - you should probably capture this value somewhere in the original HTML, and then read the value when you run the AJAX call; this script should return 3 values: (1) the count of new items, (2) the contents of new items, (3) the most recent tweet ID checked
• the AJAX function in the browser receives the callback value and the callback function then processes it:
• you alter the title of the page (again using jQuery) to reflect the new item count
• you display a notification on the top of the page (with a count) which when clicked --
• displays the contents of the new items

Now, I know this is a little dense and I may not be explaining it perfectly but let me know if this makes sense to you conceptually and if you're having trouble with any steps.

[davidbauer]

Ok, so this is what I've got so far:

• After fetching the links for the first time, I do this to check for new content every minute: setInterval(autorefresh(searchApiMaxId), 60000);
• I have an autorefresh function which triggers the link fetching again, but starting from the id of the last tweet I've already fetched.

Since the setInterval function is called inside the fetching function, it will be called again when autorefresh is called after 60 seconds. Is that the point? Or should I rather use .delay instead?

[akristofcak]

You could set up the fetchingFunction() to entail the auto-loop mechanism but that's an unnecessary hack; setInterval just does this for you.

It could be as simple as

setInterval(function(){fetchingFunction()},60000);

setInterval will then run fetchingFunction() every 60 seconds after the page loads. Note how you have to wrap the function name in the function(){} expression. I wish I understood why setInterval requires this, but it does.

In your fetchingFunction() you should read the mostRecentTweetID from an existing element in your HTML (I would save this as an attribute in an element somewhere when you're generating the first page), pass that ID onto the server in the AJAX request and then update the mostRecentTweetID based on the data fetched; when setInterval triggers the fetchingFunction() again in 60 seconds, it will read the updated mostRecentTweetID and pass that onto the server in the AJAX request... etc etc etc.

[davidbauer]

Thanks, will try that. Looks like I need to pass the tweetId as a parameter to the fetchingFunction, which I currently don't (since it's always starting from the first), so that I can trigger a different behaviour for the autorefresh (display "n new links" etc).

[davidbauer]

I've started working on the autorefresh functionality, see commit above. Could anyone of you have a look at it and tell me if that's any good so far? One of the problems I've already noticed is that the setInterval function doesn't know tweetId, which it needs as a parameter to pass along to getLinks()

[akristofcak]

I would have fetchingFunction look up tweetID from a place where you "stored" it when you generated the initial load of the page. For example:

var tweetID = $('#wrapper').attr('tweetID');

Here I am assuming that in your div with id wrapper you have an attribute called tweetID where you stored the value of the most recent tweetID.

Then, when you fetch the most recent data, and figure out the most recent tweetID, you need to reset this value to the new tweetID:

$('#wrapper').attr('tweetID',newtweetID);

When fetchingFunction runs next time, it will read this updated value.

[davidbauer]

I have tweetId stored it in the script, so I actually don't need to look it up. Just – like always – a scope issue with my functions. @dergraf, @backflip to the rescue!

[backflip]

tweetId is declared inside process_data() (using var). Instead, you could declare it at the same time as since_id (https://github.com/davidbauer/Instacurate/blob/master/script.js#L225).

[davidbauer]

@backflip Thanks, added var tweetId;. I don't need to set the variable to a value up there, or do I? And do I need to pass it to process_data() or return it from it? This stuff still gives me headaches ;-)

[backflip]

You don't have to declare a value, by default it is undefined. process_data() has access to this variable, so you don't need to pass it as a parameter.

[davidbauer]

Thanks, @backflip. So I'll proceed with the implementation from here. The alert I've put in as a placeholder seems to work correctly.

[davidbauer]

I think I've now implemented everything I need for autorefresh to work as described. However – as always ;-) – it won't work. @backflip, do you have time to have a quick look at what's wrong?

[backflip]

I guess generateEmbed is not called anymore: https://github.com/davidbauer/Instacurate/blob/master/script.js#L370

EDIT: Nope, not the issue.

[backflip]

Hm, it currently works for me. I get a 500 on http://tlinkstimeline.appspot.com/statuses/home_timeline.json every now and then, but apart from that, it works fine. Can you elaborate on "it won't work" a bit? :)

[davidbauer]

All the interactions in the frontend seem to work, but the data processing doesn't seem to be correct.

Once the first autorefresh runs, it shows that a number of new links is found, but this isn't the actual number of new links in the timeline since the first load. Also, if I click on the notification, no new teasers are loaded.

I suspect that the reloading of teasers when scrolling down the page interferes with the autorefresh. I've also noticed that reloaded via scrolling doesn't work once autorefresh has run.

Do you see a different behaviour, @backflip?

[oliverwehn]

I just cloned the recent version of Instacurate. I’ll look into this. Shouldn’t be that hard to get auto-refresh working.

[dergraf]

thanks @oliverwehn, I had this on my todos as well, but don't have much time to contribute right now. let me know if you need some help.

[davidbauer]

I just cloned the recent version of Instacurate. I’ll look into this. Shouldn’t be that hard to get auto-refresh working.

Looking forward to that X-mas present ;-)

[oliverwehn]

Hmm, I get error 500 responses from the appspot url on almost every request. Any idea why? It’s hard to work on auto-refreshing without having anything to process. ;)

[davidbauer]

@dergraf knows best about this part of Instacurate...

[dergraf]

I had this as well. It is caused by an TwitterAPI request error that we don't handle correctly. So far I couldn't figure out why it isn' working in some cases. Am 19.12.2013 22:50 schrieb "David Bauer" notifications@github.com:

@dergraf https://github.com/dergraf knows best about this part of Instacurate...

— Reply to this email directly or view it on GitHubhttps://github.com/davidbauer/Instacurate/issues/63#issuecomment-30969913 .

[oliverwehn]

@dergraf, is the appspot thing a Python app you wrote? I somehow missed this part. Is there a separate repo for it?

@davidbauer, as the auto-refresh has to add the new links to the top, it would be great to overcome the fixed three column structure of the document. Otherwise we would have to remove all displayed stories and put them back to the columns in updated order. Would be much easier and faster to just let the prepended content push the other stuff down the timeline.

[dergraf]

Yes, it is a public repo https://github.com/dergraf/gae-twittertimes

2013/12/20 Oliver Wehn notifications@github.com

@dergraf https://github.com/dergraf, is the appspot thing a Python app you wrote? I somehow missed this part. Is there a separate repo for it?

@davidbauer https://github.com/davidbauer, as the auto-refresh has to add the new links to the top, it would be great to overcome the fixed three column structure of the document. Otherwise we would have to remove all displayed stories and put them back to the columns in updated order. Would be much easier and faster to just let the prepended content push the other stuff down the timeline.

— Reply to this email directly or view it on GitHubhttps://github.com/davidbauer/Instacurate/issues/63#issuecomment-30998124 .

[davidbauer]

@oliverwehn Would that have any design implications or are you merely talking about the structure of the html?

[oliverwehn]

Hmm, I guess there wouldn’t be a way around design adaptations und js-layouting to achieve a similar look. I’ll try to work something out.

[davidbauer]

@oliverwehn, you're the designer, I'm very open to your suggestions.

[davidbauer]

Any news here, @oliverwehn?

[oliverwehn]

Still working on it. Got some issues dealing with the automatic updates. And the Twitter API error from the Python script often keeps me from testing. Stuff I have to solve before adding jquery.isotope. I’ll push the current state to my github repo tonight (if I don’t forget).

[oliverwehn]

Ok, it has been hard with all the Python thing errors, but I pushed a first rough version with auto-refresh. The usability has to be optimized by just showing a notice to tell of new tweets with a link to actually prepend them to the current timeline. Haven’t tested it in detail, yet. The HTML structure of index.html has been changed but I haven’t been able to change the styles and to implement Isotope. So it still looks pretty shitty. Nevertheless auto-refreshing is on its way.

Have a look here: https://github.com/oliverwehn/Twitter-Times/tree/dev-autorefresh

[davidbauer]

@dergraf can you have a look at the python errors? i'm totally lost there.

[dergraf]

these errors are caused by the twitter rate limits. twitter counts the requests made per API endpoint and user and applies a time window on that counter. E.g. the /home_timeline endpoint can be called 15 times within 15 minutes by the same user. Once the 15 requests are reached Twitter responses with a HTTP Error status which is currently not properly handled by the python app and therefore causes the crash. You can find out more about the rate limits here.

I will fix the crash as soon as possible, but this is not the root of the problem, the twitter rate limitation is.