How to get a list of resources?

[Qqwy]

Hello!

I'm trying to create a simple wrapper in Node.js that people might access from over the internet to read data from the Twister network (so only the 'get' part, not the 'set'), to make, for instance, an embeddable Twister Updates widget that shows the last posts from a certain user.

What I am having trouble with, is what to do when calling a function like Twister.getUser(name).doLatestPostsUntil(), I end up with a recursive callback structure. It gets very hard to create a list from this to return to the browser. The basic example works: I push() the post to an array, and when I'm at the final post I want, I return the contents of the array to the user.

But what if I want to append the URL of the user's avatar in the results as well? I end up with recursive function calls that are very hard to understand and control.

What am I doing wrong?

Sincerely,

~Qqwy

[Qqwy]

Somewhat related to this: I'd like to do pagination, e.g. return only a subset of a user's posts with each request. Twister.getUser(name).doLatestPostsUntil() does not have an option to specify a starting point.

[Tschaul]

I plan to be compatible with https://github.com/yortus/asyncawait to avoid some of these problems. But in the end that is just syntactic sugar. I can not say when this will be implemented, i haven't looked how much of the code has to change. Also i don't know how that would work at all with nested for loops...

I found, the best way to get to readable code is slice everything down into small functions, so that you never have more than two layers of callback in one function.

The biggest problem i see for you, however, is how to know when the response object, you are pushing all the results on, is complete (all queries are done). The only thing that comes to my mind is to increment an integer for every query you issue and decrement it in the callback of the query. But then you have to check in the callback of every query whether this was the last one... ugly... It is probably best to abstract that out into a class.

For the starting point of the doLatestPostsUntil function you can just put an if-clause into the callback. The cache will deal with the performance:

Twister.getUser(name).doLatestPostsUntil(function(post){
  if(post.getId()<=yourStartingPoint){
    //do what you want and return false at some point
  }
});

Sorry i can't be of more help.

[Qqwy]

Thank you for your reply!

Right now, I get posts using the Twister.RPC('getposts',[{username:"someone",maxid:"something"}]) to get the proper posts, but I'm guessing that direct RPC requests are not cached.

As for checking in the callback of every query if it was the last one: One other problem I stumbled upon when doing that, is what happens when there are no more posts (e.g. I want 10 posts but there are only five).

As for the function slicing: I absolutely agree with you. :-)

Thank you very much for your reply!

[Tschaul]

No in this case there is no caching. You can detect the first post of a stream as the only post that doesn't have a value for lastk (post.getLastId() is falsy).

[Qqwy]

I tried incorporating your suggestions. However, right now I often get error 32052: DHT resource is empty. Twister-lib-js passes this on to it's default error function. However, this does mean that the .doLatestPostsUntil loop gets completely broken. I tried calling .doLatestPostsUntil with my own error function, but it doesn't seem to overwrite the default function properly:

var posts = [];
Twister.getUser(name).doLatestPostsUntil(function(post){

    posts.push(post);
    if(last_k === undefined){ //Starting point. Set in GET request or otherwise the ID of the highest known post. We stop 5 posts before this post.
        last_k = post.getId();
    }

  if(post.getId()<=last_k-5 || post.getLastId() == false){
    res.jsonp(parsePosts(posts)); //Parses the posts, returning only the data we need in the front-end, and sends them in JSONP format. ('res' is the node express request result object.)
    return false;
  }
},{errorfunc:function(error){ //This gets never called.
             console.log("Twister_js encountered an error: "+error.code+": "+error.message); res.jsonp(parsePosts(posts)); 
          }
});

I think this might be a bug.

[Tschaul]

Thanks, yes this is a bug! I pushed a fix. Should also be on npm. haven't had time to test it properly though. If new things break go back to 0.1.1. The new version is 0.2.0 because there are new features. I will write the changelog later.

[Tschaul]

I added the house keeping functionality of active simultaneous queries to the lib. Now you can register a handler that trigger when all queries are finished that are marked with the same queryId. See the new example:

https://github.com/Tschaul/twister-lib-js/blob/master/examples/021_deep_fetching.js