Continue - Githubissues

elifoster commented 8 years ago

The continue parameter needs to be utilized, somehow. It should check if the continue parameter exists, and if it does, continue to perform queries.

elifoster commented 8 years ago

I think a good way to do this would actually involve a major refactor of the existing query stuff.

First, we'd have to make a new query method, which takes the params (action: 'query' would come default obviously), as well as a new method argument on all continueable queries continue with the default value being false.

Then, if continue is true, loop until the continue parameter is not present.

Alternatively, we could use an integer argument and do int.times do, which would default to 1.

@xbony2 What do you think?

xbony2 commented 8 years ago

An angel told me this should be worked on.

elifoster commented 8 years ago

I still don't know what you think about my ideas for it. I'll work on it once we have a set design, and the limit improvements get finalized.

xbony2 commented 8 years ago

params = {
            "action": "query",
            "list": "categorymembers",
            "cmtitle": "Category:" + self.category,
            "cmlimit": "500",
            "cmprop": "title"
        }
        lastContinue = {"continue": ""}
        while True:
            par = params.copy()
            par.update(lastContinue)
            content = self.wiki.call(par)
            for page in content["query"]["categorymembers"]:
                #doshit
            if 'continue' not in content:
                break
            lastContinue = content["continue"]

(Kris' Python solution)

elifoster commented 8 years ago

That's basically what I described in that comment

ghost commented 8 years ago

Only difference is with mine, instead of redefining the variables every time, it just update the param values :P silly eli

Edit: You also want to return ALL the data in one return, since you no can constantly keep returning data's ;o

elifoster commented 8 years ago

Only difference is with mine, instead of redefining the variables every time, it just update the param values :P silly eli

I literally didn't say anything about that.

Edit: You also want to return ALL the data in one return, since you no can constantly keep returning data's ;o

I have no idea what you are saying here.

PaladinOne commented 8 years ago

THAT'S... NOT... HOW GRAMMAR WORKS

retep998 commented 8 years ago

Here's some code in Rust that is pretty robust at doing continue query stuff.

https://github.com/FTB-Gamepedia/mediawiki-rs/blob/master/src/lib.rs#L286-L325

ghost commented 8 years ago

I'll just stick with python >.> Faster, more stable, more powerful lol. Ever managed 16k API calls in 220 seconds with Ruby? ^_^

elifoster commented 8 years ago

Um, none of those are true? Ruby is actually faster than python according to most benchmarks you'll find, it's perfectly stable, and can do everything python can do (and vice versa).

ghost commented 8 years ago

Actually, if you want to argue about it, why is pretty much every linux system running python libraries and systems than ruby? Because it /is/ known to be faster in a lot of aspects. :P

Kris out.

elifoster commented 8 years ago

I've looked at 5 benchmarks this morning, all of which used older slower versions of ruby, and it was faster in every aspect than all kinds of python except cpython

Chocohead commented 8 years ago

There are also other benchmarks that put python faster than ruby though. Changing Python code from running globally to in a function can alone speed it up massively (for example), as well as depending on the version of which you use for both of them, so you could easily find a case by case basis for which one was faster if you wanted to, purely based on the way the test you were doing was written.

retep998 commented 8 years ago

Meanwhile I just write Rust and I get faster code than both Python and Ruby any day of the week.

elifoster commented 8 years ago

Apparently not if you are picking a random lines content out of a given file :)

retep998 commented 8 years ago

@elifoster You weren't comparing plain Rust to Ruby, you were comparing calling Rust code from Ruby with all the overhead of that compared to plain Ruby. Also that was you writing Rust, not me writing Rust.

elifoster commented 8 years ago

Good points