wp2mongo hangs during article insertion.

[MTKnife]

Every time I run wp2mongo (without the --worker flag) it hangs after extracting 777 articles from the (English) Wikipedia bz. When I re-run it without dropping the the database, I get a bunch of duplicate insertion errors, and, on examining them, it appears that it's trying to insert every article in the last batch except for the last article, on Ibn-al-Haytham, which means the problem must be occurring either during that insertion, or between the Algiers article and the Ibn-al-Haytham article.

There's nothing suspicious happening--no excessive CPU or RAM usage, and I'm not getting errors of any kind (other than the duplicate insertion errors mentioned above).

I've tried the obvious things, rebooting and reinstalling, but without any errors being thrown, I'm at a loss for what else to try.

[e501]

This also happed to me yesterday.

Over the last few weeks, I have been using parsing results from a dump file that loaded approx. 5.7 articles a few weeks ago. For the articles that were parsed, the results have worked great with mongolite (R package).

Yesterday, when I tried to rerun wp2mongo for the same enwiki file, the code hangs at the 777th article (Ibn al-Haytham), just as described in the posting. I then tried the most recent dump of enwiki and had same result. Rebooting and reinstalling produced same results. Note that the afwiki dump loads into mongodb with no problems.

Also, at the wtf_wikipedia web page (https://spencermountain.github.io/wtf_wikipedia/), the "Algiers" and other previous articles are immediately parsed and displayed, but a fetch for the "Ibn al-Haytham" article is not responsive.

[spencermountain]

agh, weird thanks you two. it looks like there is a bug on the Ibn al-Haytham in wtf_wikipedia. That's got to be it. I will try to fix it today.

[spencermountain]

fwiw, maybe this library should wrap wtf_wikipedia in a try/catch. It would make things slower (i think?) but these sort of errors are probably somewhat inevitable, given the size & nature of wikipedia. any insight or thoughts about doing that?

[e501]

From previous experience with using json parsers for Wikipedia articles, there has been an issue with extraneous characters and character combinations embedded in the articles. My thinking has been that to guard against confusing the respective parser, a preprocessor may need to validate that the string/blob of characters can be parsed, relative to the known specification and associated constraints/capabilities of the given parser. At this time, I use an ad-hoc collection of regex expressions that strip out potentially-problematic character strings.

Basically, the idea is to have something like a lint process (https://en.wikipedia.org/wiki/Lint_(software)) for flagging suspicious character combinations that may be in the input string (i.e. Wikipedia article). In this case where the parser is hanging (Ibn al-Haytham), there may be an issue with the additional types of characters that are embedded with the more usual English language characters.

[spencermountain]

hey, fixed the this Ibn al-Haytham issue in v2.1.0. can you guys check this out? cheers

[e501]

Many thanks for fixing the initial problem.

With the new fix, there are new recurring parsing errors, such as the following:

Julia Kristeva 10468 Error: key 0-19-518767-9}}, [https://books.google.com/books?id must not contain '.' at serializeInto (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:753:19) at serializeObject (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:308:18) at serializeInto (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:618:17) at serializeObject (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:308:18) at serializeInto (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:776:17) at serializeObject (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:308:18) at serializeInto (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:618:17) at serializeObject (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:308:18) at serializeInto (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:776:17) at BSON.serialize (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/bson.js:58:27) Juan Miro 10469 Just intonation 10470 Josephus 10471 Jan Borukowski 10472 Judy Blume 10473 Joel Marangella 10474 John Pople 10475 Jake McDuck 10476 Jerry Falwell 10477 Jebus 10478 Jay Leno 10479 Jeroboam II 10480 JTF-CNO 10481 Joan of Arc 10482 Error: key Centre Historique des Archives Nationales]], [[Paris]], AE II 2490, dated to the second half of the 15th century. "The later, fifteenth-century manuscript of [[Charles, Duke of Orléans]] contains a miniature of Joan in armour; the face has certain characteristic features known from her contemporaries' descriptions, and the artist may have worked from indications by someone who had known her." Joan M. Edmunds, The Mission of Joan of Arc (2008), [https://books.google.com/books?id must not contain '.' at serializeInto (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:753:19) at serializeObject (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:308:18) at serializeInto (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:618:17) at serializeObject (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:308:18) at serializeInto (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:776:17) at serializeObject (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:308:18) at serializeInto (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:618:17) at serializeObject (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:308:18) at serializeInto (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:776:17) at BSON.serialize (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/bson.js:58:27) Johannes Nicolaus Brønsted 10483 Janus kinase 10484 Jacob Grimm 10485 Jamiroquai 10486 John Sutter 10487 John Adams (composer) 10488 Jon Voight 10489 Error: key 978-0-306-80900-2}}, p.236. [https://books.google.com/books?id must not contain '.' at serializeInto (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:753:19) at serializeObject (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:308:18) at serializeInto (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:618:17) at serializeObject (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:308:18) at serializeInto (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:776:17) at serializeObject (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:308:18) at serializeInto (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:618:17) at serializeObject (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:308:18) at serializeInto (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/parser/serializer.js:776:17) at BSON.serialize (/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/bson/lib/bson/bson.js:58:27) John Climacus 10490 John of the Ladder 10491

At the wtf_wikipedia web page, the parser looks to be returning partial results for the problematic articles, wherein some of the sections look to be missing some content.

[MTKnife]

On the subject of the try/catch block, I did a quick search, and what I found suggests that what's true for Python (with which I'm familiar) is also true of JavaScript (with which I'm not familiar): unless there's very little happening in the try/catch block, the cost is trivial when there's no error. I think, given the size and complexity of Wikipedia, especially when you consider the multiple available languages, that you really should have error-catching, or you're likely to run into this kind of problem again. It would probably also be a good idea to produce an output that indicates any articles that have been skipped.

[spencermountain]

yeah, ok. that convinced me.

oh, @e501 i know what that is. will try to do it today.

thanks for the help!

[spencermountain]

@e501 fixed the encodings for citation keys, and @MTKnife added try/catch statements around the wikipedia parsing stuff. 2.2.0 tests passing. lemme know

[MTKnife]

I'm running it with Redis right now, and I think it's working, but I'm not entirely sure what I'm supposed to be seeing. I'm no longer getting errors (or any feedback whatsoever) when I run "worker.js", and the "wikipedia" collection in the "wikipedia_queue" database is getting steadily larger. However, I've noticed that the database is initiated and insertions start to take place before I run "worker.js", which leads me to assume that "wp2mongo" is inserting unparsed articles in the db. Presumably "worker.js" then goes back and parses those articles and modifies the records in question--so what should I look for to be sure that the records I'm seeing have been parsed?

Whoops...we just got some kind of Redis error. Here's what it looks like:

La Puente, California 71751 La Verne, California 71752 Ladera Heights, California 71753 events.js:183 throw er; // Unhandled 'error' event ^

Error: Redis connection to localhost:6379 failed - read ECONNRESET at _errnoException (util.js:1024:11) at TCP.onread (net.js:615:25)

Since I'm about to leave work, I'm re-running it without the Redis, and so far that's working.

UPDATE: The Redis thing appears to be some kind of issue external to wp2mongo. See this link.

[e501]

Many thanks for your rapid turnaround on the most recent fixes. This evening, I was able to process up to 5508058 articles from the enwiki-20171120-pages-articles.xml.bz2 dump. Unfortunately, right after the "Love You More (Ginuwine song) 5508058" string was output to the terminal, the following core dump occurred:

Stacktrace: magic1=bbbbbbbb magic2=bbbbbbbb ptr1=0xdc8db802459 ptr2=(nil) ptr3=(nil) ptr4=(nil) ptr5=(nil) ptr6=(nil) ptr7=(nil) ptr8=(nil)

==== JS stack trace =========================================

Security context: 0x38b8e6425729 #0# 2: parse [/usr/local/lib/node_modules/wikipedia-to-mongodb/src/doPage.js:64] [bytecode=0x2b06ad8b2651 offset=50](this=0x11d500ff4251

[MTKnife]

Oh, I forgot to say thanks as well!

I'm not having the same problem as @e501 --I'm at 7,360,000 articles and counting right now.

[MTKnife]

Interestingly, the app slowed down at some point (or it's slowing down gradually): in the 16 hours I was absent from work, it got through over 7,000,000 articles. In the 5.5 hours since, it's managed to handle only 300,000. Of course, I've been using the computer in the meantime, but not for anything intensive.

[e501]

When I ran the script with the previous wikipedia dump (enwiki-20171020-pages-articles.xml.bz2) and not with the Redis option, the code ran up to 9314943 articles. Seems that the "List of compositions by Franz Schubert" is a bit long and exceeds the default memory size allocated. From a quick web search, looks like I need to figure out how to increase the available memory, such as putting "--max_old_space_size=8192 --optimize_for_size --stack_size=8192" on the command line or somehow incorporated into the package installation.

Many thanks again for providing the script. The ability to work with the JSON formated parsing results has already been a great help !

Looking forward to eventually having the entire dump of articles available, once we are able to work through the various details.

The following is the last few lines output to the terminal before crashing:

Akdam, Alanya 9314940 Akçatı, Alanya 9314941 Alacami, Alanya 9314942 List of compositions by Franz Schubert 9314943

<--- Last few GCs --->

[28360:0x371f240] 108080669 ms: Mark-sweep 1308.6 (1424.9) -> 1308.6 (1424.9) MB, 644.9 / 0.0 ms allocation failure GC in old space requested [28360:0x371f240] 108081313 ms: Mark-sweep 1308.6 (1424.9) -> 1308.6 (1423.9) MB, 643.7 / 0.0 ms last resort GC in old space requested [28360:0x371f240] 108081956 ms: Mark-sweep 1308.6 (1423.9) -> 1308.6 (1423.9) MB, 643.1 / 0.0 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0xf789d0a5729 1: push(this=0xc613fe1f3b9 <JSArray[257594]>) 2: infobox(aka parse_recursive) [/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/wtf_wikipedia/src/parse/infobox/index.js:~16] [pc=0x247a2c9f340f](this=0x235a6b03d599

[spencermountain]

haha, yeah, wikipedia struggles with that one too ;)

fixed some bugs, and release a new version, 2.3.0.

@e501 i couldn't reproduce any issues with those articles, but there have been some fixes in wtf_wikipedia in the latest version. (re-)doing npm install should fix the cannot-find 'commander' issue, i believe.

this supports skipping redirects and disambiguation pages.

lemme know if it helps.

[MTKnife]

Great, thanks!

For the record, I got a crash similar to @e501's, but in my case it got a bit further, to article 9891192, Mansarovar Park (Delhi Metro)--that's not a long article, so I'm not sure what was going on.

[spencermountain]

you know what, if nobody has gotten the redis-version to work, i bet it makes a lot of sense to try one of these cool multi-threaded node libraries, i bet that would help this situation greatly.

i have a little bit of history using these, but the tricky part is that xml-stream runs at its own pace - so somehow we'd need to get the xml-streaming, and article-parsing, running in time with each other. I'm not sure how to do this. I think the redis version writes things to a temporary mongo db? I'm actually not sure.

[MTKnife]

OK, just started it up with this command (note that the syntax you use in the README, creating a JSON object and then adding it after the command in parentheses, doesn't work in Windows, at least not with the compiled "wp2mongo" file):

wp2mongo 'C:/data/Wikipedia/enwiki-20171103-pages-articles.xml.bz2' --db='enwiki' --skip_redirects=true --skip_disambig=true --worker

Before doing that, I started up kue, and then, after starting wp2mongo, I started up worker.js. Never got any output in either of those two windows, except for "Running on http://127.0.0.1:3000" in the kue window.

In any case, wp2mongo crashed pretty quickly:

Playstation 16200 Pterodactylus 16201 Pterosaur 16202

<--- Last few GCs --->

[2240:000001E8B40C0AF0] 203715 ms: Mark-sweep 1403.0 (1478.1) -> 1402.9 (1475.1) MB, 703.5 / 0.0 ms (+ 0.0 ms in 0 st eps since start of marking, biggest step 0.0 ms, walltime since start of marking 704 ms) last resort GC in old space req uested [2240:000001E8B40C0AF0] 204407 ms: Mark-sweep 1402.9 (1475.1) -> 1403.0 (1475.1) MB, 692.4 / 0.0 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0000038F45DA5EC1 0: builtin exit frame: stringify(this=0000038F45D89021

[MTKnife]

OK, it's been running since this afternoon, and it's now at over 5.7 million articles; double-checking in the MongoDB client, I see the count is the same there--so the figure is the number of actual articles, not the number before redirects and disambiguation pages have been excluded. Since the English Wikipedia currently has only just over 5.5 million articles, something is obviously wrong.

I think I typed the command-line options wrong, but I'm not sure how to do it right: the README has two hyphens in front of them, but the app's internal help specifies one hyphen--and it's very confusing about whether or not there are supposed to be arguments ("true") after "skip_redirects" and "skip_disambig".

I'm trying this right now:

PS C:\code\JavaScript\wikipedia-to-mongodb> wp2mongo 'C:/data/Wikipedia/enwiki-20171103-pages-articles.xml.bz2' -skip_redirects -skip_disambig

At any rate, given that it takes hours to verify if the app is proceeding correctly, it might be useful to have it throw an error in the case of invalid options, rather than running normally with no feedback.

[MTKnife]

Nope, that didn't work: woke up and we were at 7 million articles.

Trying it with 2 hyphens after I get to work.

[MTKnife]

Ah, I see now how to check for redirects directly (no pun intended). None of these commands succeeds in skipping them:

wp2mongo 'C:/data/Wikipedia/enwiki-20171103-pages-articles.xml.bz2' --skip_redirects --skip_disambig wp2mongo 'C:/data/Wikipedia/enwiki-20171103-pages-articles.xml.bz2' --skip_redirects true --skip_disambig true wp2mongo 'C:/data/Wikipedia/enwiki-20171103-pages-articles.xml.bz2' -skip_redirects true -skip_disambig true wp2mongo 'C:/data/Wikipedia/enwiki-20171103-pages-articles.xml.bz2' -skip_redirects=true -skip_disambig=true

I'm at a loss here. What do I need to type?

[MTKnife]

There we go....The options assignment to the commander object ("program") in wp2mongo.js needs to look like this:

program .usage('node index.js enwiki-latest-pages-articles.xml.bz2 [options]') .option('-w, --worker', 'Use worker (redis required)') .option('-plain, --plaintext', 'if true, store plaintext wikipedia articles') .option('--skip_redirects', 'if true, skips-over pages that are redirects') .option('--skip_disambig', 'if true, skips-over disambiguation pages') .parse(process.argv)

Note that, because of the way the app is structured, with the "skip" options enabled, the article numbers in the console output will no longer match the number of records in the database.

Oddly, the change doesn't seem to have made a difference in the Redis version. I can't figure out the commander syntax well enough to understand why the old version was working for the "worker" option and not for the "skip" options.

[spencermountain]

thanks scott. The skipping thing works in the test here, but yeah, shoulda tested the cli version too.

lemme know if this gets further without a slowdown, that would be great.

fwiw, i just downloaded a en-wikipedia dump (finally) last night, so i can start stressing it out too. cheers

[e501]

Many thanks again for creating and sharing your parser. Good news is that we are making progress.

The latest script (version, 2.3.0) parsed up to 9310799 articles of the enwiki-20171103-pages-articles.xml.bz2 dump. Bad news is that the "Punk rock in France" article caused a core dump (see below). The article is not very large. My machine has 64GB of RAM and lots of swap space. My thinking is that even if there is a memory leak, I could try making the javascript heap extremely large.

Thus, still seems that I may need to figure out how to increase the available memory, such as putting "--max_old_space_size=8192 --optimize_for_size --stack_size=8192" on the command line or somehow incorporated into the package installation.

The immediate goal is to be able to do a complete run through the most recent dump of Wikipedia. From tracking my mongodb log, the latest version seems to be logging the articles that may have had issues. Please confirm that is how I can go back and track down the "problem articles" that are not parsed and stored in the mongodb db. Using the mediawiki API, these articles can be addressed on a case-by-case basis.

Also, I would like to explore the possibility of working with the javascript code for minor tailoring and augmenting of the functionality. For example, "mediawiki template" (e.g. navigation template) parsing capabilities would be a big plus. Problem is that I already need to ramp up on PHP, while still trying to continue to make progress with using R/RStudio (and Python) packages for NLP and statistical analysis. Being new to javascript (and node.js), initial web searches indicate that PHPStorm seems to be a viable IDE for both PHP and javascript (and node.js). Feedback and even a rough-draft HowTo for setting up the IDE and edit-build-test cycle would help scope out the viability of being able to work more directly with the source code.

The output of the core dump follows:

Aaron Sopher 9310797 Danny Talbot 9310798 Punk rock in France 9310799

<--- Last few GCs --->

[19270:0x2a732c0] 105842469 ms: Mark-sweep 1293.2 (1449.7) -> 1293.2 (1449.7) MB, 646.8 / 0.0 ms allocation failure GC in old space requested [19270:0x2a732c0] 105843118 ms: Mark-sweep 1293.2 (1449.7) -> 1293.2 (1416.7) MB, 649.0 / 0.0 ms last resort GC in old space requested [19270:0x2a732c0] 105843773 ms: Mark-sweep 1293.2 (1416.7) -> 1293.2 (1416.7) MB, 654.9 / 0.0 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3f7f6e425729 1: f [/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/unbzip2-stream/index.js:~25] [pc=0x91227a48be9](this=0x187bf410c0f9 ,b=229) 2: / anonymous / [/usr/local/lib/node_modules/wikipedia-to-mongodb/node_modules/unbzip2-stream/lib/bzip2.js:~163] [pc=0x91215c8af3e](this=0x41292882249

[MTKnife]

Still running, but one finding: I started it about 11am, and it was at over 6 million articles (including all articles--that's under 2.5 million inserted in the database) when I left work at 6:00pm, with articles streaming by faster than I could read the titles. When I got home at 7:00, I glanced at it and it seemed to be moving slowly, and three hours later, the articles were scrolling past noticeably more slowly, and it still hadn't hit 7 million. I haven't been monitoring it the whole time, but it seems like it might have slowed down suddenly, rather than the gradual slowing that would have been the case if the database indexing were to blame.

I apparently forgot to hit the "Comment" button and post the above, so I'll add this update: as of 2:00pm, I'm at 7.95 million, which means that it's proceeding at about the same speed it did before the "skip" options were added. I assume that it's going to crash at around 10 million, since that's what it just did for @e501, and for both of us the last time through.

I'm mystified that adding the "skip" options didn't speed it up, because that implies that there's something about the JavaScript app itself that makes it go slower for the later articles. I wonder if whatever it is is related to the crashes?

[MTKnife]

The crash came just now, at 9906759, at "List of United States counties and county equivalents". For the record, we got 4.1 million articles into the database before that happened.

[spencermountain]

hey cool, yeah we're getting pretty close. Also, it seems pretty likely there is a memory leak somewhere. it would be great if we could find the memory leak. There are some tools for identifying this in node applications. It's also good news that it's not a mongo-related slowdown. That db can handle a lot more action than this, i bet.

maybe it's a good idea to add more --skip flags, to pass-through to wtf_wikipedia. If you're not, say, using the citations, or images, we could skip those steps in the parser.

but yeah, the --max-old-space-size stuff could also get to the end. That's worth a shot. I'll try that mem-leak tool today, if i can find some time.

oh hey, didn't someone say they tried adding a condition in the parser to skip the first-n pages, but it still collapsed? I mean, it would be interesting to skip the first 4m articles, and see if you can get the last-half done on a second pass. If anything, that would isolate our mem-leak a little. ok, cheers!

[spencermountain]

ah, it's not a memory-leak. just tested it. I think i know what's happening.

for each page, the xml-parser isn't waiting for the mongo insert to complete before continuing. I think it's just building-up pending mongo-inserts, after a long time. That would make sense.

the good news is that means you guys should be able skip-n articles and complete the remaining ones, without any memory problems. I'll implement a skip-n flag now.

[MTKnife]

OK, thanks, that sounds good.

You should be able to tell it to wait on insertions so that we don't run into the problem in the future, right?

[spencermountain]

hey, so in 2.4.0 it now 'takes a break' every 30s, for 3 seconds. It ran-though the simple-english wiki without any issue.

i also added the --skip_first 500 option, so you can just breeze-through the first 4m or whatever.

changed the logging a bit, so that errors could be found easier. turn on --verbose if you want to see the articles cheers

[MTKnife]

OK, I'll give it a try right now.

[MTKnife]

Foo. It's running, but it's not making insertions in the database.

[spencermountain]

did you add a skip parameter?

[MTKnife]

Yes, told it to skip the first 9,906,000--since it last stopped at 9,906,759, it should have started adding within 1,000 articles.

Where would the log be, BTW?

[spencermountain]

if you tell it to skip the first 9 million, it won't insert anything into the database until the 9000001th article, so you'll have to wait. if you want a log, you can just put a console.log in.

[MTKnife]

Ah....Yeah, I guess there's no way around that, given the size of the file. We'll see if that speeds things up. I'm not sure we need the pauses in the early stages, though--I would imagine the crashes happen when the MongoDB index gets to a certain size, which slows things down too much.

BTW, why is the number of pages different in each batch? Does it grab a fixed number of lines or something like that?

[MTKnife]

Actually, let me try something on my fork.

[MTKnife]

I modified the setInterval function to avoid the pause if "i" is under 8,000,000, then to start at 1s and increase linearly from there (since the insertion time is probably O(N), with N being the index or database size). It's working, but it occurs to me that, even with a 3s pause every 30s, the thing shouldn't be proceeding much more slowly than the old version, and some quick calculations suggest it isn't--I guess it just seems slower without the article titles whizzing by.

BTW, I can't get the "--verbose" option to work, though, looking at the code (with my zero knowledge of JavaScript), I can't see why not.

[spencermountain]

thanks scott. lemme know if you get to the end. I'm happy to merge a pr for more sophisticated sleeping rules.

i'm gonna try this today, to see if we can speed this bad-boy up a bit.

[MTKnife]

Since the impact is minor (less than 10% of running time), I'm going to wait until I'm sure it works before I push it and submit a pull request. Right now, we're at 7.1 million articles.

The workerpool library looks like a good idea, but I'm still wondering what the limiting factor actually is. The current speed is not that different from the original version, even without any parsing, which suggests that the limiting factor is the disk read. That wouldn't be in the hard sense of physical I/O, but rather a matter of the overhead associated with iterating through the articles. I'm not sure if it's relevant to this or not, but I just timed the interval between messages from "logger", and it's more like 7s rather than the 5s specified in the script. I wonder if the "setInterval" loops are themselves slowing things up?

That leaves open the question of why things slow down at around 6 million articles, and maybe it has something to do with the expanding queue as MongoDB slows down with larger and larger indices? If that's the case, the workerpool thing might not help much.

Or maybe the key is to put multiple threads on the MongoDB insertion, rather than on the parser?

[MTKnife]

I'm close to 9.3 million this morning, so not inserting new records yet, but the modified delay code is working so far, so I'll push it and send you a pull request in a bit here.

[MTKnife]

OK, something's gone wonky. As I mentioned above, in the last run-through, I got to 9,906,759 articles. Once you added the skip option, I restarted and told it to skip the first 9,906,000. It passed those numbers some time yesterday, and it's over 10.1 million now--but every single insert attempt is still failing with a "duplicate key" error. I'm trying to understand how that could happen: is it possible the count now proceeds differently than it did before?

BTW, the "--verbose" option is in fact working--when I mentioned it before, I didn't realize the article names are triggered only for articles that are parsed. It would be nice, though, to return the article number in each line, because the "logger" summary output gets buried in all the article names.

[MTKnife]

A short update: the articles in question do actually appear to be in the database already, so the "duplicate key" error is valid.

[MTKnife]

I think I found where the count is going off: in the first commit of Dec. 4, the check of the "ns" (namespace) field got moved from "index.js" to "parse.js"; one consequence of that move is that it now occurs after the increment of "i" (the article count variable) rather than before.

You'd think everything in the dump would have a namespace of "0" (actual content), but evidently that's not the case.

Anyway, a more significant problem: once the app started parsing articles rather than skipping them things slowed to a crawl with only 500K or so articles processed in the last ~24 hours. I wonder if the insertion errors are slowing things down?

[MTKnife]

Nearly two days now since we finished skipping articles, and still no new ones: we're at 11.2 million now.

I wonder what the heck all the non-"0" records in the dump are? Given the differences in numbers, there are at least 1.5 million records that weren't counted last time, or greater than 1 in every 10 records. And yet I don't see any titles scrolling by that don't look like article titles.

EDIT: Duh...the titles of the non-"0" pages don't get printed in the log. But they do contribute to the count.

[MTKnife]

It crashed some time this morning, at the same article as last time, "List of United States counties and county equivalents", which is 12.7 million-something under the new numbering. IIRC, the way I had it programmed when it started, that the delay would have been a little over 1.5s at that point.

The probability that it crashed on exactly the same article by chance is vanishingly small, though I can't explain why it would have crashed on another article during my first run.

[spencermountain]

wow. yeah that's something. thanks for sharing this. I can't figure-out why that page triggers any problem. i tested it a bunch of ways just now i'm pretty busy the next few days. lemme know if you figure something out. Perhaps you're at the end? perhaps that page is bad-xml?

just now i opened the ./tests/smallwiki and added it to the end, zipped it back up, and run the tests, and it worked fine. sorry i can't help any further.

[MTKnife]

The one thing I do know is that I'm not at the end--it's only got 4.1 million records in the database, and there are supposed to be over 5.5 million articles in the English Wikipedia. Plus, in watching the articles go buy, I've noticed they seem to be in the order they were originally inserted in the wiki--and the crash appears to occur during the 2013 articles.

I don't know enough about JavaScript to have any good ideas on this end :(.

[spencermountain]

oh, that's good to know.

what is the error when it crashes? is it still the process out of memory one?

if so, i recommend just splitting the xml file this looks like it can split bz2 compressed xml file.

hey, if it's actually crashing on this specific place each time, and there's some kindof queue-backup, it could definetly be a few articles forward to backward from the counties-or-equivalents page. if you can find a page that throws an error, that'd be an easy fix presumably.

[MTKnife]

It's still the memory error, yes.

I think if it were a queue problem, it would crash on a different article once the delay has been introduced, unless (as I think you're implying) the back-up occurs very quickly, between the last pause and the parsing of the article whose title shows up when the crash occurs--and that in turn implies that the problematic article is one that's parsed after that last pause.

It's a pain to test, though, since, even with that tool you linked, it'll take a while to split the file.