jcanny
commented
9 years ago Hi, I noticed in the code for dataparser taht you're concatenating data to matrices on lines 102 and 113. If you think about it, what that's doing is taking whatever matrix you have for "labels" or "sBagOfWords", creating a new matrix one element longer, and then discarding the old matrix. There's no efficient way to do that (e.g. C++ vector requires you to turn the same object with an element appended, not a new object). That will take quadratic time, and consume an enormous amount of memory. The simple way to do this is matrices is to make a matrix large enough to hold everything and maintain a pointer (say nextLabel) to the next "empty" location. When you're done filling it, just create a new matrix (once) as "labels(0->nextLabel")
One line 97 you do a string comparison with the Dict lookup of an integer each time instead of (once) doing a lookup of "
I would avoid ever concatenating sparse matrices in a loop. Instead repsent the sparse elements with three matrices "rows" and "cols" (nx IMat) and "vals" (nx1 FMat) with values. add to those matrices using the method above. When you're done, call sparse(rows(0->size),cols(0->size),vals(0,size),maxrows, maxcols).
With a bit of practice it should be possible to write that routine without looping over word locations (which allows it to run fast on a GPU). But I know that's a radically different way of thinking.
coryschillaci
In the last week before break @anasrferreira and @coryschillaci reprocessed the raw data into smaller chunks (max ~100Mb), tokenized without any of the
<base64>stuff included (thanks to @lambdaloop).The raw concatenated xml files are currently in
/var/local/destress/combined2/events/. These are concatenated using the script/path/to/destress/clean_data/combine_data.sh. For some reason, a few of the biggest ones are over 100Mb, but I'm not sure the bug is worth chasing down. Here are the largest:After tokenizing with
/path/to/destress/process_data/tokenize.sh, the various output files are all stored in/var/local/destress/tokenized2/The biggest IMat files after tokenizing areHowever, when @anasrferreira and I have tried running the
dataparser.scalacode to generate features from the tokenized data, Mercury runs out of memory and then of course slows way down. Does anybody have an idea of why this might be happening? I don't see why it should be using almost 16Gb of memory to process 100Mb or smaller files into 25Mb output chunks.