search

KWARC / llamapun

common language and mathematics processing algorithms, in Rust
https://kwarc.info/systems/llamapun/
GNU General Public License v3.0
25 stars 6 forks source link

corpus_token_model leaks memory #9

Closed dginev closed 6 years ago

dginev commented 6 years ago

I am seeing an allocated ~16g of RES memory in top, after traversing 384,000 documents.

The corpus_token_model was designed to be constant in memory as it uses a fixed size buffer in writing to a dedicated file.

As such, I assume the leak is coming from one of the dependency libraries, and the only one I updated recently (and is hence the most likely culprit) is the rust-libxml crate.

I would need to run under valgrind on a small mock corpus and file a patch in either crate, as this makes things problematic for larger corpora. Luckily I have a very high RAM capacity machine, but this should be fixed before the next llamapun release.

dginev commented 6 years ago

I have a first guess that Nodes are just leaking memory in general in the libxml crate: https://github.com/KWARC/rust-libxml/commit/b8cce377431d58e8bb1ce5e826a4f4412cc24041#diff-ecc927991c14fae6e03a511ed1d458b3L42

will wait for a valgrind run before jumping to conclusions though.

dginev commented 6 years ago

It seems https://github.com/KWARC/rust-libxml/issues/25 will keep haunting me until a great solution is found...

dginev commented 6 years ago

I had to enable the jemallocator explicitly to run valgrind, but that payed off and I can verify a variety of libxml2 related leaks, running over 100+ documents in the 9702 subset

Command:

cargo test; valgrind -v --show-reachable=yes --leak-check=full  ./target/debug/examples/corpus_token_model /data/datasets/dataset-arXMLiv-08-2017/no_problem/9702

Result:

==52894== 
==52894== HEAP SUMMARY:
==52894==     in use at exit: 3,610,829 bytes in 68,316 blocks
==52894==   total heap usage: 32,524,489 allocs, 32,456,173 frees, 2,524,790,398 bytes allocated
==52894== 
==52894== Searching for pointers to 68,316 not-freed blocks
==52894== Checked 2,919,816 bytes
==52894== 
==52894== 5 bytes in 1 blocks are still reachable in loss record 1 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E66B: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894== 
==52894== 6 bytes in 1 blocks are still reachable in loss record 2 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E66B: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E46F: xmlInitCharEncodingHandlers (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894== 
==52894== 6 bytes in 1 blocks are still reachable in loss record 3 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E66B: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E4FF: xmlInitCharEncodingHandlers (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894== 
==52894== 7 bytes in 1 blocks are still reachable in loss record 4 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E66B: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E4CB: xmlInitCharEncodingHandlers (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894== 
==52894== 7 bytes in 1 blocks are still reachable in loss record 5 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E72EF6: xmlCopyError (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E7334F: __xmlRaiseError (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4EC2E9E: ??? (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4EC9EA0: ??? (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4EC9F90: ??? (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4ECDD75: htmlReadFile (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4358A4: libxml::parser::Parser::parse_file (parser.rs:101)
==52894==    by 0x2B5103: llamapun::data::Document::new (data.rs:186)
==52894==    by 0x2B47E3: <llamapun::data::DocumentIterator<'iter> as core::iter::iterator::Iterator>::next (data.rs:135)
==52894==    by 0x2AF5D7: corpus_token_model::main (corpus_token_model.rs:49)
==52894== 
==52894== 9 bytes in 1 blocks are still reachable in loss record 6 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E66B: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E489: xmlInitCharEncodingHandlers (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894== 
==52894== 9 bytes in 1 blocks are still reachable in loss record 7 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E66B: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E4AA: xmlInitCharEncodingHandlers (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894== 
==52894== 9 bytes in 1 blocks are still reachable in loss record 8 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E66B: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E519: xmlInitCharEncodingHandlers (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894== 
==52894== 10 bytes in 3 blocks are definitely lost in loss record 9 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894== 
==52894== 10 bytes in 5 blocks are definitely lost in loss record 10 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894== 
==52894== 11 bytes in 1 blocks are still reachable in loss record 11 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E66B: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E4E5: xmlInitCharEncodingHandlers (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894== 
==52894== 19 bytes in 1 blocks are indirectly lost in loss record 12 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894== 
==52894== 20 bytes in 1 blocks are still reachable in loss record 13 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E72EDE: xmlCopyError (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E7334F: __xmlRaiseError (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4EC2E9E: ??? (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4EC9EA0: ??? (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4EC9F90: ??? (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4ECDD75: htmlReadFile (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4358A4: libxml::parser::Parser::parse_file (parser.rs:101)
==52894==    by 0x2B5103: llamapun::data::Document::new (data.rs:186)
==52894==    by 0x2B47E3: <llamapun::data::DocumentIterator<'iter> as core::iter::iterator::Iterator>::next (data.rs:135)
==52894==    by 0x2AF5D7: corpus_token_model::main (corpus_token_model.rs:49)
==52894== 
==52894== 21 bytes in 5 blocks are definitely lost in loss record 14 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894== 
==52894== 32 bytes in 1 blocks are still reachable in loss record 15 of 62
==52894==    at 0x4C31B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x5B5382E: _dlerror_run (dlerror.c:141)
==52894==    by 0x5B53091: dlopen@@GLIBC_2.2.5 (dlopen.c:87)
==52894==    by 0x6531D25: ??? (in /lib/x86_64-linux-gnu/liblzma.so.5.2.2)
==52894==    by 0x6538617: lzma_auto_decoder (in /lib/x86_64-linux-gnu/liblzma.so.5.2.2)
==52894==    by 0x4F9EB7E: ??? (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4FA0137: __libxml2_xzread (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4EAC558: ??? (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4EAEAF8: xmlParserInputBufferGrow (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E73FFF: xmlParserInputGrow (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4ECDE69: htmlReadFile (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4358A4: libxml::parser::Parser::parse_file (parser.rs:101)
==52894== 
==52894== 32 bytes in 1 blocks are still reachable in loss record 16 of 62
==52894==    at 0x4C72DF: _rjem_mallocx (jemalloc.c:2292)
==52894==    by 0x2EE3B3: <&'a jemallocator::Jemalloc as alloc::allocator::Alloc>::alloc (lib.rs:135)
==52894==    by 0x2B40C7: __rg_alloc (lib.rs:25)
==52894==    by 0x42598D: <alloc::heap::Heap as alloc::allocator::Alloc>::alloc (heap.rs:84)
==52894==    by 0x4258A3: alloc::allocator::Alloc::alloc_array (allocator.rs:972)
==52894==    by 0x4284EC: <alloc::raw_vec::RawVec<T, A>>::double (raw_vec.rs:324)
==52894==    by 0x4255C0: <alloc::vec::Vec<T>>::push (vec.rs:971)
==52894==    by 0x428A6A: <alloc::binary_heap::BinaryHeap<T>>::push (binary_heap.rs:557)
==52894==    by 0x423A1B: thread_local::thread_id::ThreadIdManager::free (thread_id.rs:36)
==52894==    by 0x423B3A: <thread_local::thread_id::ThreadId as core::ops::drop::Drop>::drop (thread_id.rs:53)
==52894==    by 0x4245A4: core::ptr::drop_in_place (ptr.rs:59)
==52894==    by 0x4244AA: core::ptr::drop_in_place (ptr.rs:59)
==52894== 
==52894== 40 bytes in 1 blocks are still reachable in loss record 17 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F0E50A: xmlNewMutex (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4F0D798: xmlInitGlobals (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E954F9: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894== 
==52894== 40 bytes in 1 blocks are still reachable in loss record 18 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F0E50A: xmlNewMutex (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4EB1EB9: xmlInitMemory (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E9551A: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894== 
==52894== 42 bytes in 21 blocks are indirectly lost in loss record 19 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894==    by 0x2B56A1: <llamapun::data::ParagraphIterator<'iter> as core::iter::iterator::Iterator>::next (data.rs:234)
==52894==    by 0x2AF757: corpus_token_model::main (corpus_token_model.rs:51)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894==    by 0x62B0F7: {{closure}} (rt.rs:59)
==52894==    by 0x62B0F7: _ZN3std9panicking3try7do_call17h173b0f0e2d5a0273E.llvm.9CC80E2B (panicking.rs:480)
==52894== 
==52894== 44 bytes in 1 blocks are still reachable in loss record 20 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x5355BA5: _dl_signal_error (dl-error-skeleton.c:99)
==52894==    by 0x401536B: _dl_open (dl-open.c:714)
==52894==    by 0x5B52FF5: dlopen_doit (dlopen.c:66)
==52894==    by 0x5355D63: _dl_catch_error (dl-error-skeleton.c:198)
==52894==    by 0x5B53758: _dlerror_run (dlerror.c:163)
==52894==    by 0x5B53091: dlopen@@GLIBC_2.2.5 (dlopen.c:87)
==52894==    by 0x6531D25: ??? (in /lib/x86_64-linux-gnu/liblzma.so.5.2.2)
==52894==    by 0x6538617: lzma_auto_decoder (in /lib/x86_64-linux-gnu/liblzma.so.5.2.2)
==52894==    by 0x4F9EB7E: ??? (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4FA0137: __libxml2_xzread (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4EAC558: ??? (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894== 
==52894== 48 bytes in 1 blocks are still reachable in loss record 21 of 62
==52894==    at 0x4C72DF: _rjem_mallocx (jemalloc.c:2292)
==52894==    by 0x2EE3B3: <&'a jemallocator::Jemalloc as alloc::allocator::Alloc>::alloc (lib.rs:135)
==52894==    by 0x2B40C7: __rg_alloc (lib.rs:25)
==52894==    by 0x42598D: <alloc::heap::Heap as alloc::allocator::Alloc>::alloc (heap.rs:84)
==52894==    by 0x4256AA: alloc::heap::exchange_malloc (heap.rs:249)
==52894==    by 0x423CD4: <std::sync::mutex::Mutex<T>>::new (mutex.rs:180)
==52894==    by 0x4242DB: __static_ref_initialize (thread_id.rs:40)
==52894==    by 0x4242DB: core::ops::function::FnOnce::call_once (function.rs:223)
==52894==    by 0x4249A2: <lazy_static::lazy::Lazy<T>>::get::{{closure}} (lazy.rs:24)
==52894==    by 0x424859: std::sync::once::Once::call_once::{{closure}} (once.rs:227)
==52894==    by 0x629C3C: std::sync::once::Once::call_inner (once.rs:340)
==52894==    by 0x4247E3: std::sync::once::Once::call_once (once.rs:227)
==52894==    by 0x423BFA: get<std::sync::mutex::Mutex<thread_local::thread_id::ThreadIdManager>,fn() -> std::sync::mutex::Mutex<thread_local::thread_id::ThreadIdManager>> (lazy.rs:23)
==52894==    by 0x423BFA: __stability (<__lazy_static_internal macros>:13)
==52894==    by 0x423BFA: <thread_local::thread_id::THREAD_ID_MANAGER as core::ops::deref::Deref>::deref (<__lazy_static_internal macros>:14)
==52894== 
==52894== 48 bytes in 1 blocks are still reachable in loss record 22 of 62
==52894==    at 0x4C72DF: _rjem_mallocx (jemalloc.c:2292)
==52894==    by 0x2EE3B3: <&'a jemallocator::Jemalloc as alloc::allocator::Alloc>::alloc (lib.rs:135)
==52894==    by 0x2B40C7: __rg_alloc (lib.rs:25)
==52894==    by 0x42598D: <alloc::heap::Heap as alloc::allocator::Alloc>::alloc (heap.rs:84)
==52894==    by 0x4256AA: alloc::heap::exchange_malloc (heap.rs:249)
==52894==    by 0x4249B5: new<std::sync::mutex::Mutex<thread_local::thread_id::ThreadIdManager>> (boxed.rs:242)
==52894==    by 0x4249B5: <lazy_static::lazy::Lazy<T>>::get::{{closure}} (lazy.rs:24)
==52894==    by 0x424859: std::sync::once::Once::call_once::{{closure}} (once.rs:227)
==52894==    by 0x629C3C: std::sync::once::Once::call_inner (once.rs:340)
==52894==    by 0x4247E3: std::sync::once::Once::call_once (once.rs:227)
==52894==    by 0x423BFA: get<std::sync::mutex::Mutex<thread_local::thread_id::ThreadIdManager>,fn() -> std::sync::mutex::Mutex<thread_local::thread_id::ThreadIdManager>> (lazy.rs:23)
==52894==    by 0x423BFA: __stability (<__lazy_static_internal macros>:13)
==52894==    by 0x423BFA: <thread_local::thread_id::THREAD_ID_MANAGER as core::ops::deref::Deref>::deref (<__lazy_static_internal macros>:14)
==52894==    by 0x423A46: thread_local::thread_id::ThreadId::new (thread_id.rs:48)
==52894==    by 0x423C1C: thread_local::thread_id::THREAD_ID::__init (thread_id.rs:56)
==52894== 
==52894== 56 bytes in 1 blocks are still reachable in loss record 23 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4E6E685: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E46F: xmlInitCharEncodingHandlers (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894== 
==52894== 56 bytes in 1 blocks are still reachable in loss record 24 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4E6E685: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E489: xmlInitCharEncodingHandlers (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894== 
==52894== 56 bytes in 1 blocks are still reachable in loss record 25 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4E6E685: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E4AA: xmlInitCharEncodingHandlers (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894== 
==52894== 56 bytes in 1 blocks are still reachable in loss record 26 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4E6E685: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E4CB: xmlInitCharEncodingHandlers (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894== 
==52894== 56 bytes in 1 blocks are still reachable in loss record 27 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4E6E685: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E4E5: xmlInitCharEncodingHandlers (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894== 
==52894== 56 bytes in 1 blocks are still reachable in loss record 28 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4E6E685: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E4FF: xmlInitCharEncodingHandlers (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894== 
==52894== 56 bytes in 1 blocks are still reachable in loss record 29 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4E6E685: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E6E519: xmlInitCharEncodingHandlers (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894== 
==52894== 56 bytes in 1 blocks are still reachable in loss record 30 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4E6E685: xmlNewCharEncodingHandler (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894==    by 0x62B0F7: {{closure}} (rt.rs:59)
==52894==    by 0x62B0F7: _ZN3std9panicking3try7do_call17h173b0f0e2d5a0273E.llvm.9CC80E2B (panicking.rs:480)
==52894== 
==52894== 60 bytes in 13 blocks are indirectly lost in loss record 31 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894== 
==52894== 74 bytes in 1 blocks are still reachable in loss record 32 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E72EEA: xmlCopyError (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E7334F: __xmlRaiseError (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4EC2E9E: ??? (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4EC9EA0: ??? (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4EC9F90: ??? (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4ECDD75: htmlReadFile (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4358A4: libxml::parser::Parser::parse_file (parser.rs:101)
==52894==    by 0x2B5103: llamapun::data::Document::new (data.rs:186)
==52894==    by 0x2B47E3: <llamapun::data::DocumentIterator<'iter> as core::iter::iterator::Iterator>::next (data.rs:135)
==52894==    by 0x2AF5D7: corpus_token_model::main (corpus_token_model.rs:49)
==52894== 
==52894== 74 bytes in 3 blocks are definitely lost in loss record 33 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894== 
==52894== 104 bytes in 1 blocks are still reachable in loss record 34 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F0E5BA: xmlNewRMutex (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4F7887C: __xmlInitializeDict (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x5943138: __pthread_once_slow (pthread_once.c:116)
==52894==    by 0x4F0E960: xmlIsMainThread (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4F0DEA8: __xmlGenericError (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E954FE: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894== 
==52894== 104 bytes in 52 blocks are indirectly lost in loss record 35 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894==    by 0x2B56A1: <llamapun::data::ParagraphIterator<'iter> as core::iter::iterator::Iterator>::next (data.rs:234)
==52894==    by 0x2AF757: corpus_token_model::main (corpus_token_model.rs:51)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894==    by 0x62B0F7: {{closure}} (rt.rs:59)
==52894==    by 0x62B0F7: _ZN3std9panicking3try7do_call17h173b0f0e2d5a0273E.llvm.9CC80E2B (panicking.rs:480)
==52894== 
==52894== 163 bytes in 1 blocks are indirectly lost in loss record 36 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894== 
==52894== 175 bytes in 9 blocks are indirectly lost in loss record 37 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894== 
==52894== 199 bytes in 53 blocks are definitely lost in loss record 38 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894== 
==52894== 217 bytes in 3 blocks are indirectly lost in loss record 39 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894== 
==52894== 400 bytes in 1 blocks are still reachable in loss record 40 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4E6E43F: xmlInitCharEncodingHandlers (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E95524: xmlInitParser (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x433B3A: libxml::global::_libxml_global_init::{{closure}} (global.rs:11)
==52894==    by 0x433C10: libxml::global::with_lock (global.rs:50)
==52894==    by 0x433B18: libxml::global::_libxml_global_init (global.rs:9)
==52894==    by 0x4354EC: <libxml::parser::Parser as core::default::Default>::default (parser.rs:68)
==52894==    by 0x2B4A9F: <llamapun::data::Corpus as core::default::Default>::default (data.rs:154)
==52894==    by 0x2B4CEF: llamapun::data::Corpus::new (data.rs:167)
==52894==    by 0x2AF552: corpus_token_model::main (corpus_token_model.rs:48)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894==    by 0x62B0F7: {{closure}} (rt.rs:59)
==52894==    by 0x62B0F7: _ZN3std9panicking3try7do_call17h173b0f0e2d5a0273E.llvm.9CC80E2B (panicking.rs:480)
==52894== 
==52894== 751 bytes in 47 blocks are definitely lost in loss record 41 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894== 
==52894== 866 bytes in 12 blocks are definitely lost in loss record 42 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894== 
==52894== 1,297 bytes in 13 blocks are indirectly lost in loss record 43 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894== 
==52894== 1,319 bytes in 10 blocks are indirectly lost in loss record 44 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894== 
==52894== 1,922 bytes in 16 blocks are indirectly lost in loss record 45 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894== 
==52894== 1,943 bytes in 16 blocks are indirectly lost in loss record 46 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894== 
==52894== 4,874 bytes in 599 blocks are definitely lost in loss record 47 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894== 
==52894== 8,055 bytes in 2,095 blocks are definitely lost in loss record 48 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894== 
==52894== 11,312 bytes in 5,656 blocks are definitely lost in loss record 49 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894==    by 0x2B56A1: <llamapun::data::ParagraphIterator<'iter> as core::iter::iterator::Iterator>::next (data.rs:234)
==52894==    by 0x2AF757: corpus_token_model::main (corpus_token_model.rs:51)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894==    by 0x62B0F7: {{closure}} (rt.rs:59)
==52894==    by 0x62B0F7: _ZN3std9panicking3try7do_call17h173b0f0e2d5a0273E.llvm.9CC80E2B (panicking.rs:480)
==52894== 
==52894== 19,322 bytes in 9,659 blocks are definitely lost in loss record 50 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894==    by 0x2B56A1: <llamapun::data::ParagraphIterator<'iter> as core::iter::iterator::Iterator>::next (data.rs:234)
==52894==    by 0x2AF757: corpus_token_model::main (corpus_token_model.rs:51)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894==    by 0x62B0F7: {{closure}} (rt.rs:59)
==52894==    by 0x62B0F7: _ZN3std9panicking3try7do_call17h173b0f0e2d5a0273E.llvm.9CC80E2B (panicking.rs:480)
==52894== 
==52894== 22,405 bytes in 1,749 blocks are definitely lost in loss record 51 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894== 
==52894== 26,760 bytes in 223 blocks are indirectly lost in loss record 52 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4E9F666: xmlNewDocNode (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4320A7: libxml::tree::Node::new (tree.rs:301)
==52894==    by 0x432261: libxml::tree::Node::mock (tree.rs:326)
==52894==    by 0x2B5CAC: <llamapun::dnm::DNM as core::default::Default>::default (mod.rs:46)
==52894==    by 0x2B5FAF: llamapun::dnm::DNM::new (mod.rs:114)
==52894==    by 0x2B56A1: <llamapun::data::ParagraphIterator<'iter> as core::iter::iterator::Iterator>::next (data.rs:234)
==52894==    by 0x2AF757: corpus_token_model::main (corpus_token_model.rs:51)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894==    by 0x62B0F7: {{closure}} (rt.rs:59)
==52894==    by 0x62B0F7: _ZN3std9panicking3try7do_call17h173b0f0e2d5a0273E.llvm.9CC80E2B (panicking.rs:480)
==52894==    by 0x641C1E: __rust_maybe_catch_panic (lib.rs:102)
==52894==    by 0x62EF9F: try<i32,closure> (panicking.rs:459)
==52894==    by 0x62EF9F: catch_unwind<closure,i32> (panic.rs:365)
==52894==    by 0x62EF9F: std::rt::lang_start_internal (rt.rs:58)
==52894== 
==52894== 28,385 bytes in 5,677 blocks are indirectly lost in loss record 53 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4E9F6A9: xmlNewDocNode (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4320A7: libxml::tree::Node::new (tree.rs:301)
==52894==    by 0x432261: libxml::tree::Node::mock (tree.rs:326)
==52894==    by 0x2B5CAC: <llamapun::dnm::DNM as core::default::Default>::default (mod.rs:46)
==52894==    by 0x2B5FAF: llamapun::dnm::DNM::new (mod.rs:114)
==52894==    by 0x2B56A1: <llamapun::data::ParagraphIterator<'iter> as core::iter::iterator::Iterator>::next (data.rs:234)
==52894==    by 0x2AF757: corpus_token_model::main (corpus_token_model.rs:51)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894==    by 0x62B0F7: {{closure}} (rt.rs:59)
==52894==    by 0x62B0F7: _ZN3std9panicking3try7do_call17h173b0f0e2d5a0273E.llvm.9CC80E2B (panicking.rs:480)
==52894==    by 0x641C1E: __rust_maybe_catch_panic (lib.rs:102)
==52894== 
==52894== 30,434 bytes in 749 blocks are definitely lost in loss record 54 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894== 
==52894== 36,195 bytes in 910 blocks are definitely lost in loss record 55 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894== 
==52894== 39,915 bytes in 151 blocks are indirectly lost in loss record 56 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894==    by 0x2B56A1: <llamapun::data::ParagraphIterator<'iter> as core::iter::iterator::Iterator>::next (data.rs:234)
==52894==    by 0x2AF757: corpus_token_model::main (corpus_token_model.rs:51)
==52894== 
==52894== 41,407 bytes in 665 blocks are definitely lost in loss record 57 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894== 
==52894== 47,240 bytes in 708 blocks are definitely lost in loss record 58 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894== 
==52894== 65,238 bytes in 525 blocks are indirectly lost in loss record 59 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894==    by 0x2B56A1: <llamapun::data::ParagraphIterator<'iter> as core::iter::iterator::Iterator>::next (data.rs:234)
==52894==    by 0x2AF757: corpus_token_model::main (corpus_token_model.rs:51)
==52894== 
==52894== 808,972 bytes in 6,394 blocks are definitely lost in loss record 60 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B86C2: llamapun::dnm::DNM::intermediate_node_create (mod.rs:283)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894==    by 0x2B56A1: <llamapun::data::ParagraphIterator<'iter> as core::iter::iterator::Iterator>::next (data.rs:234)
==52894==    by 0x2AF757: corpus_token_model::main (corpus_token_model.rs:51)
==52894== 
==52894== 822,039 (654,480 direct, 167,559 indirect) bytes in 5,454 blocks are definitely lost in loss record 61 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4E9F666: xmlNewDocNode (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4320A7: libxml::tree::Node::new (tree.rs:301)
==52894==    by 0x432261: libxml::tree::Node::mock (tree.rs:326)
==52894==    by 0x2B5CAC: <llamapun::dnm::DNM as core::default::Default>::default (mod.rs:46)
==52894==    by 0x2B5FAF: llamapun::dnm::DNM::new (mod.rs:114)
==52894==    by 0x2B56A1: <llamapun::data::ParagraphIterator<'iter> as core::iter::iterator::Iterator>::next (data.rs:234)
==52894==    by 0x2AF757: corpus_token_model::main (corpus_token_model.rs:51)
==52894==    by 0x2B3F51: std::rt::lang_start::{{closure}} (rt.rs:74)
==52894==    by 0x62B0F7: {{closure}} (rt.rs:59)
==52894==    by 0x62B0F7: _ZN3std9panicking3try7do_call17h173b0f0e2d5a0273E.llvm.9CC80E2B (panicking.rs:480)
==52894==    by 0x641C1E: __rust_maybe_catch_panic (lib.rs:102)
==52894==    by 0x62EF9F: try<i32,closure> (panicking.rs:459)
==52894==    by 0x62EF9F: catch_unwind<closure,i32> (panic.rs:365)
==52894==    by 0x62EF9F: std::rt::lang_start_internal (rt.rs:58)
==52894== 
==52894== 1,755,244 bytes in 26,791 blocks are definitely lost in loss record 62 of 62
==52894==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==52894==    by 0x4F11DAC: xmlStrdup (in /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.4)
==52894==    by 0x4324C5: libxml::tree::Node::get_content (tree.rs:470)
==52894==    by 0x2B65B9: llamapun::dnm::DNM::text_node_create (mod.rs:156)
==52894==    by 0x2B6541: llamapun::dnm::DNM::recurse_node_create (mod.rs:148)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B874D: llamapun::dnm::DNM::intermediate_node_create (mod.rs:286)
==52894==    by 0x2B6550: llamapun::dnm::DNM::recurse_node_create (mod.rs:150)
==52894==    by 0x2B64B0: llamapun::dnm::DNM::new (mod.rs:118)
==52894==    by 0x2B56A1: <llamapun::data::ParagraphIterator<'iter> as core::iter::iterator::Iterator>::next (data.rs:234)
==52894==    by 0x2AF757: corpus_token_model::main (corpus_token_model.rs:51)
==52894== 
==52894== LEAK SUMMARY:
==52894==    definitely lost: 3,441,871 bytes in 61,557 blocks
==52894==    indirectly lost: 167,559 bytes in 6,731 blocks
==52894==      possibly lost: 0 bytes in 0 blocks
==52894==    still reachable: 1,399 bytes in 28 blocks
==52894==         suppressed: 0 bytes in 0 blocks
dginev commented 6 years ago

I have patched the memory leak in https://github.com/KWARC/rust-libxml/pull/28

mostly .get_content being too careless. There are other one-off leaks in e.g. instantiating some of the parser objects, but those are by no means noticeable in a corpus run.

I have restarted the 08.2017 dataset run and am seeing constant memory use ~100 MB, which is the expected footprint for holding a single document in memory. After 2000 processed documents, no sign of the leaks, so I will close here and keep monitoring.