search

bbcarchdev / twine

An RDF workflow engine
https://bbcarchdev.github.io/twine/
Apache License 2.0
8 stars 2 forks source link

twine_sparql_put() struggles with large graphs #14

Open nevali opened 9 years ago

nevali commented 9 years ago

e.g., http://examples.acropolis.org.uk/progsubjects/index.ttl

Mid-processing backtrace suggests:

#0  0x00007f7c3c04c212 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f7c3c04ea70 in malloc () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f7c3f465c9f in ?? () from /usr/lib/libraptor2.so.0
#3  0x00007f7c3f465cf9 in ?? () from /usr/lib/libraptor2.so.0
#4  0x00007f7c3f465c49 in ?? () from /usr/lib/libraptor2.so.0
#5  0x00007f7c3f465cf9 in ?? () from /usr/lib/libraptor2.so.0
#6  0x00007f7c3f465cf9 in ?? () from /usr/lib/libraptor2.so.0
#7  0x00007f7c3f465cf9 in ?? () from /usr/lib/libraptor2.so.0
#8  0x00007f7c3f465cf9 in ?? () from /usr/lib/libraptor2.so.0
#9  0x00007f7c3f46662e in raptor_avltree_add () from /usr/lib/libraptor2.so.0
#10 0x00007f7c3f458281 in raptor_new_uri_from_counted_string ()
   from /usr/lib/libraptor2.so.0
#11 0x00007f7c3f45f72f in raptor_new_term_from_uri_string ()
   from /usr/lib/libraptor2.so.0
#12 0x00007f7c3ca06e53 in librdf_node_decode () from /usr/lib/librdf.so.0
#13 0x00007f7c3ca0808c in librdf_statement_decode2 () from /usr/lib/librdf.so.0
#14 0x00007f7c3ca0a578 in ?? () from /usr/lib/librdf.so.0
#15 0x00007f7c3c9fd97c in ?? () from /usr/lib/librdf.so.0
#16 0x00007f7c3c9fdb85 in librdf_stream_end () from /usr/lib/librdf.so.0
#17 0x00007f7c3ca0a5f3 in ?? () from /usr/lib/librdf.so.0
#18 0x00007f7c3ca0b929 in ?? () from /usr/lib/librdf.so.0
#19 0x00007f7c3c9ffdc4 in ?? () from /usr/lib/librdf.so.0
#20 0x00007f7c3f470eaa in ?? () from /usr/lib/libraptor2.so.0
#21 0x00007f7c3f471fd1 in turtle_parser_parse () from /usr/lib/libraptor2.so.0
#22 0x00007f7c3f472332 in ?? () from /usr/lib/libraptor2.so.0
#23 0x00007f7c3c9ff90d in ?? () from /usr/lib/librdf.so.0
#24 0x00007f7c3c9ffa90 in ?? () from /usr/lib/librdf.so.0
#25 0x00007f7c3fb22eec in twine_rdf_model_parse_base ()
   from /usr/lib/libtwine.so
#26 0x00007f7c3fb229be in twine_sparql_put () from /usr/lib/libtwine.so
#27 0x00007f7c3fb22b56 in twine_sparql_put_stream () from /usr/lib/libtwine.so
#28 0x00007f7c385f7d34 in ?? () from /usr/lib/twine/rdf.so
#29 0x00007f7c3fb21ece in twine_plugin_process () from /usr/lib/libtwine.so
#30 0x0000000000401e11 in ?? ()
#31 0x00007f7c3bff2ead in __libc_start_main ()
   from /lib/x86_64-linux-gnu/libc.so.6
#32 0x0000000000402091 in ?? ()
#33 0x00007fffcd7d2778 in ?? ()
#34 0x000000000000001c in ?? ()
#35 0x0000000000000005 in ?? ()
#36 0x00007fffcd7d2def in ?? ()
#37 0x00007fffcd7d2df5 in ?? ()
#38 0x00007fffcd7d2df8 in ?? ()
#39 0x00007fffcd7d2e0c in ?? ()
#40 0x00007fffcd7d2e0f in ?? ()
#41 0x0000000000000000 in ?? ()
nevali commented 9 years ago

Admittedly, the buffer ends up being 13MB as N-Triples, but it parses as N-Quads far faster than this intermediate parsing step (which can likely be removed)