Mkr serializer

[rhmccullough]

Re-implementation of mKR serializer using raptor_serialize_turtle.c and raptor_turtle_writer.c context->emit_mkr = 1 for mkr, = 0 for turtle in raptor_serialize_turtle.c Ditto for new emit_mkr argument for _prefix and _base procedures in raptor_turtle_writer.c

[rhmccullough]

As far as I know, all tests pass now. Travis CI build fails on your turtle tests. My local computer build fails because of missing raptor2.h file in build. I assume that's because I have not pulled your raptor2 changes into my repository.

[dajobe]

I made several comments but the Travis CI build of this branch failed in the turtle tests, so you'll need to fix that too so that every existing test passes. https://travis-ci.org/dajobe/raptor/jobs/39962376

[rhmccullough]

I made several comments but the Travis CI build of this branch failed in the turtle tests, so you'll need to > fix that too so that every existing test passes. https://travis-ci.org/dajobe/raptor/jobs/39962376

I've been reading all your comments, but not responding to all of them.

I think your comments are very reasonable, and I will make the suggested changes.

The Travis CI failure is: no @prefix for the "a" in Turtle. Is that something that you changed in Raptor2?

[dajobe]

The travis failure is due to this pull request. The GIT head / master branch passes all the tests. So it must be something in your code change that caused it, presumably the code path for serializing turtle isn't entirely correct.

[rhmccullough]

re: missing raptor2.h. I did not run configure because ./autogen.sh ran configure (without arguments). So if I run configure after autogen.sh, the results are different?

[dajobe]

No the results are the same. Is there a src/raptor.h present or not?

[rhmccullough]

Yes, but src/raptor2.h is there when I get the error. It appears to me that the sequence ./autogen.sh ./configure make does not compile the scripts/build-formats.c program, which is when the error occurs.

[rhmccullough]

The more I look at this, the less sure I am about what's happening. I scrolled back thru my printout. Below where autogen.sh said it was running configure without arguments, autogen.sh said it was running configure with args --enable-maintainer-mode --disable-gtk-doc --enable-silent-rules. So apparently it is remembering args from previous builds, even though I did a make clean. And, apparently, I am supposed to ignore autogen.sh when it tells me to do make.

[dajobe]

Don't run configure after autogen.sh. autogen.sh runs configure for you with the correct arguments. If you watch the output of autogen.sh, you'll see it runs --enable-maintainer-mode. It's that which builds build-formats

[dajobe]

autogen.sh does not remember arguments, but it adds the default ones you need: --enable-maintainer-mode --disable-gtk-doc --enable-silent-rules This is detail you need not worry about. The raptor instructions have always said how to build from GIT using autogen.sh this way. Section 2.1 of http://librdf.org/raptor/INSTALL.html You can see all of this happening on any Travis CI run, which I've pointed to you earlier.

[rhmccullough]

re: raptor2.h error:

I reread http://librdf.org/raptor/INSTALL.html and I compared the next-to-last Travis Cl run to my latest build of Raptor 2 on Windows8.1/Cygwin. This is Raptor2 without mkr-serializer, from raptor-master.zip.

In my build, there is an error from a perl script in docs directory: /bin/sh: -chmod: command not found The attached PDF file is a screen shot of the last page of my build output. If the attachment doesn't get through to GitHub, I can send it to your regular email address.

[rhmccullough]

re: bug that caused failed tests 1) I made the error of inserting "_:" in raptor_turtle_emit_blank() before the call of raptor_turtle_writerbnodeid(). 2) Can all insertions of ":" can be avoided by using raptor_turtle_writer_bnodeid() in all the other places in raptor_serialize_turtle.c where a blank node is emitted?

re: merge conflicts I don't know what the conflicts are. When I cloned dajobe/raptor to a temporary directory on my local computer and did "git status", the only file listed was: modified: tests/ntriples/test.nt When I did "vi test.nt" there were no >>>> & <<<<< markers for conflicts. That file is one of about a dozen files which are listed in "Files to commit" on my local computer repository of branch mkr-serializer of rhmccullough/raptor. I have not done "git add" or "git commit" on any of these files, nor have I changed them. I have edited some of them, but have restored them all to the current versions in dajobe/raptor.

my most recent changes: I fixed failed test bug, and I made cosmetic changes to avoid trivial changes (e.g. empty lines) in dajobe/raptor. I have not finished all the changes that you suggested, because I want to understand the "merge conflicts" and unexplained "Files to commit" before proceeding.

re: missing raptor2.h and perl error My Cygwin perl is v5.14.2

[dajobe]

You should be able to switch to raptor_turtle_writer_bnodeid().

Maybe the conflicts are line endings; DOS <> Unix. git status should show something as a diference.

[dajobe]

GitHub now reports you can't automatically merge this, so I think you'll need to merge with origin/master and then fix any conflicts there.

[rhmccullough]

status report: 1) I deleted all the files that you said to remove. 2) I used Brendan Forster's work flow -- Git resolved the merge conflicts automatically. 3) As before, make aborts on missing -chmod and raptor2.h. I reconfigured with --disable-gtk-doc and make succeeds. 4) The next problem: all "check-eval-turtle" tests fail in tests/raptor-2013. I'd appreciate any hints on the source of this problem.

[rhmccullough]

Travis CI build fails immediately because Git removes execute permission from autogen.sh.

[rhmccullough]

After going back to my local repository, all tests pass. I have not yet replaced strcpy() & strcat().

[rhmccullough]

Now using raptor_stringbuffer. All tests pass. Travis CI build fails because of 644 mode of autogen.sh

[dajobe]

As well as autogen.sh there are several scripts/* directory mode changes. Please undo all mode changes.

[dajobe]

tests/ntriples/all-escape.nt also has some kind of change you need to undo

[dajobe]

I ran 'make distcheck' with this branch and it also fails. This is a standard check that the package builds when the build directory is different from the source directory.

In tests/mkr/Makefile.am it seems that the test write to '$$name.out' but the compare uses '$(srcdir)/$$name.out' . These should be the same. Also $$name.mkr should be $(srcdir)/$$name.mkr

After I made those changes I got more failures because rdfq-results.ttl assumes a specific base URI which won't be the case for me or anyone else:

Checking rdfq-results.ttl FAILED
../../utils/rapper -q -i turtle -o mkr ../../../tests/mkr/rdfq-results.ttl - > rdfq-results.out 2> rdfq-results.err
--- rdfq-results.out    2014-11-10 11:50:43.528446529 -0800
+++ ../../../tests/mkr/rdfq-results.mkr 2014-11-10 11:40:34.000000000 -0800
@@ -1,10 +1,10 @@
 @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> ;
 @prefix rs: <http://jena.hpl.hp.com/2003/03/result-set#> ;

-<file:///home/dajobe/dev/redland/raptor/raptor2-2.0.16/tests/mkr/rdfq-results.ttl> has
+<file:///cygdrive/c/Users/rhm/github/raptor/tests/mkr/rdfq-results.ttl> has
 rs:resultVariable = ["x", "y"],
 rs:size = [4] ;
-result is <file:///home/dajobe/dev/redland/raptor/raptor2-2.0.16/tests/mkr/rdfq-results.ttl> ;
+result is <file:///cygdrive/c/Users/rhm/github/raptor/tests/mkr/rdfq-results.ttl> ;
 result is relation with
 format = [value:1, value:2],
 meaning = {x := $1; y := $2;} ;

You'll need to pass one into the test; it's an argument to rapper. There are other test examples that use it - see BASE_URI definition and use in tests/turtle/Makefile.am

[dajobe]

I should say that we're close to getting it ready and I appreciate all the work you've done and learning you've had to do.

[rhmccullough]

I don't know how to do that. I think all these false changes started with the dos<->unix problem. But I have been unable to change the mode. All of my current files have the execute bits set. But Git keeps changing my 755 mode to 644 mode.

[rhmccullough]

I will have another look at that one.

I noticed that one of your dos-test files has a CR return at the end of the line, and Git removed. I had to put two CRs at the end of the line, and then Git only removed one CR. It still lists that file as a File to be committed every time -- I uncheck it so Git won't remove the CR again.

[rhmccullough]

I never ran 'make distcheck'. I deleted everything from tests/mkr/Makefile.am except the exact serialization test. I guess I have to put some things back in the Makefile.

I presume anything related to docs would also fail, since a build with --enable-gtk-doc always fails on Windows/Cygwin.

[rhmccullough]

1) baseuri problem fixed. 2) I asked GitHub gurus for help on execute bits. 3) When I run 'make distcheck' it fails because gtk-doc fails.

[rhmccullough]

4) tests/ntriples/all-escape.nt is another dos<->unix problem. I tried the same trick of adding 2 CRs. Let me know if it worked this time (possibly sensitive to odd/even byte count).

[dajobe]

With this code plus some commits I made to revert the tests/ntriples/all-escape.nt and the script modes, this branch / Pull Request now passes 'make distcheck'. So we're ready to go, apart from some documentation updates.

I propose we could make this a new pull request with just 1 commit rather than all the to & Fro of the 70 commits here.

[rhmccullough]

I'm OK with new pull request.

I see some complicating factors in my current "Files to commit". 1) tests/trig/bug-584-dos.trig (CR problem) 2) raptor2-2.0.16/* (new directory) But I expect I can just ignore the CR problem and delete the new directory.

How about waiting for one more commit on PR#17 for the error checking changes?

[dajobe]

ok, I'll wait for your commit and will prepare a clean PR after that

[rhmccullough]

We're slightly out of sync here. Before I saw this note, I added a new one: I don't think there is any error checking problem.

Let me know if you disagree.

I did delete the new directory from my local repository.

[rhmccullough]

gtk-doc dependency errors 0) autogen.sh runs configure --enable-gtk-doc 1) gcc build-formats.c fails because it does not contain $(DEFAULT_INCLUDES) 2) when I change to "../src/raptor2.h" it fails because of many undefined references, mostly raptoriostream functions. see screen shot: http://rhmccullough.net/raptor.error.Nov12.pdf

[dajobe]

1. I generally run ./autogen.sh --disable-gtk-doc when I want things to go faster (obviously 'make distcheck' won't work in this case)
2. I can see I need to add AM_CPPFLAGS to utils/Makefile.am to get the right headers
3. However that is unrelated to this issue because that is about missing link libraries; your compile line does not link with libraptor2 which is very odd since if build-formats is compiled at all, it includes build_formats_LDADD

[rhmccullough]

3) build-formats.c has never compiled on Windows/Cygwin. Maybe because of that strange: "-chmod not found" error in docs directory? Dick

                  =

[rhmccullough]

3) you mentioned "libraptor2". I did not see that name anywhere in my "make" output. Here are the missing lines from the top of the screen shot, when docs directory is entered. Making all in docs make[1]: Entering directory '/cygdrive/c/Users/rhm/github/raptor/docs' DOC Preparing build perl ../scripts/fix-gtkdoc-header.pl < ../src/raptor2.h > raptor-fake.h DOC Scanning header files DOC Introspecting gobjects DOC Rebuilding template files ./raptor2-unused.txt:1: warning: 7 unused declarations.They should be added to raptor2-sections.txt in the appropriate place. DOC Building XML /bin/sh: -chmod: command not found ./../src/raptor_general.c:121: warning: Parameter described in source code comment block but does not exist. MACRO: raptor_new_world Parameter: version_decimal. ./raptor2-unused.txt:1: warning: 7 unused declarations.They should be added to raptor2-sections.txt in the appropriate place. cd ../scripts && make build-formats make[2]: Entering directory '/cygdrive/c/Users/rhm/github/raptor/scripts' gcc -Wall -Wc++-compat -Wextra -Wunused -Waggregate-return -Wbad-function-cast -Wcast-align -Wdeclaration-after-statement -Wdisabled-optimization -Wdiv-by-zero -Wendif-labels -Werror-implicit-function-declaration -Wfloat-equal -Wframe-larger-than=4096 -Winit-self -Winline -Wmissing-declarations -Wmissing-format-attribute -Wmissing-noreturn -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls -Wshadow -Wsign-compare -Wstrict-prototypes -Wswitch-enum -Wundef -Wunreachable-code -Wunsafe-loop-optimizations -Wwrite-strings -Wno-conversion -Wno-format-nonliteral -Wno-missing-field-initializers -Wno-sign-conversion -Wno-system-headers -Wno-unused-parameter -DRAPTOR_INTERNAL=1 -I/usr/include/libxml2 build-formats.c -o build-formats

[rhmccullough]

3) correction: right at the end of the src directory, I found CCLD libraptor2.la libtool: link: warning: undefined symbols not allowed in i686-pc-cygwin shared libraries

[rhmccullough]

I know that. Git would not let me change the mode of scripts/install-bison3.sh, so I changed the mode with the Travis CI script. Now bison does get updated to bison3 and Travis CI executes most of the checks successfully. Currently, all "eval Turtle" tests fail. I don't know why. I saw that once in my testing, but on the next test run I did, all the "eval Turtle" tests passed.

[dajobe]

Ok. In this case, I'm guessing scripts/rdfcompare is not executable so the test comparisons fail.

[rhmccullough]

I'm guessing this result is also related to the gtk-doc problem. I'm going to run my tests again, starting with make clean ./autogen.sh --disable-gtk-doc and see if all tests pass.

[rhmccullough]

Using "--disable-gtk-doc" on Windows/Cygwin, "make check" passes all tests !

[rhmccullough]

Just saw your note re: scripts/rdfcompare. I will add that to the Travis CI script, let CI try again. PS. The file diffs also show that other executables in scripts are still 644.

[rhmccullough]

Travis CI passed. Bottom line: the "core.filemode=false" only prevents problems on future executable files. After a round trip to GitHub for Windows, GitHub won't let you change from 644 back to 755. It aborts any commit because the file has "not changed".

[rhmccullough]

A thought re "-chmod not found" error in docs directory. If the script is actually trying to execute "-chmod", you will get that error because the command is "chmod" not "-chmod". PS. If you try to execute "exec chmod", the Terminal program exits.

[rhmccullough]

rasqal dependency error Using the Nov/13/2014 .gz files from librdf.org, rasqal "./configure" concludes: checking for RAPTOR2... no configure: error: Raptor is not installed - see http://librdf.org/raptor/ to get 2.0.7 or newer FYI: raptor "make check" and "make installcheck" pass; "rapper -version" is 2.0.15

[rhmccullough]

rasqal "./autogen.sh" also says Raptor not installed.

[rhmccullough]

/usr/local/lib/pkgconfig/raptor2.pc contains prefix=/usr/local exec_prefix=${prefix} libdir=${exec_prefix}/lib includedir=${prefix}/include/raptor2

Name: Raptor RDF Syntax Library Description: RDF Syntax Library Version: 2.0.15 Libs: -L${libdir} -lraptor2 Libs.private: -L/usr/lib -lxml2 -lz -liconv -lm Cflags: -I${includedir} Requires:

[rhmccullough]

As I continued reading, I got to config.log, where I was advised to export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig Then "./configure", "make" and "make install" succeeded. I'm surprised the auto programs did not supply this path, since installs default to /usr/local.

[rhmccullough]

FYI: the Windows/Cygwin "make" of rasqal includes this warning: /bin/sh ../libtool --tag=CC --mode=link gcc -I/usr/local/include/raptor2 -I/usr/include/uuid -g -O2 -DSV_CONFIG -I../libsv -DMTWIST_CONFIG -I../libmtwist -g -O2 -version-info 3:0:0 -o librasqal.la -rpath /usr/local/lib rasqal_algebra.lo rasqal_expr.lo rasqal_expr_evaluate.lo rasqal_expr_datetimes.lo rasqal_expr_numerics.lo rasqal_expr_strings.lo rasqal_general.lo rasqal_query.lo rasqal_query_results.lo rasqal_engine.lo rasqal_raptor.lo rasqal_literal.lo rasqal_formula.lo rasqal_graph_pattern.lo rasqal_map.lo rasqal_feature.lo rasqal_result_formats.lo rasqal_xsd_datatypes.lo rasqal_decimal.lo rasqal_datetime.lo rasqal_rowsource.lo rasqal_format_sparql_xml.lo rasqal_variable.lo rasqal_rowsource_empty.lo rasqal_rowsource_union.lo rasqal_rowsource_rowsequence.lo rasqal_query_transform.lo rasqal_row.lo rasqal_engine_algebra.lo rasqal_triples_source.lo rasqal_rowsource_triples.lo rasqal_rowsource_filter.lo rasqal_rowsource_sort.lo rasqal_engine_sort.lo rasqal_rowsource_project.lo rasqal_rowsource_join.lo rasqal_rowsource_graph.lo rasqal_rowsource_distinct.lo rasqal_rowsource_groupby.lo rasqal_rowsource_aggregation.lo rasqal_rowsource_having.lo rasqal_rowsource_slice.lo rasqal_rowsource_bindings.lo rasqal_rowsource_service.lo rasqal_row_compatible.lo rasqal_format_table.lo rasqal_query_write.lo rasqal_format_json.lo rasqal_format_sv.lo rasqal_format_html.lo rasqal_format_rdf.lo rasqal_rowsource_assignment.lo rasqal_update.lo rasqal_triple.lo rasqal_data_graph.lo rasqal_prefix.lo rasqal_solution_modifier.lo rasqal_projection.lo rasqal_bindings.lo rasqal_service.lo rasqal_dataset.lo rasqal_random.lo rasqal_digest.lo rasqal_iostream.lo rasqal_regex.lo snprintf.lo rasqal_double.lo rasqal_ntriples.lo sparql_lexer.lo sparql_parser.lo rasqal_digest_md5.lo rasqal_digest_sha1.lo -L/usr/local/lib -lraptor2 -lxml2 -lz -liconv -lm -luuid -lgmp ../libsv/libsv.la ../libmtwist/libmtwist.la -L/usr/local/lib -lraptor2 -lxml2 -lz -liconv -lm -luuid -lgmp

* Warning: This system can not link to static lib archive /usr/local/lib/libraptor2.la. * I have the capability to make that library automatically link in when * you link to this library. But I can only do this if you have a * shared version of the library, which you do not appear to have. libtool: link: warning: undefined symbols not allowed in i686-pc-cygwin shared libraries

[dajobe]

This is ready for me to put into one new PR commit and merge. We can address other issues after that.

[rhmccullough]

OK. What do you want me to do?