search

jeffreykegler / Marpa--R2

Parse any language you can describe in BNF
https://jeffreykegler.github.io/Marpa-web-site/
Other
157 stars 23 forks source link

fails to build on i386 and s390x #150

Closed jonassmedegaard closed 9 years ago

jonassmedegaard commented 10 years ago

Debian packaged Marpa::R2 builds fine in general, but i386 and s390x consistently fail: https://buildd.debian.org/status/logs.php?pkg=libmarpa-r2-perl

NB! Debian packaging uses patched code to compile the C code separately, so as to apply configure and make flags in accordance to policies of Debian. Patch is here: https://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libmarpa-r2-perl.git;a=blob;f=debian/patches/2001_libmarpa_external_linkage.patch

I fully understand if you have no desire to support my ugly patching, but would then appreciate help finding an alternative approach which supports hardening flags and multiarch support.

jonassmedegaard commented 10 years ago

Failures seem unrelated to the Debian patch: Same archs also failed before Debian packaging release 2.086000~dfsg-2 when the patch was applied: https://buildd.debian.org/status/logs.php?pkg=libmarpa-r2-perl

jeffreykegler commented 10 years ago

More food for thought.

The patch you reference is actually dangerous in some respects. For inclusion in Perl, libmarpa should not be built in the distribution directory. I want to keep libmarpa_dist vanilla, and the Perl build is very kludge-ish -- anything but vanilla, and likely to get worse.

Second, the version is in the library name because libmarpa does not guarantee either backward or forward compatibility. This is the GNU-recommended way to enforce linking with one and only one version of a library.

jonassmedegaard commented 10 years ago

Quoting Jeffrey Kegler (2014-07-02 18:55:23)

More food for thought.

The patch you reference is actually dangerous in some respects. For inclusion in Perl, libmarpa should not be build in the distribution directory. I want to keep this vanilla, and the Perl build is very kludge-ish -- anything but vanilla, and likely to get worse.

Second, the version is in the library name is to enforce the curent libmarpa on backward compatibility -- there isn't any.

Above comments seem tied to my patch.

Please for this issue disregard my patch and instead look at earlier build logs for the versions efore that patch was applied.

Thinking out loud, since these issues are all with the developer's facility, I might move the code for creating shared libraries into a non-Debian directory.

Not sure what you mean by "non-Debian directory". Is that perhaps also tied to second-guessing how my patch might have wreaked havoc?

jeffreykegler commented 10 years ago

Are the policies were discussing available on-online somewhere. I searched, and was able to find the DFSG, but nothing about guidelines and requirements for builds.

jeffreykegler commented 10 years ago

OK. Sorry to have gotten distracted. I will ignore the patch.

jeffreykegler commented 10 years ago

There is a MARPA_USE_PERL_AUTOCONF, which invokes an alternative build procedure using CPAN's Config::Autoconf instead of the GNU autoconf. If our issues are autoconf-related, it is likely to make them go away, and this is how the Marpa community has been handling a lot of platforms where autoconf proves problematic.

Example of usage

MARPA_USE_PERL_AUTOCONF=1 ./Build disttest;
jeffreykegler commented 10 years ago

Looking at the i386 build log: https://buildd.debian.org/status/fetch.php?pkg=libmarpa-r2-perl&arch=i386&ver=2.086000~dfsg-3&stamp=1404316889 -- these lines look like an issue.

   Can't find source file /«PKGBUILDDIR»/libmarpa_build/config.h for up-to-date check at /usr/share/perl/5.18/Module/Build/Base.pm line 5635.
   Can't find source file /«PKGBUILDDIR»/libmarpa_build/marpa.h for up-to-date check at /usr/share/perl/5.18/Module/Build/Base.pm line 5635.
   Can't find source file /«PKGBUILDDIR»/libmarpa_build/marpa_slif.h for up-to-date check at /usr/share/perl/5.18/Module/Build/Base.pm line 5635.
   Can't find source file /«PKGBUILDDIR»/libmarpa_build/marpa_codes.c for up-to-date check at /usr/share/perl/5.18/Module/Build/Base.pm line 5635.

They say that the Perl build logic is not finding these files where it expects them to be. By the way, I note the Unicode in the directory name. I've heard somewhere this can be problematic in Perl.

jeffreykegler commented 10 years ago

Looking at the s390 build log: https://buildd.debian.org/status/fetch.php?pkg=libmarpa-r2-perl&arch=s390x&ver=2.086000~dfsg-3&stamp=1404317159

These lines are very, very worrisome -- they seem to indicate that the compiler and the header files disagree on whether int's are 32-bit or 64-bit:

    lib/Marpa/R2.xs: In function 'XS_Marpa__R2__Thin__SLR_g1_lexeme_complete':
    /usr/lib/perl/5.18/CORE/perl.h:1818:26: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
        #  define INT2PTR(any,d) (any)(d)
                              ^
    lib/Marpa/R2.xs:6091:10: note: in expansion of macro 'INT2PTR'
             INT2PTR (void *, lexeme_length));
             ^

Here is what seems to be the same problem, in a CPANtesters setup: http://www.cpantesters.org/cpan/report/93a62eb0-f94e-11e3-82c2-bc3ea1235623

In the CPANtesters case, the output is more verbose, and there is a suggestion of what the issue is. Perl is built with "intsize=4" (4 bytes, that is, or 32 bits) as reported by "perl -V". This seems consistent with the "ccflags" Config value. But the flags from the environment as reported by "perl -V" specify "USE_64_BIT_INT ", a conflict which you'd expect to cause trouble.

Libmarpa itself is using autoconf, which builds a 32-bit library, which seems consistent with the way that the tester's Perl was actually built.

jeffreykegler commented 10 years ago

Also, I've put out word about these issues on the Marpa IRC channel, and the Marpa mailing list. Several of the people out there know a good deal about Marpa builds -- depending on the issue, often more than I do.

jonassmedegaard commented 10 years ago

Quoting Jeffrey Kegler (2014-07-02 19:41:59)

Are the policies were discussing available on-online somewhere. I searched, and was able to find the DFSG, but nothing about guidelines and requirements for builds.

Well, generally Debian need to be able to pass custom build flags.

I recommend against trying to hardcode the concrete build flags currently used in Debian - derivatives (like Ubuntu) have different policies and sometimes need different build flags applied.

Specifically, Debian currently apply the following build flags:

CFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security CPPFLAGS=-D_FORTIFY_SOURCE=2 CXXFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security FCFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 FFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 GCJFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 LDFLAGS=-Wl,-z,relro OBJCFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security OBJCXXFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security

...and commonly (but not always) pass these to configure:

--build=$(DEB_BUILD_GNU_TYPE) --prefix=/usr --includedir="\${prefix}/include" --mandir="\${prefix}/share/man" --infodir="\${prefix}/share/info" --sysconfdir=/etc --localstatedir=/var --libexecdir="\${prefix}/lib/$(DEB_SOURCE_PACKAGE)" --disable-maintainer-mode --disable-dependency-tracking --disable-silent-rules

(the last three options are harmless when unsupported)

Specifically for libmarpa (with the patch applied, allowing me to pass arguments at all) these are additionally passed (for now, until I maybe later switch to maintaining a shared library):

--with-pic --disable-shared

NB! Beware that most of above applies only to the most recent builds where my patch is applied. Build before that has fewer Debian options applied, but also those builds are less verbose (I'd be happy to learn how to enable verbose building of MakeMaker-wrapped autotools).

jonassmedegaard commented 10 years ago

Quoting Jonas Smedegaard (2014-07-03 00:58:35)

Quoting Jeffrey Kegler (2014-07-02 19:41:59)

Are the policies were discussing available on-online somewhere. I searched, and was able to find the DFSG, but nothing about guidelines and requirements for builds.

Some documentation of hardening flags are here: https://wiki.debian.org/Hardening

...and on Multiarch (mainly relevant for shared libraries): https://wiki.debian.org/Multiarch/Implementation

jonassmedegaard commented 10 years ago

Quoting Jeffrey Kegler (2014-07-02 19:52:22)

There is a MARPA_USE_PERL_AUTOCONF, which invokes an alternative build procedure using CPAN's Config::Autoconf instead of the GNU autoconf. If our issues are autoconf-related, it is likely to make them go away, and this is how the Marpa community has been handling a lot of platforms where autoconf proves problematic.

Example of usage

MARPA_USE_PERL_AUTOCONF=1 ./Build disttest;

Thanks, but the needs for Debian is not yet another way to build things, but a way to not build but let us build separately (with plain autotools but called from our helper tools instead of your Perl wrapper, so that all flags are properly applied).

jonassmedegaard commented 10 years ago

Quoting Jeffrey Kegler (2014-07-02 20:07:42)

Looking at the i386 build log: https://buildd.debian.org/status/fetch.php?pkg=libmarpa-r2-perl&arch=i386&ver=2.086000~dfsg-3&stamp=1404316889 -- these lines look like an issue.

Can't find source file /«PKGBUILDDIR»/libmarpa_build/config.h for up-to-date check at /usr/share/perl/5.18/Module/Build/Base.pm line 5635. Can't find source file /«PKGBUILDDIR»/libmarpa_build/marpa.h for up-to-date check at /usr/share/perl/5.18/Module/Build/Base.pm line 5635. Can't find source file /«PKGBUILDDIR»/libmarpa_build/marpa_slif.h for up-to-date check at /usr/share/perl/5.18/Module/Build/Base.pm line 5635. Can't find source file /«PKGBUILDDIR»/libmarpa_build/marpa_codes.c for up-to-date check at /usr/share/perl/5.18/Module/Build/Base.pm line 5635.

They say that the Perl build logic is not finding these files where it expects them to be.

Please beware that 2.086000~dfsg-3 which you are looking at above has my patch applied.

2.086000~dfsg-1 and earlier does not include my patch, does not include above warning, but still fails with seemingly same error.

Suggestions on how to enable more verbose builds is quite welcome.

By the way, I note the Unicode in the directory name. I've heard somewhere this can be problematic in Perl.

I guess you are referring to the "«" and "»" characters. Those do not exist in the actual build environment, but are the result of normalizing the build logs, as noted at the top of the file:

I: NOTICE: Log filtering will replace 'build/libmarpa-r2-perl-nXsZXJ/libmarpa-r2-perl-2.086000~dfsg' with '«PKGBUILDDIR»' I: NOTICE: Log filtering will replace 'build/libmarpa-r2-perl-nXsZXJ' with '«BUILDDIR»' I: NOTICE: Log filtering will replace 'var/lib/schroot/mount/sid-i386-sbuild-51fa3310-c262-4223-a386-1456c157671c' with '«CHROOT»'

  • Jonas

jeffreykegler commented 10 years ago

I am starting to study the Debian packaging docs -- please bear with me in the meantime if I show lack of understanding.

Re the MARPA_USE_PERL_AUTOCONF, I think the existence of autoconf here may be misleading. This code is destined to be staticly linked into an XS module, in which context I assume the usual Perl XS-hackery is allowed. That is, most Perl XS modules on Debian I assume are built with no use of autotools, so insisting on it here would make Marpa::R2 a very special, and disadvantaged, case.

Confusing the issue is that libmarpa is ultimately intended to be standalone, and anticipating that day, carries a lot of the autotools stuff with it. I understand that, if a libmarpa shared library were to be prepared for Debian, new and different restrictions would apply.

But, if I read things correctly, if we insist on using autotools to build libmarpa for static linking with the Perl XS, we would be saying that, because libmarpa tried to use autotools, it must use autotools. This, if I understand things correctly, would impose a burden on building Marpa in an XS environment which is imposed on few or no other XS modules.

jeffreykegler commented 10 years ago

Re MARPA_USE_PERL_AUTOCONF: if you agree this is an acceptable way to go, I can take on the task of changing the Perl code to pull in the special flags, checking with you that it fits Debian guidelines. I understand that the right way is not to hardcode a set of flags, but to invoke the Debian utilties to get Debian's current idea of the correct flags.

jonassmedegaard commented 10 years ago

Quoting Jeffrey Kegler (2014-07-03 02:36:33)

Re MARPA_USE_PERL_AUTOCONF: if you agree this is an acceptable way to go, I can take on the task of changing the Perl code to pull in the special flags, checking with you that it fits Debian guidelines. I understand that the right way is not to hardcode a set of flags, but to invoke the Debian utilties to get Debian's current idea of the correct flags.

If you tell me that the existence of libmarpa and the fact that it happens to use autotools are both internal artifacts of your coding style, not something I should mess with at all, then yes, I will roll back my packaging code to behave as they did in 2.086000~dfsg-1, which means a) it still fails on i386 and s390x, and b) I lack knowledge on how to make build more verbose, so that I can both c) verify that custom values for common build flags are correctly applied, and d) inspect warnings.

The build flags relevant for your code are CFLAGS CPPFLAGS and LDFLAGS.

You should not need to follow Debian coding guidelines specifically, those build flags should be common across all C programming.

You should not need to wait for me to change packaging to continue chase this bug: Simply look at the build logs for 2.086000~dfsg-1 and earlier at https://buildd.debian.org/status/logs.php?pkg=libmarpa-r2-perl .

If you are telling me that I should enable MARPA_USE_PERL_AUTOCONF to compile properly on i386 and s390x then I will do that - but am then wondering why that is not enabled by default.

It seems more sensible to me for the C library libmarpa to be released and maintained as an independent library (static-only at the very beginning, and shared but without a stable SONAME until deemed stable), and the perl Marpa::R2 code continuing to include that same libmarpa library just as now (to ease use directly from CPAN) but also adjusted to link against a separately build and installed libmarpa library - i.e. a single flag that ensures the included libmarpa library is not touched at all, unlike the current odd choice of build-and-test-but-not-install-if shared.

jeffreykegler commented 10 years ago

Before going into more practical suggestions, I hope you'll let me give some background. I think it might help you understand my approach, and why the build procedure came into its, admittedly, "odd" state. My suggested solutions are open to question, but they are based on experience.

Libmarpa's attempts to do something no other Perl XS module does as far as I know. It's a large body of code which, eventually intended to be a separate library, but whose only audience and test bed, at present, is as a PERL XS module. There are large codebases inside other Perl XS modules, but these make no attempt to some day be separable from Perl.

Trying to do this committed me to a "compromise" between two notoriously difficult and largely incompatible build methodologies, and the result was not very pretty. For what it's worth, my XS codebase has been studied and recommended as a model. I myself simply view it as an attempt to cope.

The Marpa team has often encountered problems on platforms with autoconf, and the MARPA_USE_PERL_AUTOCONF workaround is the idea of Jean-Damien Durand. It is the way we solve autoconf build problems on new targets. Trying to get autoconf to work might be possible, but experience has taught us that giving up and working around it is easiest.

Why not give up on autoconf? Jean-Damien, of the Marpa team, in fact suggests this. I have resisted, because it means abandoning testing of autoconf builds, and I still want Marpa to be able to build and install Libmarpa separately from Perl someday.

I'll hope you'll forgive this long digression from the immediate practical issues, but I thought knowing the reasons for my ideas would be helpful. They are offered, not as "policy", but as a best guess based on experience.

jeffreykegler commented 10 years ago

This (that is, to release Libmarpa as a separate library and allow Marpa::R2 to be configured to use it) is a very reasonable alternative.
But it would make the Debian community a "guinea pig" for an untested configuration. That is, right now Libmarpa testing relies on CPANtesters, and CPANtesters is only able to handle Perl dependencies -- it cannot deal with the need for an outside library.

But perhaps Debian's own packaging and test procedures could be considered sufficient safeguard, and Libmarpa can cut the umbilical cord to CPANtesters.

I am interested in your thoughts on this,

jeffrey

On 07/03/2014 01:07 AM, Jonas Smedegaard wrote:

It seems more sensible to me for the C library libmarpa to be released and maintained as an independent library (static-only at the very beginning, and shared but without a stable SONAME until deemed stable), and the perl Marpa::R2 code continuing to include that same libmarpa library just as now (to ease use directly from CPAN) but also adjusted to link against a separately build and installed libmarpa library - i.e. a single flag that ensures the included libmarpa library is not touched at all, unlike the current odd choice of build-and-test-but-not-install-if shared.

jeffreykegler commented 10 years ago

I believe the s390 and i386 issues may be different problems.

Specifically on the i386 build problem: The relevant error messages may be

Bailout called.  Further testing stopped:  Could not load Marpa::R2
FAILED--Further testing stopped: Could not load Marpa::R2

This usually means there was some issue with the file blib/arch/auto/Marpa/R2/R2.so. If it exists and looks like an intact shareable library, I would next check it for undefined symbols. The undefined symbols in it should only be those for system libraries and the Perl core, nothing else. In particular, no Marpa symbols should be indicated as undefined.

jonassmedegaard commented 10 years ago

Quoting Jeffrey Kegler (2014-07-03 19:00:34)

This (that is, to release Libmarpa as a separate library and allow Marpa::R2 to be configured to use it) is a very reasonable alternative. But it would make the Debian community a "guinea pig" for an untested configuration. That is, right now Libmarpa testing relies on CPANtesters, and CPANtesters is only able to handle Perl dependencies -- it cannot deal with the need for an outside library.

But perhaps Debian's own packaging and test procedures could be considered sufficient safeguard, and Libmarpa can cut the umbilical cord to CPANtesters.

I am interested in your thoughts on this,

I strongly believe Debian infrastructure is robust enough to catch linking issues across separately maintained projects.

Hit me!

P.S. Thanks for the historical insight. I kinda guessed it was something along those lines, but great to get it from you.

jonassmedegaard commented 10 years ago

Quoting Jeffrey Kegler (2014-07-03 19:33:05)

I believe the s390 and i386 issues may be different problems.

Specifically on the i386 build problem: The relevant error messages may be

Bailout called. Further testing stopped: Could not load Marpa::R2 FAILED--Further testing stopped: Could not load Marpa::R2

This usually means there was some issue with the file blib/arch/auto/Marpa/R2/R2.so. If it exists and looks like an intact shareable library, I would next check it for undefined symbols. The undefined symbols in it should only be those for system libraries and the Perl core, nothing else. In particular, no Marpa symbols should be indicated as undefined.

To inspect those things I need to gain console access to a developer machine for those archs. That is not a routine task for me, so will require some attention that I will lack for the next week due to travel¹.

Could you please try help me locate ways to expose more details during build? It seems to me that the test target is doing some linking, and I would like to know exactly what is the commandline it is invoking.

I can easily add options to turn builds more verbose also while traveling, so that could win us some time.

¹) I lead a pilot project at the European Parliament in using GnuPG: https://wiki.debian.org/DebianParl/GreensEFA

jeffreykegler commented 10 years ago

In which case, rather than bother with half-way approaches, I will start to set Marpa::R2 up to link with separate libraries.

A separate Libmarpa is something I've really wanted to do, but haven't had a sufficient audience for. Debian is certainly a big enough audience.

-- jeffrey

On 07/03/2014 11:24 AM, Jonas Smedegaard wrote:

Quoting Jeffrey Kegler (2014-07-03 19:00:34)

This (that is, to release Libmarpa as a separate library and allow Marpa::R2 to be configured to use it) is a very reasonable alternative. But it would make the Debian community a "guinea pig" for an untested configuration. That is, right now Libmarpa testing relies on CPANtesters, and CPANtesters is only able to handle Perl dependencies -- it cannot deal with the need for an outside library.

But perhaps Debian's own packaging and test procedures could be considered sufficient safeguard, and Libmarpa can cut the umbilical cord to CPANtesters.

I am interested in your thoughts on this,

I strongly believe Debian infrastructure is robust enough to catch linking issues across separately maintained projects.

Hit me!

  • Jonas

P.S. Thanks for the historical insight. I kinda guessed it was something along those lines, but great to get it from you.

  • Jonas Smedegaard - idealist & Internet-arkitekt
  • Tlf.: +45 40843136 Website: http://dr.jones.dk/

[x] quote me freely [ ] ask before reusing [ ] keep private

— Reply to this email directly or view it on GitHub https://github.com/jeffreykegler/Marpa--R2/issues/150#issuecomment-47966624.

jeffreykegler commented 10 years ago

Have a pleasant and successful trip. I'll need to do some refactoring of Marpa, so I probably have a week's worth of homework to do.

-- jeffrey

On 07/03/2014 11:32 AM, Jonas Smedegaard wrote:

Quoting Jeffrey Kegler (2014-07-03 19:33:05)

I believe the s390 and i386 issues may be different problems.

Specifically on the i386 build problem: The relevant error messages may be

Bailout called. Further testing stopped: Could not load Marpa::R2 FAILED--Further testing stopped: Could not load Marpa::R2

This usually means there was some issue with the file blib/arch/auto/Marpa/R2/R2.so. If it exists and looks like an intact shareable library, I would next check it for undefined symbols. The undefined symbols in it should only be those for system libraries and the Perl core, nothing else. In particular, no Marpa symbols should be indicated as undefined.

To inspect those things I need to gain console access to a developer machine for those archs. That is not a routine task for me, so will require some attention that I will lack for the next week due to travel¹.

Could you please try help me locate ways to expose more details during build? It seems to me that the test target is doing some linking, and I would like to know exactly what is the commandline it is invoking.

I can easily add options to turn builds more verbose also while traveling, so that could win us some time.

  • Jonas

¹) I lead a pilot project at the European Parliament in using GnuPG: https://wiki.debian.org/DebianParl/GreensEFA

  • Jonas Smedegaard - idealist & Internet-arkitekt
  • Tlf.: +45 40843136 Website: http://dr.jones.dk/

[x] quote me freely [ ] ask before reusing [ ] keep private

— Reply to this email directly or view it on GitHub https://github.com/jeffreykegler/Marpa--R2/issues/150#issuecomment-47967898.

jonassmedegaard commented 10 years ago

Quoting Jeffrey Kegler (2014-07-03 20:48:56)

Have a pleasant and successful trip. I'll need to do some refactoring of Marpa, so I probably have a week's worth of homework to do.

Thanks.

Looking forward to serve as your guniea pig when we (soon) both are ready to for next chapter in this adventure :-)

jeffreykegler commented 10 years ago

The refactoring is going well. I've separated the code into two repositories and created a new developer's release totest that it does not cause regression on the CPAN side. All tests pass.

I've added an option to build both with external static and shared Libmarpa libraries, and have tested these.

I've also found and fixed a bug that, when Marpa::R2 linking failed, caused the error message not to be passed on. This probably explains the lack of verbosity in the i386 problem. This fix will find its way into a new CPAN release shortly.

Right now I am creating documentation of the various install and build options. It will be very terse, but I hope complete and accurate.
Doing even terse documentation right is a slow process, and I expect will take up the new few days.

-- jeffrey

On 07/04/2014 01:03 AM, Jonas Smedegaard wrote:

Quoting Jeffrey Kegler (2014-07-03 20:48:56)

Have a pleasant and successful trip. I'll need to do some refactoring of Marpa, so I probably have a week's worth of homework to do.

Thanks.

Looking forward to serve as your guniea pig when we (soon) both are ready to for next chapter in this adventure :-)

  • Jonas

  • Jonas Smedegaard - idealist & Internet-arkitekt
  • Tlf.: +45 40843136 Website: http://dr.jones.dk/

[x] quote me freely [ ] ask before reusing [ ] keep private

— Reply to this email directly or view it on GitHub https://github.com/jeffreykegler/Marpa--R2/issues/150#issuecomment-48018010.

jonassmedegaard commented 10 years ago

Quoting Jeffrey Kegler (2014-07-06 05:32:18)

The refactoring is going well. I've separated the code into two repositories and created a new developer's release totest that it does not cause regression on the CPAN side. All tests pass.

I've added an option to build both with external static and shared Libmarpa libraries, and have tested these.

I've also found and fixed a bug that, when Marpa::R2 linking failed, caused the error message not to be passed on. This probably explains the lack of verbosity in the i386 problem. This fix will find its way into a new CPAN release shortly.

Cool!

Right now I am creating documentation of the various install and build options. It will be very terse, but I hope complete and accurate. Doing even terse documentation right is a slow process, and I expect will take up the new few days.

Can you already now reveal if you'll stay with an unstable API or if not which will be the API you'll introduce? Or maybe easier, can you share with me the actual separated libmarpa code you've prepared so far?

I ask because it takes some days (potentially weeks or even months) for approval of new package names into Debian, and a shared library should include major version of API in its name - e.g. libmarpa0 for an unstable API or libmarpa1 for API "1.4.2".

So if you seed me your unfinished work now, I can release that to the experimental branch of Debian, simply to start the process of having the package names approved. Then when your code is ready for actual consumption I can re-release that into the Debian unstable branch (which will then follow the standard proess of maturing to testing (hopefully this fall when before the "freeze") and then into Debian stable.

jeffreykegler commented 10 years ago

Thanks! For now, let's think only in terms of an experimental/unstable API. I'd like the name to include the entire version number -- major, minor and micro -- to reflect the complete lack of any promise of backward/forward compatibility.

Quoting the Debian docs: "project/experimental/: This directory contains packages and tools which are still being developed, and are still in the alpha testing stage. Users shouldn't be using packages from here, because they can be dangerous and harmful even for the most experienced people." That sounds right for the newly separated Libmarpa.

The Libmarpa code is in a public github repository: https://github.com/jeffreykegler/libmarpa -- the documentation at this point is only partially converted, so beware. I'm in the process of cleaning out the inaccuracies and inconsistencies -- that will happen over the next few days.

-- jeffrey

On 07/06/2014 02:00 AM, Jonas Smedegaard wrote:

Quoting Jeffrey Kegler (2014-07-06 05:32:18)

The refactoring is going well. I've separated the code into two repositories and created a new developer's release totest that it does not cause regression on the CPAN side. All tests pass.

I've added an option to build both with external static and shared Libmarpa libraries, and have tested these.

I've also found and fixed a bug that, when Marpa::R2 linking failed, caused the error message not to be passed on. This probably explains the lack of verbosity in the i386 problem. This fix will find its way into a new CPAN release shortly.

Cool!

Right now I am creating documentation of the various install and build options. It will be very terse, but I hope complete and accurate. Doing even terse documentation right is a slow process, and I expect will take up the new few days.

Can you already now reveal if you'll stay with an unstable API or if not which will be the API you'll introduce? Or maybe easier, can you share with me the actual separated libmarpa code you've prepared so far?

I ask because it takes some days (potentially weeks or even months) for approval of new package names into Debian, and a shared library should include major version of API in its name - e.g. libmarpa0 for an unstable API or libmarpa1 for API "1.4.2".

So if you seed me your unfinished work now, I can release that to the experimental branch of Debian, simply to start the process of having the package names approved. Then when your code is ready for actual consumption I can re-release that into the Debian unstable branch (which will then follow the standard proess of maturing to testing (hopefully this fall when before the "freeze") and then into Debian stable.

  • Jonas

  • Jonas Smedegaard - idealist & Internet-arkitekt
  • Tlf.: +45 40843136 Website: http://dr.jones.dk/

[x] quote me freely [ ] ask before reusing [ ] keep private

— Reply to this email directly or view it on GitHub https://github.com/jeffreykegler/Marpa--R2/issues/150#issuecomment-48106497.

jonassmedegaard commented 10 years ago

Quoting Jeffrey Kegler (2014-07-06 18:42:40)

Thanks! For now, let's think only in terms of an experimental/unstable API. I'd like the name to include the entire version number -- major, minor and micro -- to reflect the complete lack of any promise of backward/forward compatibility.

Yes, I've noticed your use of autotools "-release" hinting, which is strongly discouraged it the documentation of autotools. I cannot stop you, but will most likely patch the Debian packaging to instead use 0 as SONAME which as I understand it is common notation for unstable API.

The Libmarpa code is in a public github repository: https://github.com/jeffreykegler/libmarpa -- the documentation at this point is only partially converted, so beware. I'm in the process of cleaning out the inaccuracies and inconsistencies -- that will happen over the next few days.

Ohh - it seems you have abandoned autotools. Why? :-(

Hand-written makefiles are quite difficult to work properly (that's why autotools was invented in the first place!).

jeffreykegler commented 10 years ago

Libmarpa has not abandoned autotools. The directory layout is perhaps confusing, but they are still there. If you do a "make dist" at the top level you get a dist/ directory containing what is (I hope) a relatively standard autotools distribution. "make tar" produces its equivalent as a tar file at the top level.

Perhaps what's confusing is that I don't develop directly in the distribution directory -- the development directories produce pieces, which are gathered into an autotools distribution directory.

-- jeffrey

On 07/06/2014 10:41 AM, Jonas Smedegaard wrote:

Ohh - it seems you have abandoned autotools. Why? :-(

Hand-written makefiles are quite difficult to work properly (that's why autotools was invented in the first place!).

  • Jonas

  • Jonas Smedegaard - idealist & Internet-arkitekt
  • Tlf.: +45 40843136 Website: http://dr.jones.dk/

[x] quote me freely [ ] ask before reusing [ ] keep private

— Reply to this email directly or view it on GitHub https://github.com/jeffreykegler/Marpa--R2/issues/150#issuecomment-48117977.

jeffreykegler commented 10 years ago

I take it the semantics of the zero in "libmarpa0" is really that of a wildcard. That is, it matches the libmarpa, rather than libmarpa 0..*. So it's saying, just give me any version, and don't check at all. If my understanding is correct, and this is standard for Debian experiment releases, then that's fine.

Libmarpa has calls and macros which enable the user to examine the library's version, as well as the header's version. So applications can choose their own level of risk, which in the experimental context seems a sensible approach. An experimental application may want to just run with whatever happens to be out there.

By the way, Marpa::R2 at run time does sanity checks on the version numbers -- it does not rely on the OS and packaging environment to ensure that they match properly.

I'll go back to the GNU docs and re-examine the '-release' issue. But I think anything I am likely to make the default, you'd wind up patching anyway.

My focus at this point is not to try to meet Debian standards directly, but instead to switch to a build setup that present fewer obstacles to the Debian tools in doing their customizations. Many of the Debian ideas look like things I should take upstream, but I'm making conversion to a less "odd" build environment the priority. (I particularly like some of the hardenings.)

-- jeffrey

On 07/06/2014 10:41 AM, Jonas Smedegaard wrote:

Quoting Jeffrey Kegler (2014-07-06 18:42:40)

Thanks! For now, let's think only in terms of an experimental/unstable API. I'd like the name to include the entire version number -- major, minor and micro -- to reflect the complete lack of any promise of backward/forward compatibility.

Yes, I've noticed your use of autotools "-release" hinting, which is strongly discouraged it the documentation of autotools. I cannot stop you, but will most likely patch the Debian packaging to instead use 0 as SONAME which as I understand it is common notation for unstable API.

The Libmarpa code is in a public github repository: https://github.com/jeffreykegler/libmarpa -- the documentation at this point is only partially converted, so beware. I'm in the process of cleaning out the inaccuracies and inconsistencies -- that will happen over the next few days.

Ohh - it seems you have abandoned autotools. Why? :-(

Hand-written makefiles are quite difficult to work properly (that's why autotools was invented in the first place!).

  • Jonas

  • Jonas Smedegaard - idealist & Internet-arkitekt
  • Tlf.: +45 40843136 Website: http://dr.jones.dk/

[x] quote me freely [ ] ask before reusing [ ] keep private

— Reply to this email directly or view it on GitHub https://github.com/jeffreykegler/Marpa--R2/issues/150#issuecomment-48117977.

jeffreykegler commented 10 years ago

Quoting from the libtool manual: "In general, you should only use -release for package-internal libraries or for ones whose interfaces change very frequently." Right now Libmarpa's use is nearly 100% package-internal, and its interface is allowed to change with every release.

I think the discouraging tone of much of the language about "--release" is to warn those lured by its name into thinking it's simply a convenient way to name their releases, applicable in most or all cases.

Once I stabilize the Libmarpa API, I'll switch to another mechanism.
For now I think I'm using "--release" precisely as GNU intended.

-- jeffrey

On 07/06/2014 10:41 AM, Jonas Smedegaard wrote:

, I've noticed your use of autotools "-release" hinting, which is strongly discouraged it the documentation of autotools.

jeffreykegler commented 10 years ago

I've finished documenting the external library installation. The docs can be found by following the trail from the INSTALL doc at the top of the two repositories. The trail is long, but that's deliberate -- I put the stuff the ordinary user needs at the top and the more advanced stuff off in separate docs.

I've tested this, and I have uploaded a developer's release containing all this to CPAN, to check for regressions. All looks well.

No doubt you'll find issues, but I hope this is a solid starting point.

-- jeffrey

jeffreykegler commented 10 years ago

Status: the changes I've made in the build have been out on CPANtesters, with 100% OK results, and ordinarily I'd be ready to do an indexed, stable CPAN release. But there's a problem -- it's gotten no Win32 testing.

A lot of my power users are Win32 users, but I don't have a Win32 setup. Since this release heavily refactors the build, and the build is very different on Win32, I'd really like to see some Win32 testing before I go ahead. So I'm waiting a bit.

jeffreykegler commented 10 years ago

The changes I've made are now out in a stable, indexed CPAN release: Marpa-R2 2.088000.

@jonassmedegaard -- with this, are you able to proceed?

jonassmedegaard commented 10 years ago

Quoting Jeffrey Kegler (2014-07-23 19:23:35)

The changes I've made are now out in a stable, indexed CPAN release: Marpa-R2 2.088000.

@jonassmedegaard -- with this, are you able to proceed?

I believe so, yes. I have had a brief look already, but not yet found/taken enough time to finish a packaging update.

jeffreykegler commented 10 years ago

@jonassmedegaard A question. Inside the Marpa community, we're discussing choices of build tools, and I am defending autoconf as standard. I understand you and the Debian installers prefer autotools-based configuration over other alternatives, if perhaps only because it imposes a "common language". Is this right? Or would we be better off with other build/configuration tools?

jeffreykegler commented 10 years ago

For tracking, the title bugs are Debian bugs

750356, "libmarpa-r2-perl: FTBFS on i386: Could not load XS"; and

750358; "libmarpa-r2-perl: FTBFS on s390x: multiple tests segfault".

Both of these appear to be still open.

jeffreykegler commented 9 years ago

@jonassmedegaard Can I close this? The issue has sort of meandered on, and I'm not sure what's still open and still not. Or is there something I should give attention to?

The closing is of course without prejudice to re-raising any issues discussed, or any new ones.

jonassmedegaard commented 9 years ago

Quoting Jeffrey Kegler (2014-11-22 05:31:48)

[1]@jonassmedegaard Can I close this? The issue has sort of meandered on, and I'm not sure what's still open and still not. Or is there something I should give attention to?

Uhm, well...

Not necessarily something you need to attend to, but for Debian I did not find enough time to update to your newer refactored version before the freeze for next stable distribution release.

Status right now is that the isolated libmarpa-r2 library (version 6.1.0 so not your newest release) https://packages.qa.debian.org/libmarpa builds succesfully for all our architectures https://buildd.debian.org/status/package.php?p=libmarpa&suite=experimental, whereas the Perl library does not yet make use of that but is still at 2.086000 https://packages.qa.debian.org/libm/libmarpa-r2-perl.html which fails for i386 and s390x https://buildd.debian.org/status/package.php?p=libmarpa-r2-perl&suite=unstable.

You have given me some hints as to how to try locate the cause of the build failures, and I appreciate that. If you, in your refactoring, ran into some details that might be additionally useful for patching that 2.086000 release (as opposed to upgrading), then that would be tremendously helpful.

...but really, you are obviously not required to care about obsolete releases - that is a burden we in Debian put on ourselves by how we choose to stabilize our distribution. So feel free to close this bugreport - if you then anyway find time and interest in following up on our Debian-specific issue you are more than welcome to do so by reading https://bugs.debian.org/769265 and posting to 769265@bugs.debian.org.

The closing is of course without prejudice to re-raising any issues discussed, or any new ones.

Right :-)

mbanck commented 9 years ago

I had a look at this today. So the general failure is in t/00-load.t, line 25, in the "require Marpa::R2;" eval. That one in turns fails around line 82 of blib/lib/Marpa/R2.pm. At this point this is rather mysterious to me, on x86_64, the XSLoader::load() seems to work fine and continue, whereas on i386, the second eval ("require DynaLoader;" ,"Dynaloader::bootstrap Marpa::R2") is called, and subsequently the Carp::croak to bail out.

However, on both archs bootstrap in Dynaloader (which is also called from XSLoader) calls "Marpa::R2 2.086000" as argument to $xs at line 210 of Dynaloader.pm, but on i386 that call fails with "Could not load XS version of Marpa::R2: Can't use an undefined value as a subroutine reference at /usr/lib/i386-linux-gnu/perl/5.20/DynaLoader.pm line 210."

jeffreykegler commented 9 years ago

I recommend upgrading to the latest Marpa::R2, which includes among other things, some Debian-targeted fixes. If that is not possible, and you really need to patch instead, you can start with the one which follows. It fixes some over-engineering on my part, together with an outright screw-up which causes error messages not to be reported. I hope it will be at least a good start toward a solution.

index 1c1404d..74fe683 100644
--- a/cpan/lib/Marpa/R2.pm
+++ b/cpan/lib/Marpa/R2.pm
@@ -20,7 +20,7 @@ use strict;
 use warnings;

 use vars qw($VERSION $STRING_VERSION @ISA $DEBUG);
-$VERSION        = '2.086000';
+$VERSION        = '2.101_000';
 $STRING_VERSION = $VERSION;
 ## no critic (BuiltinFunctions::ProhibitStringyEval)
 $VERSION = eval $VERSION;
@@ -29,6 +29,7 @@ $DEBUG = 0;

 use Carp;
 use English qw( -no_match_vars );
+use XSLoader;

 use Marpa::R2::Version;

@@ -76,18 +77,7 @@ LOAD_EXPLICIT_LIBRARY: {
     $Marpa::R2::LIBMARPA_FILE = $file;
 }

-eval {
-    require XSLoader;
-    XSLoader::load( 'Marpa::R2', $Marpa::R2::STRING_VERSION );
-    1;
-} or eval {
-    require DynaLoader;
-## no critic(ClassHierarchies::ProhibitExplicitISA)
-    push @ISA, 'DynaLoader';
-    Dynaloader::bootstrap Marpa::R2 $Marpa::R2::STRING_VERSION;
-    1;
-} or Carp::croak("Could not load XS version of Marpa::R2: $EVAL_ERROR");
-
+XSLoader::load( 'Marpa::R2', $Marpa::R2::STRING_VERSION );

 if ( not $ENV{'MARPA_AUTHOR_TEST'} ) {
     $Marpa::R2::DEBUG = 0;
mbanck commented 9 years ago

Hrm, so it looks like if I comment out this code in lib/Marpa/R2.c, line 13941, it runs fine:

if (PERL_REVISION == 5 && PERL_VERSION >= 9)

if (PL_unitcheckav) call_list(PL_scopestack_ix, PL_unitcheckav);

endif

Unfortunately, I currently have no idea how to prevent this C-code from being generated.

jeffreykegler commented 9 years ago

I don't know if it's relevant, but in the Marpa::R2 build, lib/Marpa/R2.c is not a source file. It is created from xs/R2.xs.

Also, the patch above may need a touch up -- it increments the Marpa::R2 version #, which is you are not upgrading, you will not want to do.

On Sat, Nov 22, 2014 at 7:17 AM, Michael Banck notifications@github.com wrote:

Hrm, so it looks like if I comment out this code in lib/Marpa/R2.c, line 13941, it runs fine:

if (PERL_REVISION == 5 && PERL_VERSION >= 9)

if (PL_unitcheckav) call_list(PL_scopestack_ix, PL_unitcheckav);

endif

— Reply to this email directly or view it on GitHub https://github.com/jeffreykegler/Marpa--R2/issues/150#issuecomment-64083338 .

jeffreykegler commented 9 years ago

@mbanck I also replied to Debian's bug list 769265 and am getting responses from gregor herrmann on that. Do we want to move this discussion over there?

mbanck commented 9 years ago

I'll try to summarize in the Debian bug.

So yeah, R2.c is auto-generated, and it seems the faulty(?) part is part of the auto-generated code. Another thing I want to note is that Debian compiles with -fstack-protector-strong, not sure that is relevant in this context. I checked the generated C-code for R2.c on i386 and x86_64, and those are identical.

mbanck commented 9 years ago

So, after reading http://blogs.perl.org/users/david_mertens/2013/02/cudasimple-back-where-it-should-be-but-why-did-it-break.html this patch seems to be a work-around:

--- ./xs/R2.xs.orig     2014-11-22 16:59:50.708349291 +0100
+++ ./xs/R2.xs  2014-11-22 17:00:02.556179089 +0100
@@ -6428,6 +6428,8 @@
 INCLUDE: general_pattern.xsh

 BOOT:
+#undef PERL_VERSION
+#define PERL_VERSION 0
     marpa_debug_handler_set(marpa_r2_warn);

     /* vim: set expandtab shiftwidth=2: */

I don't claim it's a valid fix, though, setting PERL_VERSION to 0 looks pretty ugly to me.

jeffreykegler commented 9 years ago

Perhaps you could add a comment for a later maintainer who otherwise is sure to be baffled by such a strange change.

On Sat, Nov 22, 2014 at 8:08 AM, Michael Banck notifications@github.com wrote:

So, after reading http://blogs.perl.org/users/david_mertens/2013/02/cudasimple-back-where-it-should-be-but-why-did-it-break.html this patch seems to be a work-around:

--- ./xs/R2.xs.orig 2014-11-22 16:59:50.708349291 +0100 +++ ./xs/R2.xs 2014-11-22 17:00:02.556179089 +0100 @@ -6428,6 +6428,8 @@ INCLUDE: general_pattern.xsh

BOOT: +#undef PERL_VERSION +#define PERL_VERSION 0 marpa_debug_handler_set(marpa_r2_warn);

 /* vim: set expandtab shiftwidth=2: */

I don't claim it's a valid fix, though, setting PERL_VERSION to 0 looks pretty ugly to me.

— Reply to this email directly or view it on GitHub https://github.com/jeffreykegler/Marpa--R2/issues/150#issuecomment-64085031 .

mbanck commented 9 years ago

Right, I suggested this patch to Debian now:

Index: libmarpa-r2-perl-2.086000~dfsg/xs/R2.xs
===================================================================
--- libmarpa-r2-perl-2.086000~dfsg.orig/xs/R2.xs
+++ libmarpa-r2-perl-2.086000~dfsg/xs/R2.xs
@@ -6428,6 +6428,19 @@ PPCODE:
 INCLUDE: general_pattern.xsh

 BOOT:
+  /* XS generates this code for boot:
+   *
+   *|#if (PERL_REVISION == 5 && PERL_VERSION >= 9)
+   *|        if (PL_unitcheckav)
+   *|        call_list(PL_scopestack_ix, PL_unitcheckav);
+   *|#endif
+   *
+   * On i386 Debian, this results to a segfault, so Marpa::R2
+   * cannot be loaded.  Setting PERL_VERSION to 0 circumvents
+   * running call_list().
+   */
+#undef PERL_VERSION
+#define PERL_VERSION 0
     marpa_debug_handler_set(marpa_r2_warn);

     /* vim: set expandtab shiftwidth=2: */
jeffreykegler commented 9 years ago

At this point, by my reading of the Debian bug logs, i386 is OK. s390 is not -- that's https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=750358

jeffreykegler commented 9 years ago

The i386 problem seems to be solved. For s390, I've subscribed to the bug log, which looks as if it is a lot less active than the i386 one. Further discussion can take place there.

A lot got discussed above, and I don't claim all of it is resolved. Please feel free to raise this issue again. Unless some of this thread is vital to your point, please do so in a new issue. This one became pretty messy.