search

thewml / website-meta-language

An old offline HTML preprocessor (which can be used for static site generation), written in Perl and C that is still maintained for legacy reasons, but probably not recommended for new sites.
https://www.shlomifish.org/open-source/projects/website-meta-language/
GNU General Public License v2.0
14 stars 8 forks source link

[TODO] Update the Debian Package To The Latest Upstream Release #37

Closed shlomif closed 4 years ago

shlomif commented 4 years ago

Here is my progress so far. With this docker script (note the branch and the commit), I am getting this "before" output:

And with this patch in the repo:

commit d9bbe435205e34f21082bc6a118ec61e48a16658
Author: Shlomi Fish <shlomif@shlomifish.org>
Date:   Thu Jun 11 18:26:09 2020 +0300

    fix

diff --git a/debian/rules b/debian/rules
index 6eae824..75a5f32 100755
--- a/debian/rules
+++ b/debian/rules
@@ -70,7 +70,7 @@ ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS) $(DEB_BUILD_PROFILES)))
    ln -vis /usr/bin/eperl $(o)/wml_test/tests/installation/lib/wml/exec/wml_p3_eperl
    ln -vis /usr/bin/m4    $(o)/wml_test/tests/installation/lib/wml/exec/wml_p4_gm4
    ln -vis /usr/bin/slice $(o)/wml_test/tests/installation/lib/wml/exec/wml_p9_slice
-   sed -e 's:/usr/\(lib\|share\)/wml:$(o)/wml_test/tests/installation/\1/wml:g; s:\(WML_LOC_.*\)=/usr:\1=$(o)/wml_test/tests/installation:g;' \
+   sed -e 's:^\(my $$EXEC_DIR *=\).*:\1 "$(o)/wml_test/tests/installation/lib/wml/exec";:g; s:/usr/\(lib\|share\)/wml:$(o)/wml_test/tests/installation/\1/wml:g; s:\(WML_LOC_.*\)=/usr:\1=$(o)/wml_test/tests/installation:g;' \
        -i $(o)/wml_test/tests/installation/bin/wm? \
           $(o)/wml_test/tests/installation/lib/wml/exec/wml_p* \
           $(o)/wml_test/tests/installation/lib/perl5/TheWML/*.pm \

The output is better:

I tried restoring the SliceTermParser.pm too but dpkg-source did not approve.

wml-debian-after-1.txt wml-debian-before-1.txt

@xtaran : any idea? can you tell me how to restore the missing files to the git repo?

shlomif commented 4 years ago

@xtaran : hi! I made a lot of progress in this branch: https://salsa.debian.org/shlomif-guest/wml/-/commits/shlomify4 and this script: https://github.com/shlomif/fortune-mod/blob/docker-build-debian-website-meta-lang-package/scripts/debian-packaging/docker-test.pl . The changes and git history are quite ugly, but it is just a proof of concept.

shlomif commented 4 years ago

Seems to be mostly working now.

xtaran commented 4 years ago

Thanks! Will have a look into it, but probably no more today.

xtaran commented 4 years ago

I tried restoring the SliceTermParser.pm too

Did I remove that from the Debian package by accident?

but dpkg-source did not approve.

Yeah, dpkg-source doesn't like if you change upstream code directly. Just ignore building the source package. I'll take care of that. Just use dpkg-buildpackage -b and if that works again, I'm happy!

Thanks for the help!

shlomif commented 4 years ago

Hi @xtaran !

I tried restoring the SliceTermParser.pm too

Did I remove that from the Debian package by accident?

It was present in wml_backend/p9_slice and was not installed into the test installation.

but dpkg-source did not approve.

Yeah, dpkg-source doesn't like if you change upstream code directly. Just ignore building the source package. I'll take care of that. Just use dpkg-buildpackage -b and if that works again, I'm happy!

I managed to fix that, and now the build, test and packaging is working (modulo some lintian/etc. warnings).

Thanks for the help!

shlomif commented 4 years ago

@xtaran ; hi, ping! Any idea?

xtaran commented 4 years ago

Sorry, quite busy at work these days. Haven't found time for it yet.

shlomif commented 4 years ago

Sorry, quite busy at work these days. Haven't found time for it yet.

OK, thanks for the update.

xtaran commented 4 years ago

Working on it now.

I made a lot of progress in this branch: https://salsa.debian.org/shlomif-guest/wml/-/commits/shlomify4

Thanks again for your work on this. Your help is really appreciated.

The changes and git history are quite ugly,

Indeed. :-)

but it is just a proof of concept.

Will try either get the essence out of your changes or squash all your changes in one commit. Will see. (Probably the former.)

xtaran commented 4 years ago

I tried restoring the SliceTermParser.pm too

Did I remove that from the Debian package by accident?

Not by accident but on purpose. And that brought me onto the right track…

Since slice so far is a separate (source and binary) package in Debian, the file SliceTermParser.pm is not in the wml source package but in the slice source package (and named slice_term.pl there). The fact, that it is tried to being used at all despite the whole p9_slice is not in the wml source package in Debian (on purpose), led me to the following question:

Am I right that since approximately WML 2.20.x, slice is no more called by wml as program slice but just by calling a function inside the according Perl modules (via wml_include/TheWML/Backends/Slice/Main.pm)?

If so, I guess, it is time to build the binary package slice from the wml source package. At least the slice.pl file is still being generated so that slice still can be used standalone. (I used slice for quite a while in my .Xsession to generate config files depending on the hardware the X session is running on, but replaced it with a simpler script more tailored towards that use case at some point.)

BTW: Does this also apply to any other backends shipped in Debian separately? Currently this is the case with these backends:

xtaran commented 4 years ago

I guess, it is time to build the binary package slice from the wml source package.

Yep, this is it! Thanks again for the right hint! Have a passing test suite now again!

Working now on splitting off the splice binary package from the wml package.

xtaran commented 4 years ago

Finally the master branch of https://salsa.debian.org/debian/wml builds again. Still need to complete the slice package separation and to import the most recent upstream release.

shlomif commented 4 years ago

Working on it now.

Thanks! Sorry for the delayed reply.

I made a lot of progress in this branch: https://salsa.debian.org/shlomif-guest/wml/-/commits/shlomify4

Thanks again for your work on this. Your help is really appreciated.

You're welcome.

The changes and git history are quite ugly,

Indeed. :-)

but it is just a proof of concept.

Will try either get the essence out of your changes or squash all your changes in one commit. Will see. (Probably the former.)

Thanks!

I tried restoring the SliceTermParser.pm too

Did I remove that from the Debian package by accident?

Not by accident but on purpose. And that brought me onto the right track…

Since slice so far is a separate (source and binary) package in Debian, the file SliceTermParser.pm is not in the wml source package but in the slice source package (and named slice_term.pl there). The fact, that it is tried to being used at all despite the whole p9_slice is not in the wml source package in Debian (on purpose), led me to the following question:

Am I right that since approximately WML 2.20.x, slice is no more called by wml as program slice but just by calling a function inside the according Perl modules (via wml_include/TheWML/Backends/Slice/Main.pm)?

Yes, I am "require"ing it and calling it here:

sub pass9
{
    my ( $_pass_mgr, $opt, $from, $to, $tmp ) = @_;

    #   First check whether a shebang line is found and no
    #   output files were assigned on command line.
    #   This is needed to unprotect output files.
    if ( !@{ $_pass_mgr->_opt_o } )
    {
        local @ARGV = @{ $_pass_mgr->_read_slices($from) };
        if (@ARGV)
        {
            $_pass_mgr->out_istmp(0);
            $_pass_mgr->_process_argv_cb->($_pass_mgr);
            $opt = $_pass_mgr->pass(9)->opt_pass;
        }
    }

    #   slice contains "package" commands and
    #   other stuff, so we cannot source it.
    return $_pass_mgr->_generic_do(
        {
            pass_idx => 9,
            EXE      => 'wml_p9_slice',
            opt      => $opt,
            from     => $from,
            to       => $to,
            cb       => sub {
                require TheWML::Backends::Slice::Main;

                return TheWML::Backends::Slice::Main->new( argv => [@_] )->main;
            },
        }
    );
}

This is more idiomatic Modern Perl and may be somewhat faster than the previous scheme of slurping the code from the file and evalling it.

If so, I guess, it is time to build the binary package slice from the wml source package. At least the slice.pl file is still being generated so that slice still can be used standalone. (I used slice for quite a while in my .Xsession to generate config files depending on the hardware the X session is running on, but replaced it with a simpler script more tailored towards that use case at some point.)

Yes.

BTW: Does this also apply to any other backends shipped in Debian separately? Currently this is the case with these backends:

* `wml_backend/p2_mp4h`

I reworked mp4h's C code somewhat but I call it using a separate process from the wml frontend.

* `wml_backend/p3_eperl`

This is also called separately (as it is in C)

* `wml_backend/p4_gm4`

Here I search for "gm4" with a fallback on "m4" and calling them:

Note: all of these are written in C so not easy to embed in perl5 code. 

* `wml_backend/p9_slice`

This was extracted into perl modules and embedded as it is mostly pure-Perl code.

xtaran commented 4 years ago

Working on it now.

Thanks! Sorry for the delayed reply.

Hey, I was the one who was lagging with replies. :-)

Am I right that since approximately WML 2.20.x, slice is no more called by wml as program slice but just by calling a function inside the according Perl modules (via wml_include/TheWML/Backends/Slice/Main.pm)?

Yes, I am "require"ing it and calling it here

Thanks for the confirmation.

This is more idiomatic Modern Perl and may be somewhat faster than the previous scheme of slurping the code from the file and evalling it.

Sure.

BTW: Does this also apply to any other backends shipped in Debian separately? Currently this is the case with these backends:

  • wml_backend/p2_mp4h

I reworked mp4h's C code somewhat but I call it using a separate process from the wml frontend.

I see, so this is a potential candidate to also build from the wml source package, but not necessary immediately as it probably still builds fine with the external mp4h. It though would be nice to get your improvements.

  • wml_backend/p3_eperl

This is also called separately (as it is in C)

Similar here, but probably even less urgent.

  • wml_backend/p4_gm4

Here I search for "gm4" with a fallback on "m4" and calling them:

Ah, right, forgot that this is actually just m4 and you surely don't want to maintain a copy of it as it's widespread elsewhere (and IIRC more or less actively maintained), too.

Interesting writeup. I though dislike the "solution" with the gmake for Debian. I'd rather try to port my Makefiles to make them compatible with FreeBSD's make, too. (On the other hand, there's also NetBSD's make in Debian as bmake, formerly pmake.)

Note: all of these are written in C so not easy to embed in perl5 code.

Right.

xtaran commented 4 years ago

You can close this issue now. I just uploaded wml 2.28.0~ds1-1 to Debian Unstable. :-)

(The ~ds1 stands for "Debian Source" because I still remove some embedded code copies from the tar ball.)

shlomif commented 4 years ago

You can close this issue now. I just uploaded wml 2.28.0~ds1-1 to Debian Unstable. :-)

(The ~ds1 stands for "Debian Source" because I still remove some embedded code copies from the tar ball.)

Thanks! I'll wait for this page to reflect this change: https://packages.debian.org/source/sid/wml . "It ain't over till the fat lady sings."

shlomif commented 4 years ago

Hi @xtaran !

Working on it now.

Thanks! Sorry for the delayed reply.

Hey, I was the one who was lagging with replies. :-)

Heh, maybe. We can all strive for improvement.

Am I right that since approximately WML 2.20.x, slice is no more called by wml as program slice but just by calling a function inside the according Perl modules (via wml_include/TheWML/Backends/Slice/Main.pm)?

Yes, I am "require"ing it and calling it here

Thanks for the confirmation.

YW.

This is more idiomatic Modern Perl and may be somewhat faster than the previous scheme of slurping the code from the file and evalling it.

Sure.

BTW: Does this also apply to any other backends shipped in Debian separately? Currently this is the case with these backends:

  • wml_backend/p2_mp4h

I reworked mp4h's C code somewhat but I call it using a separate process from the wml frontend.

I see, so this is a potential candidate to also build from the wml source package, but not necessary immediately as it probably still builds fine with the external mp4h. It though would be nice to get your improvements.

It should work fine with the older mp4h, yes. From what I seem to recall, most of my changes were either switching the build sys to cmake, removing no-longer-relevant portability checks and #ifdefs , and some small cleanups.

  • wml_backend/p3_eperl

This is also called separately (as it is in C)

Similar here, but probably even less urgent.

  • wml_backend/p4_gm4

Here I search for "gm4" with a fallback on "m4" and calling them:

Ah, right, forgot that this is actually just m4 and you surely don't want to maintain a copy of it as it's widespread elsewhere (and IIRC more or less actively maintained), too.

Yes. Note that I had issues using the native FreeBSD m4 instead of GNU m4 on fbsd with wml, so I first check for gm4.

Interesting writeup. I though dislike the "solution" with the gmake for Debian. I'd rather try to port my Makefiles to make them compatible with FreeBSD's make, too. (On the other hand, there's also NetBSD's make in Debian as bmake, formerly pmake.)

Well, I'm using quite a few GNU make-specific features in my websites' build systems and elsewhere for this to be easy (even if desirable). The shlomifish.org main makefile is a 1,145 lines monstrocity and has several generated includes. Limiting myself to POSIX-only make features may make it even harder to main.

I've been contemplating trying to transition to using Ruby rake or similar ( https://www.shlomifish.org/open-source/resources/software-tools/#build ) but I'm not sure they'll be an improvement (I don't have enough familiarity or experience with them).

Note: all of these are written in C so not easy to embed in perl5 code.

Right.

shlomif commented 4 years ago

Yay! The Eagle... err "Wamle" has landed in Debian sid:

@xtaran : (and all) thanks for everything. Closing as FIXED.