Devel::PPPort internals for hackers
So you probably want to hack Devel::PPPort
?
Well, here's some information to get you started with what's lying around in this distribution.
First, make sure that what you want to backport is documented. If it's worth
backporting, it surely is worth documenting. Submit a documentation patch to
https://github.com/Perl/perl5/issues if necessary. Also, Devel::PPPort
cannot automatically generate proper information about the item without at
least knowing its API prototype. It can get this from embed.fnc
if the item
is a function, but if it is a macro, there needs to be at least a =for apidoc
line for Devel::PPPort
to be able to figure things out on its own.
Next, figure out where to place your implementation. Look at all the files in
parts/inc/
for one that fits what you're planning. If there isn't one,
just start a new one and remember to include it from within PPPort_pm.PL
.
If you do create a new file, it's usually the best approach to just copy an
existing file and use it as a template.
Each file holds all relevant data for implementing a certain part of the API:
=provides
section.=dontwarn
section is used to suppress warnings about particular
API elements. Don't use this unless you get such a warning, and be sure to
think about using other other alternatives before resorting to adding something
in this section.ppport.h
in the =implementation
section. See "Implementation Section Details".=xshead
, =xsinit
, =xsmisc
, =xsboot
and =xsubs
section. Have a look at the template at the bottom
of RealPPPort_xs.PL
to see where the code ends up.The tests in the =tests
section. Remember not to use any fancy
modules or syntax elements, as the test code needs to be able to run
with Perl 5.003. (This is because Devel::PPPort itself will run all test files
in the process of generating the information about when a feature came into
existence.) This means, for example
my
isn't supported in for
-loops
for my $x (1, 2, 3) { } # won't work with 5.003
Instead declare $x
just before the statement
The postfix for
statement modifier isn't supported
foo for 1..2
won't compile. Instead enclose foo
in a loop.
You can't use plain qr//
Instead, wrap it in a string eval eval "qr//"
, and be sure it's skipped at
execution time on perls earlier than 5.005
As of version 3.56 of Devel::PPPort, the old Test style tests have been
replaced with the more modern Test::More style, with some limitations. This
means, for example, that is
is finally available, as well as
done_testing
. You can pass the number of tests to skip
, instead of
having to have your own for
loop.
There is no like
nor unlike
(as those require qr
which didn't exist in
the earliest perls that Devel::PPPort runs on).
skip
doesn't do a last SKIP
. (Perhaps it could, but that would mean
converting all the skips in the existing tests.)
The existing tests have been changed only as much as necessary so as to get things to work. But feel free to use the full functionality for any new tests you write.
Here's a list of the supported functions:
cmp_ok
curr_test
diag
display
done_testing
eq_array
eq_hash
fail
is
isnt
next_test
note
ok
pass
plan
skip
skip_all
within
These are copied from t/test.pl
in the perl distribution. Not all of them
have been tested back as far as Devel::PPPort supports. Bug reports welcome.
It's fine to backport an element only as far as convenient and necessary. But
remember that your test file will end up being called on all perl versions
available, likely including ones earlier than your backport. That may mean
that elements in the =xs
sections will have to be #idef
'd out so that the
object will not get missing symbols when loaded.
It also means you have to check for and skip tests that aren't relevant to this version. The recommended way to do this is like:
if (ivers($]) < ivers(5.6)) { # No quotes needed
skip "reason", $count;
}
elsif (if (ivers($]) > ivers("5.5.4") { # Quotes needed for double-
# dotted versions prior to 5.6.0
skip "other reason", $count;
}
else {
do_the_test
}
ivers()
is a function automatically made available to all .t
files. It
converts any reasonble expression of a version number into an integer, which
can reliably be compared using numeric comparison operators, with the output of
a second ivers()
call on a different version number, like in the result above.
It's worth emphasizing that, due to bugs in early perl parsing, if you use a version number containing two dots on a version befor 5.6.0, it has to be quoted.
In all sections, lines that begin with ##
are completely ignored.
You can implement API elements via C functions or macros, or simple variables.
It is preferable to use a macro if feasible. Otherwise, the user must
explicitly request that it get loaded, by defining a NEED__function_
(or
variable) as described in ppport.h
. If a function, foo is required,
place its body in this =implementation
section, like so:
#if { NEED foo }
char *
foo(pTHX_ const U8 *arg1, const U32 arg2, ...)
{
...
}
#endif
Similarly for a variable.
It's obviously best to use a macro if at all feasible. Sometimes what once
was implemented with a macro now requires a function; perhaps an edge case was
overlooked. Doing so will cause the new ppport.h
to not be drop-in
compatible with the older version, and can hence cause breakage. This
incompatiblity (while easily solved) really needs to be stressed in
documentation.
__UNDEFINED__
If you add the line __UNDEFINED__
to the =provides
section, you can use
lines like this in the =implementation
section:
__UNDEFINED__ macro some definition
to both define macro
and indicate that it is provided by ppport.h
. This
replaces these =implementation
section lines:
#ifndef macro
# define macro some definition
#endif
besides automagically making it be considered to be provided. macro
can
have optional arguments and the definition can even span multiple lines, like
in
__UNDEFINED__ SvMAGIC_set(sv, val) \
STMT_START { assert(SvTYPE(sv) >= SVt_PVMG); \
(((XPVMG*) SvANY(sv))->xmg_magic = (val)); } \
STMT_END
This usually makes the code more compact and readable.
But you should only use this on things that you plan to publicly provide. If something, such as a mnemonic for a constant needs to be defined but isn't really needed for the public at large to know about, you should use
__UNDEF_NOT_PROVIDED__ macro some definition
instead. To avoid name space conflicts, follow what's in "Helper macros", below.
Helper macros
If you need to define a helper macro which is not part of Devel::PPPort
API
and its usage is only for the definition of other Devel::PPPort
macros, then
use the D_PPP_
prefix for this macro name (e.g. D_PPP_SVPV_NOLEN_LP_ARG
).
This suppresses any warnings when a macro is defined which is not part of the
Perl public API.
Version numbers
Version checking used to be tricky to get correct (besides being buggy in some
perl versions).
ivers()
is used in the =tests
section to overcome this. and constructs
like the following in the C language sections.
#if { VERSION < 5.9.3 }
You SHOULD be using this construct or the alternatives listed below for ALL version checks, and not come up with something on your own.
In this form, the version number can be either of the new form 5.x.y
or the
older form 5.00x_yy
. Both are translated into the correct preprocessor
statements. It is also possible to combine this with other statements:
#if { VERSION >= 5.004 } && !defined(sv_vcatpvf)
/* a */
#elif { VERSION < 5.004_63 } && { VERSION != 5.004_05 }
/* b */
#endif
This not only works in the =implementation
section, but also in
the =xsubs
, =xsinit
, =xsmisc
, =xshead
and =xsboot
sections.
Alternatively, you can use the forms now available in regular Perl:
#if PERL_VERSION_EQ(5,9,3)
#if PERL_VERSION_NE(5,9,3)
#if PERL_VERSION_LT(5,9,3)
#if PERL_VERSION_GT(5,9,3)
#if PERL_VERSION_LE(5,9,3)
#if PERL_VERSION_GE(5,9,3)
These forms have the advantage over the '{ VERSION ... }' form in that you may use the special value '*' for the final number to mean ALL possible values for it. Thus,
#if PERL_VERSION_EQ(5,31,'*')
means all perls in the 5.31 series. And
#if PERL_VERSION_NE(5,24,'*')
means all perls EXCEPT 5.24 ones. And
#if PERL_VERSION_LE(5,9,'*')
is effectively
#if PERL_VERSION_LT(5,10,0)
Hints
If you add a comment like so:
/* Hint: PL_expect, PL_copline, PL_rsfp
paragraphs of stuff you want to have shown when ppport.h outputs
something about any one of PL_expect, PL_copline, or PL_rsfp
*/
Earlier versions of ppport.h
required an asterisk at the beginning of every
continuation line, or else the content would be silently dropped.
Warnings
A more serious caution can be displayed by instead saying
/* Warning: PL_expect, PL_copline, PL_rsfp
paragraphs of stuff you want to have shown when ppport.h outputs
something about any one of PL_expect, PL_copline, or PL_rsfp
*/
Earlier versions of ppport.h
required an asterisk at the beginning of every
continuation line, or else the content would be silently dropped.
Replace
When ppport.h
is run on a file(s), you can cause it to automatically flag
occurrences of the constructs you specify, encouraging the author to replace
them with different (presumably better) ones. These also are used in any
suggested edits and generated patches.
There are three ways to do this
in-line comment
You can add a trailing comment like so:
#define bar foo /* Replace */
__UNDEFINED__ bar foo /* Replace */
These say that foo
should be replaced by bar
. NOT the other way around.
separate comment
For situations not amenable to the above, you can say
/* Replace foo with bar */
define a replacement region
It you have several replacements, you can group them together like so:
/* Replace: 1 */
#define foo bar
#define bat baz
/* Replace: 0 */
These replace bar
with foo
; baz
with bat
. NOT the other way
around.
Dependencies
ppport.h
automatically gathers information as to what functions are
dependent on what other things from inspecting the source, but if this is
insufficient for you, you can add lines like the following:
/* foo, bar depends on baz, bat */
Each of foo
, bar
depends on each of baz
, bat
.
After you have furnished your implementation, you need to test it.
You can use
make regen
to regenerate all of the autogenerated files. To get rid of all
generated files (except for parts/todo/*
and parts/base/*
),
use
make purge_all
That's it.
To automatically test Devel::PPPort
with lots of different Perl
versions, you can use the soak
script. Just pass it a list of
all Perl binaries you want to test.
ppport.h
and PPPort.pm
Devel::PPPort
keeps two directories of generated files, in parts/base
and
parts/todo
. The files in each are named after Perl version numbers. When a
function or macro came into existence is indicated by placing its name in the
corresponding file in parts/base
. The files in parts/todo
are the same,
except they indicate the earliest release that ppport.h
supports the
element. The delta is effectively what ppport.h
buys you.
The generation process described in this section creates these files. It does so by examining as many perl versions as are available to it. It tries to make sure each element actually compiles, and it runs the test scripts you have furnished on every version.
Ideally, this should be done before every release that includes new backporting and/or when blead has added new public API. At a minimum, it should be done as the next major Perl release comes out.
The process isn't platform independent. It has currently been tested only under
Linux, and it definitely requires at least gcc
and the nm
utility.
The process used to be problematic, with random failures. But it has now been
fixed to be reliable.
Before starting the regeneration, you need to have gathered certain data. (Options listed below apply to the tools that eventually will use the data, and which are described further below).
You will first need a whole bunch of different Perls, the more, the better, but
only one per version tag (which one is random) will actually get used.
dromedary has a sufficient set. They should all have the same Configure
options with respect to what functions and macros are enabled. For example,
they should all be threaded, or all non-threaded. A mixture will screw up the
results. Similarly, they should all or none have quad math (at least as far
back as that became available). You can use devel/buildperl.pl
to build
them.
Previous maintainers of this module kept those perls in
/tmp/perl/install/default
, so most of the tools use this as a default, but
you'll likely simply use the --install=
option to specify where. This
should be a path where a make install
has been done, so has immediate
subdirectories of /bin
and /lib
. /bin
should contain the binaries.
It will use all files in this directory whose names begin with perl5
.
Actually, not all the binaries need be in this directory. You can specify
additional places to look since --install=
takes a comma separated list of
directories.
You also need a freshly built bleadperl. The --blead=_path_
option should
be used to specify it. (Some of the tools have a default of bleadperl-debug
if this option is omitted.) Again, it needs the same Configure options as the
earlier versions had. Using -DNO_MATHOMS
will change the results, and
probably should be avoided. True, these functions are allegedly on their way
out, so it could be argued that they shouldn't be encouraged in any way; but
some of these have been in limbo for many years, so should be documented.
And you will need updated API information. Copy the latest embed.fnc
file
from bleadperl to the parts
directory and run devel/mkapidoc.pl
to
collect the remaining information in parts/apidoc.fnc
. This needs to be
done after the perl has been compiled, as there are generated files that feed
it.
The final step before regenerating everything is to run
devel/mkppport_fnc.pl
to update the /parts/ppport.fnc
file.
Having done this, run devel/regenerate
which wraps the following steps
(which you could instead do by hand, but it's easy to forget things):
It first does some sanity checking
Then it asks you if it's ok to remove all existing todo files in the
parts/base
and parts/todo
directories. If you answer no, the process
aborts.
This is crtical to getting accurate results.
It builds the new baseline by running
perl devel/mktodo --base
in the root directory of the distribution.
If there are warnings in blead, it will ask you to examine them, and to ok if it's all right to go ahead. If there are issues with blead, everything following could be wrong.
It builds the new todo files by running
perl devel/mktodo
in the root directory of the distribution.
Finally, it adds the remaining information by running
perl Makefile.PL && make
perl devel/scanprov --mode=write
Devel::PPPort
supports Perl versions between 5.003 and bleadperl.
To guarantee this support, its good to have as many versions as possible to
test on. dromedary currently has many such versions.
There is a tool to build all the different
versions and configurations. You can find it in devel/buildperl.pl
.
It can currently build the following Perl releases:
5.003
5.004 - 5.004_05
5.005 - 5.005_04
5.6.x
5.7.x
5.8.x
5.9.x
5.1x.x
5.2x.x
5.3x.x
Knowing which parts of the API are not backwards compatible and
probably need Devel::PPPort
support is another problem that's
not easy to deal with manually. If you run
perl Makefile.PL --with-apicheck
a C file is generated by parts/apicheck.pl
that is compiled
and linked with Devel::PPPort
. This C file has the purpose of
using each of the public API functions/macros once.
The required information is derived from parts/embed.fnc
(just
a copy of bleadperl's embed.fnc
), parts/apidoc.fnc
(which
is generated by devel/mkapidoc.pl
and simply collects the rest
of the apidoc entries spread over the Perl source code) and
parts/ppport.fnc
(which lists the API provided purely by
Devel::PPPort, along with other elements that are tested only using
ppport.h
).
The generated C file (usually, apicheck.c
) won't compile as-is
with older perls. And even if it compiles, there's still a good chance of the
dynamic linker failing at make test
time. But that's on purpose!
We can use these failures to find changes in the API automatically.
The Perl script devel/mktodo
calls another script devel/mktodo.pl
repeatedly to run Devel::PPPort
on version after version of perl, in
decreasing version order, so we start with blead and work backwards. The
latter script generates an apicheck.c
. It starts with the code that
successfully worked in the previously tested Perl version, which should be the
version one higher than the current one. Call the current one n, and the
previous one n+1. The items that fail to compile in n, but did compile
in n+1 must have become available in n+1. We run the Linux command nm
to find those undefined symbols in n. We change apicheck.c
to ignore
(through #ifdef
's) those and recompile, repeating until apicheck.c
successfully compiles, the dynamic linker is happy, and make test
runs on
this version. Then we repeat the process for n-1, and so on. (Actually,
this process may generate false positives, so by default each failing API call
is checked again. If possible, this is done by generating an apicheck.c
for
just the one failing API.) Note that the make test is run using ppport.h
during both passes.
Running devel/mktodo
currently takes a couple hours on dromedary.
If you run it with the --nocheck
option, it won't recheck the API calls
that failed in the compilation stage and it'll take significantly less time.
No one currently associated with maintaining this module understands under what
circumstances it is safe to run with --nocheck
.
By repeating the process over and over, we build up information on when every
element first became supported. This information is stored in files in the
parts/base
directory, one file per version. The file for version n+1 is
generated by running version n of perl.
We actually want a second piece of information, which is how much ppport.h
buys you. What happens when regenerating is actually two entire runs through
all the perls. The first is accomplished by calling devel/mktodo
with the
--base
option. It automically will call devel/mktodo.pl
with each
version of perl, NOT using anything in ppport.h
. When done the results
indicate when each API element became available in stock perl, without using
ppport.h
.
And then the whole process is repeated, but this time ppport.h
is included.
The files are placed in parts/todo
. Thus, at the end, we know when each
element became available in modified perl, using ppport.h
.
However, only the public API that is implemented as functions (and must appear
in embed.fnc
) plus macros whose calling sequence is documented can be
checked this way. The final step in the process is calling devel/scanprov
.
It looks through the header files for when all the symbols provided by
Devel::PPPort
first became defined. It doesn't test the symbols or try to
compile them, as it doesn't generally know the API, but it can tell that
something exists in release n+1 but not n (by scanning the include files
in the CORE
directory of various Perl versions). (It does know if a macro
has zero arguments or non-zero arguments, so it does get extra information from
the zero argument ones.)
Residing in parts/inc/
is the "heart" of Devel::PPPort
. Each
of the files implements a part of the supported API, along with
hints, dependency information, XS code and tests.
The files are in a POD-like format that is parsed using the
functions in parts/ppptools.pl
.
The scripts PPPort_pm.PL
, RealPPPort_xs.PL
and mktests.PL
all
use the information in parts/inc/
to generate the main module
PPPort.pm
, the XS code in RealPPPort.xs
and various test files
in t/
.
You can get extra information from PPPort_pm.PL
by setting the environment
variable DPPP_CHECK_LEVEL
to 1 or 2.
All of these files could be generated on the fly while building
Devel::PPPort
, but not having the tests in t/
will confuse
TEST/harness in the core. Not having PPPort.pm
will be bad for
viewing the docs on search.cpan.org
. So unfortunately, it's
unavoidable to put some redundancy into the package.
If you've added some functionality to Devel::PPPort
, please
consider submitting a patch with your work to P5P by sending a pull request to
https://github.com/Dual-Life/Devel-PPPort/pulls.
When submitting patches, please only add the relevant changes
and don't include the differences of the generated files. You
can use the purge_all
target to delete all autogenerated
files.
When integrating this module into the Perl core, be sure to remove the following files from the distribution. They are either not needed or generated on the fly when building this module in the core:
MANIFEST
META.yml
PPPort.pm
No known bugs.
Version 3.x, Copyright (C) 2004-2020, Marcus Holland-Moritz and Perl 5 porters
Version 2.x, Copyright (C) 2001, Paul Marquess.
Version 1.x, Copyright (C) 1999, Kenneth Albanowski.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
See ppport.h
and devel/regenerate
.