Perl / perl5

🐪 The Perl programming language
https://dev.perl.org/perl5/
Other
1.98k stars 559 forks source link

Add non-XS extensions to known_extensions #13038

Closed p5pRT closed 11 years ago

p5pRT commented 11 years ago

Migrated from rt.perl.org#118537 (status was 'resolved')

Searchable as RT118537$

p5pRT commented 11 years ago

From @nwc10

I think that we should change Configure so that known_extensions includes non-XS extensions. Currently\, not having the non-XS modules in known_extensions means that unlike XS modules\, you can't skip building any non-XS module\, as t/TEST and t/harness will always run their tests (which will fail)\, because the extension-test-skip logic uses known_extensions.

This means that

1) you can't elect not to build to-be-evicted modules 2) we can't put any platform-specific modules in ext (eg VMS​::Filespec)

If we change things so that known_extensions includes the nonxs_ext\, then both of the above are solved.

I don't think that it will break anything - grep.cpan.me suggests that there are only 10 non-core mentions of known_extensions\, and none of them rely on this existing behaviour.

Right now we have the wonderfully inconsistent situation (for Configure-generated config.sh) of

dynamic_ext​: built dynamically linked XS modules static_ext​: built statically linked XS modules nonxs_ext​: built pure-Perl modules (from ext/\, dist/ and cpan/) extensions​: "$dynamic_ext $static_ext $nonxs_ext" known_extensions​: *just* the XS modules shipped in ext/\, dist/ and cpan/

ie "extensions" is typically much larger than "known_extensions"

Moreover\, I believe that we're inconsistent between platforms\, in that on Win32\, "known_extensions" has everything.

This situation has come about through "organic growth"\, rather than design. I guess it's summarised as

0) Perl 5 predates CPAN 1) Originally Perl 5 only shipped with XS code in ext/ 2) Originally there was no concept of dual-life - if you wanted the   extensions in ext/\, you had to build them with perl 3) 15 years ago (today)\, Configure was patched to add nonxs_ext   (commit 4318d5a0158916ac) ready to support Errno   (Errno was added about two weeks later in commit eab60bb1f2e96e20)   [curiously that commit adds Errno to known_extensions but not to   extensions] 4) A few days later commit bfb7748a896459cc updates Configure so that   nonxs_ext *are* in extensions\, but are *not* in known_extensions.   The description of the change is​:
 
  Explicitly split list of extensions into 3 kinds​: dynamic\, static\,   and non-xs. The Configure variable $extensions now holds all three.   (The only current non-xs extension is Errno).

http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/1998-07/msg00136.html

  It also updates Porting/Glossary\, explicitly changing the description   of known_extensions from "list of all extensions included" to   "list of all XS extensions included"\, and extensions from   "all extension files linked into the package" to   "all extension files (both XS and non-xs linked into the package."

  [oooh\, that lack of closing ) is still present today]

[Note that Errno *is* architecture specific\, so gets installed into the same directory subtree as all the shared object]

Fast forward from 1998 to 2006

5) Commit 1d8961043b9b86e1 (or thereabouts) in April 2006 regenerates the   sample config.sh to this​:

  nonxs_ext='Compress/IO/Base Compress/IO/Zlib Compress/Zlib Errno'

at which point\, we have 3 more non-XS extensions\, all of which are architecture independent.

Subsequent re-arranging of dual-life modules in 2009 means that we've got a lot more.

Effectively\, the term "extensions" has been meaning "things we build via Makefile.PL" for at least 7 years\, if not 15.

Even though perlfaq7 says​:

  =head2 What's an extension?  
  An extension is a way of calling compiled C code from Perl. Reading   L\ is a good place to learn more about extensions.

which was added when the FAQ was added by commit 68dc074516a6859e back in March 1997 in the run-up to 5.004\, and has only had style changes since then. That must be one of the most Infrequently Asked Questions out there.

(the above is partly based on explanations from Craig Berry in http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2009-09/msg00688.html )

So I think it's sane to apply the attached 3 patches (which have been on a smoke-me quite happily)\, amended with this change to the Glossary​:

Inline Patch ```diff diff --git a/Porting/Glossary b/Porting/Glossary index b57f990..08a5020 100644 --- a/Porting/Glossary +++ b/Porting/Glossary @@ -2666,7 +2666,7 @@ expr (Loc.U): extensions (Extensions.U): This variable holds a list of all extension files (both XS and - non-xs linked into the package. It is propagated to Config.pm + non-xs) installed with the package. It is propagated to Config.pm and is typically used to test whether a particular extension is available. @@ -3620,8 +3620,8 @@ ivtype (perlxv.U): This variable contains the C type used for Perl's IV. known_extensions (Extensions.U): - This variable holds a list of all XS extensions included in - the package. + This variable holds a list of all extensions (both XS and non-xs) + included in the package. ksh (Loc.U): This variable is defined but not used by Configure. ```

I believe that the behaviour is win32/FindExt.pm is already this way. I don't know enough to edit configure.com to change it.

Nicholas Clark

p5pRT commented 11 years ago

From @nwc10

0001-Remove-Configure-code-that-supported-the-old-style-n.patch ```diff From 9f40039bbb7c5f2c3eacb0a2045c2667a29d7607 Mon Sep 17 00:00:00 2001 From: Nicholas Clark Date: Fri, 14 Jun 2013 16:19:06 +0200 Subject: [PATCH 1/3] Remove Configure code that supported the old-style nested layout for ext/ 5.10.1 switched to the new layout, so this code would only be useful for backporting to maint-5.8. That isn't going happen, so it can go. --- Configure | 83 ++++++++++++++++++++---------------------------------------- 1 files changed, 28 insertions(+), 55 deletions(-) diff --git a/Configure b/Configure index f8982cf..dbccc28 100755 --- a/Configure +++ b/Configure @@ -22372,51 +22372,34 @@ find_extensions=' DynaLoader|dynaload) ;; *) this_ext=`echo $xxx | $sed -e s/-/\\\//g`; - leaf=`echo $xxx | $sed -e s/.*-//`; - if $test -d File; then - if $test -f $xxx/$leaf.xs -o -f $xxx/$leaf.c; then - known_extensions="$known_extensions $1$this_ext"; - elif $test -f $xxx/Makefile.PL; then - nonxs_extensions="$nonxs_extensions $1$this_ext"; - else - if $test -d $xxx -a $# -lt 10; then - set $1$xxx/ $*; - cd "$xxx"; - eval $find_extensions; - cd ..; - shift; - fi; - fi; - else - echo " $known_extensions $nonxs_extensions" > $$.tmp; - if $contains " $this_ext " $$.tmp; then - echo >&4; - echo "Duplicate directories detected for extension $xxx" >&4; - echo "Configure cannot correctly recover from this - shall I abort?" >&4; - case "$knowitall" in - "") dflt=y;; - *) dflt=n;; - esac; - . ../UU/myread; - case "$ans" in - n*|N*) ;; - *) echo >&4; - echo "Ok. Stopping Configure." >&4; - echo "Please remove the duplicate directory (e.g. using git clean) and then re-run Configure" >&4; - exit 1;; - esac; - echo "Ok. You will need to correct config.sh before running make." >&4; - fi; - $ls -1 $xxx > $$.tmp; - if $contains "\.xs$" $$.tmp > /dev/null 2>&1; then - known_extensions="$known_extensions $this_ext"; - elif $contains "\.c$" $$.tmp > /dev/null 2>&1; then - known_extensions="$known_extensions $this_ext"; - elif $test -d $xxx; then - nonxs_extensions="$nonxs_extensions $this_ext"; - fi; - $rm -f $$.tmp; - fi + echo " $known_extensions $nonxs_extensions" > $$.tmp; + if $contains " $this_ext " $$.tmp; then + echo >&4; + echo "Duplicate directories detected for extension $xxx" >&4; + echo "Configure cannot correctly recover from this - shall I abort?" >&4; + case "$knowitall" in + "") dflt=y;; + *) dflt=n;; + esac; + . ../UU/myread; + case "$ans" in + n*|N*) ;; + *) echo >&4; + echo "Ok. Stopping Configure." >&4; + echo "Please remove the duplicate directory (e.g. using git clean) and then re-run Configure" >&4; + exit 1;; + esac; + echo "Ok. You will need to correct config.sh before running make." >&4; + fi; + $ls -1 $xxx > $$.tmp; + if $contains "\.xs$" $$.tmp > /dev/null 2>&1; then + known_extensions="$known_extensions $this_ext"; + elif $contains "\.c$" $$.tmp > /dev/null 2>&1; then + known_extensions="$known_extensions $this_ext"; + elif $test -d $xxx; then + nonxs_extensions="$nonxs_extensions $this_ext"; + fi; + $rm -f $$.tmp; ;; esac; done' @@ -22433,16 +22416,6 @@ cd "$rsrc/ext" set X shift eval $find_extensions -if $test -d File-Glob; then - : All ext/ flattened -else - # Special case: Add in modules that nest beyond the first level. - # Currently threads/shared and Hash/Util/FieldHash, since they are - # not picked up by the recursive find above (and adding in general - # recursive finding breaks SDBM_File/sdbm). - # A.D. 20011025 (SDBM), ajgough 20071008 (FieldHash) - known_extensions="$known_extensions threads/shared Hash/Util/FieldHash" -fi set X $known_extensions shift known_extensions=`echo "$*" | tr ' ' $trnl | $sort | tr $trnl ' '` -- 1.7.4.3 ```
p5pRT commented 11 years ago

From @nwc10

0002-Add-non-XS-extensions-to-known_extensions.patch ```diff From 929f9910c91c0cf611368fdd864c5bed13ff966d Mon Sep 17 00:00:00 2001 From: Nicholas Clark Date: Fri, 14 Jun 2013 16:56:28 +0200 Subject: [PATCH 2/3] Add non-XS extensions to known_extensions. Previously "known_extensions" was misnamed, as it only contained known XS extensions. grep.cpan.me suggests that there are only 10 mentions of it outside the core, and none of them rely on this existing behaviour. These changes need replicating into configure.com. --- Configure | 19 ++++++++++--------- win32/FindExt.pm | 6 ++++-- 2 files changed, 14 insertions(+), 11 deletions(-) diff --git a/Configure b/Configure index dbccc28..e4e7d81 100755 --- a/Configure +++ b/Configure @@ -22353,10 +22353,10 @@ esac : Check extensions echo " " echo "Looking for extensions..." >&4 -: If we are using the old config.sh, known_extensions may contain -: old or inaccurate or duplicate values. -known_extensions='' +: If we are using the old config.sh, nonxs_extensions and xs_extensions may +: contain old or inaccurate or duplicate values. nonxs_extensions='' +xs_extensions='' : We do not use find because it might not be available. : We do not just use MANIFEST because the user may have dropped : some additional extensions into the source tree and expect them @@ -22372,7 +22372,7 @@ find_extensions=' DynaLoader|dynaload) ;; *) this_ext=`echo $xxx | $sed -e s/-/\\\//g`; - echo " $known_extensions $nonxs_extensions" > $$.tmp; + echo " $xs_extensions $nonxs_extensions" > $$.tmp; if $contains " $this_ext " $$.tmp; then echo >&4; echo "Duplicate directories detected for extension $xxx" >&4; @@ -22393,9 +22393,9 @@ find_extensions=' fi; $ls -1 $xxx > $$.tmp; if $contains "\.xs$" $$.tmp > /dev/null 2>&1; then - known_extensions="$known_extensions $this_ext"; + xs_extensions="$xs_extensions $this_ext"; elif $contains "\.c$" $$.tmp > /dev/null 2>&1; then - known_extensions="$known_extensions $this_ext"; + xs_extensions="$xs_extensions $this_ext"; elif $test -d $xxx; then nonxs_extensions="$nonxs_extensions $this_ext"; fi; @@ -22416,17 +22416,18 @@ cd "$rsrc/ext" set X shift eval $find_extensions -set X $known_extensions +set X $xs_extensions shift -known_extensions=`echo "$*" | tr ' ' $trnl | $sort | tr $trnl ' '` +xs_extensions=`echo "$*" | tr ' ' $trnl | $sort | tr $trnl ' '` set X $nonxs_extensions shift nonxs_extensions=`echo "$*" | tr ' ' $trnl | $sort | tr $trnl ' '` cd "$tdir" +known_extensions=`echo $nonxs_extensions $xs_extensions | tr ' ' $trnl | $sort | tr $trnl ' '` : Now see which are supported on this system. avail_ext='' -for xxx in $known_extensions ; do +for xxx in $xs_extensions ; do case "$xxx" in DB_File|db_file) case "$i_db" in diff --git a/win32/FindExt.pm b/win32/FindExt.pm index 2e4d6c1..41065d2 100644 --- a/win32/FindExt.pm +++ b/win32/FindExt.pm @@ -92,8 +92,10 @@ sub _ext_ne { } *extensions = _ext_ne('known'); -# faithfully copy Configure in not including nonxs extensions for the nonce -*known_extensions = _ext_ne('nonxs'); + +sub known_extensions { + sort keys %ext; +} sub is_static { -- 1.7.4.3 ```
p5pRT commented 11 years ago

From @nwc10

0003-Also-add-Encode-s-sub-modules-to-known_extensions-wh.patch ```diff From 8fecd458228dff0dea7e8325154d57a702534172 Mon Sep 17 00:00:00 2001 From: Nicholas Clark Date: Tue, 18 Jun 2013 12:00:38 +0200 Subject: [PATCH 3/3] Also add Encode's sub-modules to known_extensions when building statically. There is code in Configure to treat Encode specially when it is specified as a statically linked extension. By default, Encode builds separated shared objects for each of its subdirectories. This works well with DynaLoader, and the top level perl Makefile doesn't even notice this, because it doesn't have to list these libraries as things it links with at compile time. For a static link, Encode builds a separate *.a file for each of its subdirectories. The top level Makefile *does* need to know about these, as a static link requires them all to be listed. Hence the work-around is to treat Encode as a set of nested modules if linked statically. We can't do this in Makefile.SH because the various Encode submodules are installed as separate *.a files in the tree, and so need to continue to be treated as distinct modules in case ExtUtils::MakeMaker is asked to (re)link a static perl with an additional extension. I suspect that the most elegant fix would be to tweak Encode's top level Makefile.PL to link everything into one *.a if it is building statically. I'm not sure how to do that, and it would need to be accepted upstream. --- Configure | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Configure b/Configure index e4e7d81..21c495e 100755 --- a/Configure +++ b/Configure @@ -22730,6 +22730,7 @@ case " $static_ext " in cd "$rsrc/cpan" for xxx in `ls Encode/*/Makefile.PL|awk -F/ '{print $2}'`; do static_ext="$static_ext Encode/$xxx" + known_extensions="$known_extensions Encode/$xxx" done cd "$tdir" ;; -- 1.7.4.3 ```
p5pRT commented 11 years ago

From @steve-m-hay

Nicholas Clark (via RT) wrote on 2013-06-19​:

Right now we have the wonderfully inconsistent situation (for Configure-generated config.sh) of

dynamic_ext​: built dynamically linked XS modules static_ext​:
built statically linked XS modules nonxs_ext​: built pure-Perl modules (from ext/\, dist/ and cpan/) extensions​:
"$dynamic_ext $static_ext $nonxs_ext" known_extensions​: *just* the XS modules shipped in ext/\, dist/ and cpan/

ie "extensions" is typically much larger than "known_extensions"

Moreover\, I believe that we're inconsistent between platforms\, in that on Win32\, "known_extensions" has everything.

I've just checked and 'known_extensions' doesn't have everything on Windows. It actually has the same bizarre situation that you describe for *nix\, which has bugged me before. I would be in favour of changing it if nobody can think of anything that would be broken by it.

FYI\, I've attached the output of the relevant "perl -V​:..." options in a default build\, and again in an ALL_STATIC=define build\, on Windows.

p5pRT commented 11 years ago

From @steve-m-hay

Default build​:

C​:\Dev\Git\perl>perl -V​:dynamic_ext dynamic_ext='B Compress/Raw/Bzip2 Compress/Raw/Zlib Cwd Data/Dumper Devel/PPPort Devel/Peek Digest/MD5 Digest/SHA Encode Fcntl File/DosGlob File/Glob Filter/Uti l/Call Hash/Util Hash/Util/FieldHash IO List/Util MIME/Base64 Math/BigInt/FastCa lc Opcode POSIX PerlIO/encoding PerlIO/mmap PerlIO/scalar PerlIO/via SDBM_File S ocket Storable Sys/Hostname Tie/Hash/NamedCapture Time/HiRes Time/Piece Unicode/ Collate Unicode/Normalize Win32 Win32API/File XS/APItest XS/Typemap arybase attr ibutes mro re threads threads/shared';

C​:\Dev\Git\perl>perl -V​:static_ext static_ext='Win32CORE';

C​:\Dev\Git\perl>perl -V​:nonxs_ext nonxs_ext='Archive/Tar Attribute/Handlers AutoLoader B/Debug B/Deparse CGI CPAN CPAN/Meta CPAN/Meta/Requirements CPAN/Meta/YAML Carp Config/Perl/V Devel/SelfStu bber Digest Dumpvalue Env Errno ExtUtils/CBuilder ExtUtils/Command ExtUtils/Cons tant ExtUtils/Install ExtUtils/MakeMaker ExtUtils/Manifest ExtUtils/ParseXS File /Fetch File/Path File/Temp FileCache Filter/Simple Getopt/Long HTTP/Tiny I18N/Co llate I18N/LangTags IO/Compress IO/Zlib IPC/Cmd IPC/Open3 JSON/PP Locale/Codes L ocale/Maketext Locale/Maketext/Simple Math/BigInt Math/BigRat Math/Complex Memoi ze Module/Build Module/CoreList Module/Load Module/Load/Conditional Module/Loade d Module/Metadata NEXT Net/Ping Package/Constants Params/Check Parse/CPAN/Meta P erl/OSType PerlIO/via/QuotedPrint Pod/Checker Pod/Escapes Pod/Functions Pod/Html Pod/Parser Pod/Perldoc Pod/Simple Pod/Usage Safe Search/Dict SelfLoader Term/AN SIColor Term/Cap Term/Complete Term/ReadLine Test Test/Harness Test/Simple Text/ Abbrev Text/Balanced Text/ParseWords Text/Tabs Thread/Queue Thread/Semaphore Tie /File Tie/Memoize Tie/RefHash Time/Local XSLoader autodie autouse base bignum co nstant encoding/warnings if lib libnet parent perlfaq podlators';

C​:\Dev\Git\perl>perl -V​:extensions extensions='Archive/Tar Attribute/Handlers AutoLoader B B/Debug B/Deparse CGI CP AN CPAN/Meta CPAN/Meta/Requirements CPAN/Meta/YAML Carp Compress/Raw/Bzip2 Compr ess/Raw/Zlib Config/Perl/V Cwd Data/Dumper Devel/PPPort Devel/Peek Devel/SelfStu bber Digest Digest/MD5 Digest/SHA Dumpvalue Encode Env Errno ExtUtils/CBuilder E xtUtils/Command ExtUtils/Constant ExtUtils/Install ExtUtils/MakeMaker ExtUtils/M anifest ExtUtils/ParseXS Fcntl File/DosGlob File/Fetch File/Glob File/Path File/ Temp FileCache Filter/Simple Filter/Util/Call Getopt/Long HTTP/Tiny Hash/Util Ha sh/Util/FieldHash I18N/Collate I18N/LangTags IO IO/Compress IO/Zlib IPC/Cmd IPC/ Open3 JSON/PP List/Util Locale/Codes Locale/Maketext Locale/Maketext/Simple MIME /Base64 Math/BigInt Math/BigInt/FastCalc Math/BigRat Math/Complex Memoize Module /Build Module/CoreList Module/Load Module/Load/Conditional Module/Loaded Module/ Metadata NEXT Net/Ping Opcode POSIX Package/Constants Params/Check Parse/CPAN/Me ta Perl/OSType PerlIO/encoding PerlIO/mmap PerlIO/scalar PerlIO/via PerlIO/via/Q uotedPrint Pod/Checker Pod/Escapes Pod/Functions Pod/Html Pod/Parser Pod/Perldoc Pod/Simple Pod/Usage SDBM_File Safe Search/Dict SelfLoader Socket Storable Sys/ Hostname Term/ANSIColor Term/Cap Term/Complete Term/ReadLine Test Test/Harness T est/Simple Text/Abbrev Text/Balanced Text/ParseWords Text/Tabs Thread/Queue Thre ad/Semaphore Tie/File Tie/Hash/NamedCapture Tie/Memoize Tie/RefHash Time/HiRes T ime/Local Time/Piece Unicode/Collate Unicode/Normalize Win32 Win32API/File Win32 CORE XS/APItest XS/Typemap XSLoader arybase attributes autodie autouse base bign um constant encoding/warnings if lib libnet mro parent perlfaq podlators re thre ads threads/shared';

C​:\Dev\Git\perl>perl -V​:known_extensions known_extensions='B Compress/Raw/Bzip2 Compress/Raw/Zlib Cwd DB_File Data/Dumper Devel/PPPort Devel/Peek Digest/MD5 Digest/SHA Encode Fcntl File/DosGlob File/Gl ob Filter/Util/Call GDBM_File Hash/Util Hash/Util/FieldHash I18N/Langinfo IO IPC /SysV List/Util MIME/Base64 Math/BigInt/FastCalc NDBM_File ODBM_File Opcode POSI X PerlIO/encoding PerlIO/mmap PerlIO/scalar PerlIO/via SDBM_File Socket Storable Sys/Hostname Sys/Syslog Tie/Hash/NamedCapture Time/HiRes Time/Piece Unicode/Col late Unicode/Normalize VMS/DCLsym VMS/Stdio Win32 Win32API/File Win32CORE XS/API test XS/Typemap arybase attributes mro re threads threads/shared';

Build with ALL_STATIC=define​:

C​:\Dev\Git\perl>perl -V​:dynamic_ext dynamic_ext='SDBM_File';

C​:\Dev\Git\perl>perl -V​:static_ext static_ext='B Compress/Raw/Bzip2 Compress/Raw/Zlib Cwd Data/Dumper Devel/PPPort Devel/Peek Digest/MD5 Digest/SHA Encode Encode/Byte Encode/CN Encode/EBCDIC Enco de/JP Encode/KR Encode/Symbol Encode/TW Encode/Unicode Fcntl File/DosGlob File/G lob Filter/Util/Call Hash/Util Hash/Util/FieldHash IO List/Util MIME/Base64 Math /BigInt/FastCalc Opcode POSIX PerlIO/encoding PerlIO/mmap PerlIO/scalar PerlIO/v ia Socket Storable Sys/Hostname Tie/Hash/NamedCapture Time/HiRes Time/Piece Unic ode/Collate Unicode/Normalize Win32 Win32API/File Win32CORE XS/APItest XS/Typema p arybase attributes mro re threads threads/shared';

C​:\Dev\Git\perl>perl -V​:nonxs_ext nonxs_ext='Archive/Tar Attribute/Handlers AutoLoader B/Debug B/Deparse CGI CPAN CPAN/Meta CPAN/Meta/Requirements CPAN/Meta/YAML Carp Config/Perl/V Devel/SelfStu bber Digest Dumpvalue Env Errno ExtUtils/CBuilder ExtUtils/Command ExtUtils/Cons tant ExtUtils/Install ExtUtils/MakeMaker ExtUtils/Manifest ExtUtils/ParseXS File /Fetch File/Path File/Temp FileCache Filter/Simple Getopt/Long HTTP/Tiny I18N/Co llate I18N/LangTags IO/Compress IO/Zlib IPC/Cmd IPC/Open3 JSON/PP Locale/Codes L ocale/Maketext Locale/Maketext/Simple Math/BigInt Math/BigRat Math/Complex Memoi ze Module/Build Module/CoreList Module/Load Module/Load/Conditional Module/Loade d Module/Metadata NEXT Net/Ping Package/Constants Params/Check Parse/CPAN/Meta P erl/OSType PerlIO/via/QuotedPrint Pod/Checker Pod/Escapes Pod/Functions Pod/Html Pod/Parser Pod/Perldoc Pod/Simple Pod/Usage Safe Search/Dict SelfLoader Term/AN SIColor Term/Cap Term/Complete Term/ReadLine Test Test/Harness Test/Simple Text/ Abbrev Text/Balanced Text/ParseWords Text/Tabs Thread/Queue Thread/Semaphore Tie /File Tie/Memoize Tie/RefHash Time/Local XSLoader autodie autouse base bignum co nstant encoding/warnings if lib libnet parent perlfaq podlators';

C​:\Dev\Git\perl>perl -V​:extensions extensions='Archive/Tar Attribute/Handlers AutoLoader B B/Debug B/Deparse CGI CP AN CPAN/Meta CPAN/Meta/Requirements CPAN/Meta/YAML Carp Compress/Raw/Bzip2 Compr ess/Raw/Zlib Config/Perl/V Cwd Data/Dumper Devel/PPPort Devel/Peek Devel/SelfStu bber Digest Digest/MD5 Digest/SHA Dumpvalue Encode Encode/Byte Encode/CN Encode/ EBCDIC Encode/JP Encode/KR Encode/Symbol Encode/TW Encode/Unicode Env Errno ExtU tils/CBuilder ExtUtils/Command ExtUtils/Constant ExtUtils/Install ExtUtils/MakeM aker ExtUtils/Manifest ExtUtils/ParseXS Fcntl File/DosGlob File/Fetch File/Glob File/Path File/Temp FileCache Filter/Simple Filter/Util/Call Getopt/Long HTTP/Ti ny Hash/Util Hash/Util/FieldHash I18N/Collate I18N/LangTags IO IO/Compress IO/Zl ib IPC/Cmd IPC/Open3 JSON/PP List/Util Locale/Codes Locale/Maketext Locale/Maket ext/Simple MIME/Base64 Math/BigInt Math/BigInt/FastCalc Math/BigRat Math/Complex Memoize Module/Build Module/CoreList Module/Load Module/Load/Conditional Module /Loaded Module/Metadata NEXT Net/Ping Opcode POSIX Package/Constants Params/Chec k Parse/CPAN/Meta Perl/OSType PerlIO/encoding PerlIO/mmap PerlIO/scalar PerlIO/v ia PerlIO/via/QuotedPrint Pod/Checker Pod/Escapes Pod/Functions Pod/Html Pod/Par ser Pod/Perldoc Pod/Simple Pod/Usage SDBM_File Safe Search/Dict SelfLoader Socke t Storable Sys/Hostname Term/ANSIColor Term/Cap Term/Complete Term/ReadLine Test Test/Harness Test/Simple Text/Abbrev Text/Balanced Text/ParseWords Text/Tabs Th read/Queue Thread/Semaphore Tie/File Tie/Hash/NamedCapture Tie/Memoize Tie/RefHa sh Time/HiRes Time/Local Time/Piece Unicode/Collate Unicode/Normalize Win32 Win3 2API/File Win32CORE XS/APItest XS/Typemap XSLoader arybase attributes autodie au touse base bignum constant encoding/warnings if lib libnet mro parent perlfaq po dlators re threads threads/shared';

C​:\Dev\Git\perl>perl -V​:known_extensions known_extensions='B Compress/Raw/Bzip2 Compress/Raw/Zlib Cwd DB_File Data/Dumper Devel/PPPort Devel/Peek Digest/MD5 Digest/SHA Encode Encode/Byte Encode/CN Enco de/EBCDIC Encode/JP Encode/KR Encode/Symbol Encode/TW Encode/Unicode Fcntl File/ DosGlob File/Glob Filter/Util/Call GDBM_File Hash/Util Hash/Util/FieldHash I18N/ Langinfo IO IPC/SysV List/Util MIME/Base64 Math/BigInt/FastCalc NDBM_File ODBM_F ile Opcode POSIX PerlIO/encoding PerlIO/mmap PerlIO/scalar PerlIO/via SDBM_File Socket Storable Sys/Hostname Sys/Syslog Tie/Hash/NamedCapture Time/HiRes Time/Pi ece Unicode/Collate Unicode/Normalize VMS/DCLsym VMS/Stdio Win32 Win32API/File W in32CORE XS/APItest XS/Typemap arybase attributes mro re threads threads/shared' ;

p5pRT commented 11 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 11 years ago

From @nwc10

On Wed\, Jun 19\, 2013 at 12​:15​:36PM +0100\, Steve Hay wrote​:

Nicholas Clark (via RT) wrote on 2013-06-19​:

Moreover\, I believe that we're inconsistent between platforms\, in that on Win32\, "known_extensions" has everything.

I've just checked and 'known_extensions' doesn't have everything on Windows. It actually has the same bizarre situation that you describe for *nix\, which has bugged me before. I would be in favour of changing it if nobody can think of anything that would be broken by it.

Oh\, I'm a muppet. You're right - it doesn't. Well\, it didn't\, until I changed it in one of the patches I attached. (It's hiding behind a wall of Configure code)

FYI\, I've attached the output of the relevant "perl -V​:..." options in a default build\, and again in an ALL_STATIC=define build\, on Windows.

Thanks

Nicholas Clark

p5pRT commented 11 years ago

From @doughera88

On Wed\, Jun 19\, 2013 at 02​:25​:32AM -0700\, Nicholas Clark wrote​:

# New Ticket Created by Nicholas Clark # Please include the string​: [perl #118537] # in the subject line of all future correspondence about this issue. # \<URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=118537 >

I think that we should change Configure so that known_extensions includes non-XS extensions. Currently\, not having the non-XS modules in known_extensions means that unlike XS modules\, you can't skip building any non-XS module\, as t/TEST and t/harness will always run their tests (which will fail)\, because the extension-test-skip logic uses known_extensions.

This means that

1) you can't elect not to build to-be-evicted modules 2) we can't put any platform-specific modules in ext (eg VMS​::Filespec)

If we change things so that known_extensions includes the nonxs_ext\, then both of the above are solved.

I don't think that it will break anything - grep.cpan.me suggests that there are only 10 non-core mentions of known_extensions\, and none of them rely on this existing behaviour.

Broadly speaking\, I think it should be safe to change it to whatever works.

I think t/TEST and t/harness are wrong. Their extension-test-skip logic should just use $extensions. Nothing running after Configure should use $known_extensions. It was not intended to be useful for anything after Configure time\, though the documentation isn't explicit about that. $known_extensions lists all the extensions Configure 'knows' about. Only the subset in $extensions was actually selected by the user to be included with perl. The user is not required to install every extension shipped with the perl tarball.

Right now we have the wonderfully inconsistent situation (for Configure-generated config.sh) of

dynamic_ext​: built dynamically linked XS modules static_ext​: built statically linked XS modules nonxs_ext​: built pure-Perl modules (from ext/\, dist/ and cpan/) extensions​: "$dynamic_ext $static_ext $nonxs_ext" known_extensions​: *just* the XS modules shipped in ext/\, dist/ and cpan/

ie "extensions" is typically much larger than "known_extensions"

Yes\, but the intent was you wouldn't care\, since known_extensions wasn't intended to be used.

This situation has come about through "organic growth"\, rather than design.

Yes\, that's undoubtedly true.

I guess it's summarised as

0) Perl 5 predates CPAN 1) Originally Perl 5 only shipped with XS code in ext/

Yes. libraries were in lib/. In perl4\, you had to relink against uperl.o and build a new executable (e.g. curseperl) to link in other C libraries.
In perl5\, you could use dynamic loading to load in the extensions.

2) Originally there was no concept of dual-life - if you wanted the extensions in ext/\, you had to build them with perl

There were outside modules; they just didn't have a ready toolchain. Tk was one of the first big ones that I recall. If you dropped them in the ext/ directory\, Configure was supposed to handle them automatically\, but if you wanted to build them later\, that was ok too.

3) 15 years ago (today)\, Configure was patched to add nonxs_ext (commit 4318d5a0158916ac) ready to support Errno (Errno was added about two weeks later in commit eab60bb1f2e96e20) [curiously that commit adds Errno to known_extensions but not to extensions] 4) A few days later commit bfb7748a896459cc updates Configure so that nonxs_ext *are* in extensions\, but are *not* in known_extensions. The description of the change is​:

Explicitly split list of extensions into 3 kinds&#8203;:  dynamic\, static\,
and non\-xs\.  The Configure variable $extensions now holds all three\.
\(The only current non\-xs extension is Errno\)\.

http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/1998-07/msg00136.html

It also updates Porting/Glossary\, explicitly changing the description
of known\_extensions from "list of all extensions included" to
"list of all XS extensions included"\, and extensions from
"all extension files linked into the package" to
"all extension files \(both XS and non\-xs linked into the package\."

\[oooh\, that lack of closing \) is still present today\]

Yeah\, I tend to do that.

Anyway\, feel free to fix and document it however makes sense in the new era. However\, I'd recommend documenting that nobody should actually be *using* known_extensions for anything. They should be using $extensions.

--   Andy Dougherty doughera@​lafayette.edu

p5pRT commented 11 years ago

From @nwc10

I've patched Configure (and related files) to do this.

commit f7b3892b9e45c6994685c9a625d3ea2b52642167 Author​: Nicholas Clark \nick@&#8203;ccl4\.org Date​: Fri Jun 14 16​:56​:28 2013 +0200

  Add non-XS extensions to known_extensions.  
  Previously "known_extensions" was misnamed\, as it only contained known XS   extensions. grep.cpan.me suggests that there are only 10 mentions of it   outside the core\, and none of them rely on this existing behaviour.  
  Update the descriptions of extensions\, known_extensions and nonxs_ext in   Porting/Glossary.  
  These changes need replicating into configure.com.

On Fri\, Jun 21\, 2013 at 05​:11​:01PM -0400\, Andy Dougherty wrote​:

On Wed\, Jun 19\, 2013 at 02​:25​:32AM -0700\, Nicholas Clark wrote​:

I think that we should change Configure so that known_extensions includes non-XS extensions. Currently\, not having the non-XS modules in known_extensions means that unlike XS modules\, you can't skip building any non-XS module\, as t/TEST and t/harness will always run their tests (which will fail)\, because the extension-test-skip logic uses known_extensions.

This means that

1) you can't elect not to build to-be-evicted modules 2) we can't put any platform-specific modules in ext (eg VMS​::Filespec)

I haven't done either of these. I will move VMS​::Filespec. I'm not proposing to do (1) myself.

Broadly speaking\, I think it should be safe to change it to whatever works.

I think t/TEST and t/harness are wrong. Their extension-test-skip logic should just use $extensions. Nothing running after Configure should use $known_extensions. It was not intended to be useful for anything after Configure time\, though the documentation isn't explicit about that. $known_extensions lists all the extensions Configure 'knows' about. Only the subset in $extensions was actually selected by the user to be included with perl. The user is not required to install every extension shipped with the perl tarball.

I haven't done this. I guess I should open a new ticket in RT for this. For me it's a "nice to have" but there's a bunch of other stuff that is nicer. But it's a pure-Perl task\, so anyone on the list could take it on.

Anyway\, feel free to fix and document it however makes sense in the new era. However\, I'd recommend documenting that nobody should actually be *using* known_extensions for anything. They should be using $extensions.

I made these changes to Porting/Glossary​:

Inline Patch ```diff diff --git a/Porting/Glossary b/Porting/Glossary index b57f990..1f5b8c9 100644 --- a/Porting/Glossary +++ b/Porting/Glossary @@ -2666,7 +2666,7 @@ expr (Loc.U): extensions (Extensions.U): This variable holds a list of all extension files (both XS and - non-xs linked into the package. It is propagated to Config.pm + non-xs) installed with the package. It is propagated to Config.pm and is typically used to test whether a particular extension is available. @@ -3620,8 +3620,12 @@ ivtype (perlxv.U): This variable contains the C type used for Perl's IV. known_extensions (Extensions.U): - This variable holds a list of all XS extensions included in - the package. + This variable holds a list of all extensions (both XS and non-xs) + included in the package source distribution. This information is + only really of use during the Perl build, as the list makes no + distinction between extensions which were build and installed, and + those which where not. See "extensions" for the list of extensions + actually built and available. ksh (Loc.U): This variable is defined but not used by Configure. @@ -3990,8 +3994,10 @@ nm_so_opt (usenm.U): Maybe Linux should just always set usenm=false. nonxs_ext (Extensions.U): - This variable holds a list of all non-xs extensions included - in the package. All of them will be built. + This variable holds a list of all non-xs extensions built and + installed by the package. By default, all non-xs extensions + distributed will be built, with the exception of platform-specific + extensions (currently only one VMS specific extension). nroff (Loc.U): This variable is used internally by Configure to determine the ```

I don't think that this RT can be considered resolved until configure.com has been updated to be consistent with Configure.

Nicholas Clark

p5pRT commented 11 years ago

From @craigberry

On Tue\, Jul 2\, 2013 at 8​:14 AM\, Nicholas Clark \nick@&#8203;ccl4\.org wrote​:

nonxs_ext (Extensions.U)​: - This variable holds a list of all non-xs extensions included - in the package. All of them will be built. + This variable holds a list of all non-xs extensions built and + installed by the package. By default\, all non-xs extensions + distributed will be built\, with the exception of platform-specific + extensions (currently only one VMS specific extension).

This statement is true for non-VMS platforms\, but there is a potential confusion here in that VMS​::Filespec is an XS extension on VMS (with the XS bits living in vms/vms.c)\, but if you attempt to use it on another platform\, you get (very out-of-date) pure-Perl overrides.

I don't think that this RT can be considered resolved until configure.com has been updated to be consistent with Configure.

Working on it. My initial attempt put VMS​::Filespec in nonxs_ext\, which I think made it get the wrong arguments to TestInit.pm (in any case the test failed). It doesn't really belong in dynamic_ext since it's not dynamic\, nor does it really belong in static_ext since it's not built the way a statically linked extension would be. For now\, keeping it out of nonxs_ext is probably good enough. Until it got moved to ext/\, it didn't appear in any of the lists\, not even known_extensions.

p5pRT commented 11 years ago

From @tux

On Wed\, 3 Jul 2013 10​:45​:28 -0500\, "Craig A. Berry" \craig\.a\.berry@&#8203;gmail\.com wrote​:

On Tue\, Jul 2\, 2013 at 8​:14 AM\, Nicholas Clark \nick@&#8203;ccl4\.org wrote​:

nonxs_ext (Extensions.U)​: - This variable holds a list of all non-xs extensions included - in the package. All of them will be built. + This variable holds a list of all non-xs extensions built and + installed by the package. By default\, all non-xs extensions + distributed will be built\, with the exception of platform-specific + extensions (currently only one VMS specific extension).

This statement is true for non-VMS platforms\, but there is a potential confusion here in that VMS​::Filespec is an XS extension on VMS (with the XS bits living in vms/vms.c)\, but if you attempt to use it on another platform\, you get (very out-of-date) pure-Perl overrides.

I don't think that this RT can be considered resolved until configure.com has been updated to be consistent with Configure.

Working on it. My initial attempt put VMS​::Filespec in nonxs_ext\, which I think made it get the wrong arguments to TestInit.pm (in any case the test failed). It doesn't really belong in dynamic_ext since it's not dynamic\, nor does it really belong in static_ext since it's not built the way a statically linked extension would be. For now\, keeping it out of nonxs_ext is probably good enough. Until it got moved to ext/\, it didn't appear in any of the lists\, not even known_extensions.

For the record\, the changes have all been backported to meta already.

-- H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/ using perl5.00307 .. 5.19 porting perl5 on HP-UX\, AIX\, and openSUSE http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/ http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

p5pRT commented 11 years ago

From @craigberry

On Wed\, Jul 3\, 2013 at 10​:45 AM\, Craig A. Berry \craig\.a\.berry@&#8203;gmail\.com wrote​:

On Tue\, Jul 2\, 2013 at 8​:14 AM\, Nicholas Clark \nick@&#8203;ccl4\.org wrote​:

I don't think that this RT can be considered resolved until configure.com has been updated to be consistent with Configure.

Working on it.

Now done as 86ad859eff07e4c550.

p5pRT commented 11 years ago

@cpansprout - Status changed from 'open' to 'resolved'