Closed p5pRT closed 17 years ago
This is a bug report for perl from tcallawa@redhat.com\, generated with the help of perlbug 1.35 running under perl v5.8.8.
----------------------------------------------------------------- As part of the review process for Perl\, I'm opening bugs for the long list of patches to perl that Fedora has been carrying around (in some cases\, for years)\, in the hopes that these fixes can be applied upstream. Unfortunately\, I am not the original author of these patches\, so please be patient with me as I attempt to describe why Fedora uses them.
This patch creates an incpush_oldversion() function\, used for compatibility for older perl bits. For example\, Fedora treats 5.8.7 5.8.6 5.8.5 as compatible to 5.8.8.
STRINGIFY(PERL_VERSION) "." STRINGIFY(PERLSUBVERSION) +STATIC void +incpush_oldversion(pTHX\ char *dir) +{ +#ifdef PERL_INC_VERSION_LIST + const char *incverlist[] = { PERL_INC_VERSION_LIST }; + const char **incver; + const char *verdir; + + verdir = strstr(dir\, VERSION_DIRECTORY_STRING); + if (!verdir) + return; + + for (incver = incverlist; *incver; incver++) { + char *new_dir = malloc(strlen(dir) + strlen(*incver) + 2); + char *p = new_dir; + + strcpy(new_dir\, dir); + p += verdir - dir + 1; /* advance to char following '/' in VERSION_DIRECTORY_STRING */ + memcpy(p\, *incver\, strlen(*incver)); /* copy incver there instead */ + p += strlen(*incver); /* advance past version we just copied */ + strcpy(p\, verdir + strlen(VERSION_DIRECTORY_STRING)); /* and copy the rest of the original dir */ + + incpush(new_dir\, FALSE\, FALSE\, FALSE); + free(new_dir); + } +#endif +} + + /* Push a directory onto @INC if it exists. Generate a new SV if we do this\, to save needing to copy the SV we push onto @INC */
On 22/02/07\, via RT Tom Callaway \perlbug\-followup@​perl\.org wrote:
This patch creates an incpush_oldversion() function\, used for compatibility for older perl bits. For example\, Fedora treats 5.8.7 5.8.6 5.8.5 as compatible to 5.8.8.
Configure searches for old dirs already to add to @INC. But\, that's not enough when you're building an rpm in a clean environment for distribution. In this case\, I think you should use the Configure argument -Dotherlibdirs=\<list of dirs to add at the end of @INC>. This should be sufficient\, I suppose?
The RT System itself - Status changed from 'new' to 'open'
Tom Callaway (via RT) wrote:
This patch creates an incpush_oldversion() function\, used for compatibility for older perl bits. For example\, Fedora treats 5.8.7 5.8.6 5.8.5 as compatible to 5.8.8.
And\, if I'm not mistaken\, has a side effect of making the @INC path huge\, which causes untold difficulties with Module::Build (and probably EU::MM) when trying to create commandline equivalents during the build process. I would *strongly* recommend that this particular patch be not applied anywhere and Fedora be left to dangle in the wind with this particular misfeature.
There should be a better way to promote binary-compatibility than generating a cross-product in @INC directories...
My 2 cents
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham\, MD 20706 301-459-3366 x.5010 fax 301-429-5748
On Fri\, 23 Feb 2007\, Rafael Garcia-Suarez wrote:
On 22/02/07\, via RT Tom Callaway \perlbug\-followup@​perl\.org wrote:
This patch creates an incpush_oldversion() function\, used for compatibility for older perl bits. For example\, Fedora treats 5.8.7 5.8.6 5.8.5 as compatible to 5.8.8.
Configure searches for old dirs already to add to @INC. But\, that's not enough when you're building an rpm in a clean environment for distribution. In this case\, I think you should use the Configure argument -Dotherlibdirs=\<list of dirs to add at the end of @INC>. This should be sufficient\, I suppose?
While that should work\, the intent was that you could also simply override Configure's autodetection with Configure -Dinc_version_list='...' . If that won't work for some reason\, then I'd appreciate hearing about it.
-- Andy Dougherty doughera@lafayette.edu
On Friday 23 February 2007 06:14\, John Peacock wrote:
Tom Callaway (via RT) wrote:
This patch creates an incpush_oldversion() function\, used for compatibility for older perl bits. For example\, Fedora treats 5.8.7 5.8.6 5.8.5 as compatible to 5.8.8.
And\, if I'm not mistaken\, has a side effect of making the @INC path huge\, which causes untold difficulties with Module::Build (and probably EU::MM) when trying to create commandline equivalents during the build process.
It also slows down program startup time with lots of stat calls. This can be substantial for programs which run frequently and terminate quickly:
http://www.perl.com/pub/a/2005/12/21/a_timely_start.html
-- c
On Fri\, 23 Feb 2007\, chromatic wrote:
On Friday 23 February 2007 06:14\, John Peacock wrote:
Tom Callaway (via RT) wrote:
This patch creates an incpush_oldversion() function\, used for compatibility for older perl bits. For example\, Fedora treats 5.8.7 5.8.6 5.8.5 as compatible to 5.8.8.
And\, if I'm not mistaken\, has a side effect of making the @INC path huge\, which causes untold difficulties with Module::Build (and probably EU::MM) when trying to create commandline equivalents during the build process.
It also slows down program startup time with lots of stat calls. This can be substantial for programs which run frequently and terminate quickly:
http​://www\.perl\.com/pub/a/2005/12/21/a\_timely\_start\.html
However\, having a "huge" @INC is a perfectly reasonable configuration possibility\, and it's supposed to be explicitly supported. This precise back-searching capability was put in as one possible way to allow smooth upgrade paths. I know I use it routinely. Yes\, it does have performance implications\, but other issues may be more important in some cases.
-- Andy Dougherty doughera@lafayette.edu
Andy Dougherty wrote:
However\, having a "huge" @INC is a perfectly reasonable configuration possibility\, and it's supposed to be explicitly supported. This precise back-searching capability was put in as one possible way to allow smooth upgrade paths. I know I use it routinely. Yes\, it does have performance implications\, but other issues may be more important in some cases.
I'm not arguing that it doesn't work\, at least as long as you are in Perl. If you try and manufacture a commandline which includes all of the @INC entries (for example during testing)\, you can run into commandline-length issues on various platforms that are very non-obvious to debug. I don't think that tripling or quadrupling the number of @INC entries\, on the offchance that there might be some code installed under one of the previous releases\, is the best way to deal with this particular problem.
Perl could just as easily be taught to recursively probe ($]-0.000_001) if those paths exist\, rather than make those path's explicitly listed in @INC. Of course\, that would be modulo binary incompatibility boundaries (e.g. between 5.8.0 and 5.8.1?). Of course the other difficulty is a distro-provided i586 path and a self-compiled i686 path are technically compatible (the 686 can read the 586 binaries)\, but I don't think there is any to automate that...
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham\, MD 20706 301-459-3366 x.5010 fax 301-429-5748
On Fri\, 23 Feb 2007\, John Peacock wrote:
Andy Dougherty wrote:
However\, having a "huge" @INC is a perfectly reasonable configuration possibility\, and it's supposed to be explicitly supported. This precise back-searching capability was put in as one possible way to allow smooth upgrade paths. I know I use it routinely. Yes\, it does have performance implications\, but other issues may be more important in some cases.
I'm not arguing that it doesn't work\, at least as long as you are in Perl. If you try and manufacture a commandline which includes all of the @INC entries (for example during testing)\, you can run into commandline-length issues on various platforms that are very non-obvious to debug. I don't think that tripling or quadrupling the number of @INC entries\, on the offchance that there might be some code installed under one of the previous releases\, is the best way to deal with this particular problem.
During testing\, might it be possible to find some other way to populate @INC without using the command line? Perhaps you could make a file that you source with a bunch of 'use lib' statements? I don't know the problem in detail\, so I don't know if this is even a remotely relevant remark.
Perl could just as easily be taught to recursively probe ($]-0.000_001) if those paths exist\, rather than make those path's explicitly listed in @INC. Of course\, that would be modulo binary incompatibility boundaries (e.g. between 5.8.0 and 5.8.1?).
Provided there's a way to turn such recursion off and on\, and provided it handles versions prior to 5.6.0 (e.g. I have 5.00563 and 5.005 in @INC on this system right now)\, and provided we can put sensible limits on the recusrion\, I don't see any fundamental reason why the check couldn't be moved to runtime.
At the time the scheme was developed\, we were anticipating frequent
subversion snapshots. So if I have 5.00563 (which I actually do)\, I
really don't want perl to have failing stat()s in all of 5.00562\, 5.00561\,
5.00560\, ...\, 5.00501\, and 5.005\, none of which exist. Moving the check
to Configure time avoids this. Even today\, a user installing 5.8.8 would\,
under a naive implementation of your scheme\, automatically end up checking
all the way back to 5.8.0\, even if they only wanted 5.8.8 and 5.8.7.
(Wanting to go back one version is certainly a reasonable thing to
support.)
So teaching perl to recurse needs some careful thought\, but I don't see why it couldn't be done.
Of course\, that would be modulo binary incompatibility boundaries (e.g. between 5.8.0 and 5.8.1?). Of course the other difficulty is a distro-provided i586 path and a self-compiled i686 path are technically compatible (the 686 can read the 586 binaries)\, but I don't think there is any to automate that...
I not aware of a big demand for such 3/4/5/6 searches\, but that's rather a different issue anyway.
-- Andy Dougherty doughera@lafayette.edu
Andrew Dougherty wrote:
During testing\, might it be possible to find some other way to populate @INC without using the command line? Perhaps you could make a file that you source with a bunch of 'use lib' statements? I don't know the problem in detail\, so I don't know if this is even a remotely relevant remark.
Yes\, I think that has even been discussed. My memory of the last discussion on the M::B list was that none of the core developers was able to recreate the issue (not using those particular distros)\, but it seemed clear that the problem was the size of @INC.
At the time the scheme was developed\, we were anticipating frequent subversion snapshots. So if I have 5.00563 (which I actually do)\, I really don't want perl to have failing stat()s in all of 5.00562\, 5.00561\, 5.00560\, ...\, 5.00501\, and 5.005\, none of which exist. Moving the check to Configure time avoids this.
Configure-time doesn't help the poor Fedora user\, which gets created with the maximal set\, even if they are installing to a fresh system. I really don't want this to be a Configure option for that very reason!
Even today\, a user installing 5.8.8 would\, under a naive implementation of your scheme\, automatically end up checking all the way back to 5.8.0\, even if they only wanted 5.8.8 and 5.8.7.
(Wanting to go back one version is certainly a reasonable thing to
That would be a _very_ naive implementation. I was assuming that Perl would stat() the equivalent of ../$^V/ (and site_perl and vendor) and recurse one at a time back in history until finding the first match. You only need to check one directory (per subtree) to know how far back to recurse. This could be done today with a function hook in @INC...
hmmm... I wonder if I have time to flesh this out as a POC...
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham\, MD 20706 301-459-3366 x.5010 fax 301-429-5748
On Fri\, 23 Feb 2007\, John Peacock wrote:
Andrew Dougherty wrote:
At the time the scheme was developed\, we were anticipating frequent subversion snapshots. So if I have 5.00563 (which I actually do)\, I really don't want perl to have failing stat()s in all of 5.00562\, 5.00561\, 5.00560\, ...\, 5.00501\, and 5.005\, none of which exist.
Moving the check to Configure time avoids this.Configure-time doesn't help the poor Fedora user\, which gets created with the maximal set\, even if they are installing to a fresh system. I really don't want this to be a Configure option for that very reason!
Even today\, a user installing 5.8.8 would\, under a naive implementation of your scheme\, automatically end up checking all the way back to 5.8.0\, even if they only wanted 5.8.8 and 5.8.7. (Wanting to go back one version is certainly a reasonable thing to
That would be a _very_ naive implementation. I was assuming that Perl would stat() the equivalent of ../$^V/ (and site_perl and vendor) and recurse one at a time back in history until finding the first match. You only need to check one directory (per subtree) to know how far back to recurse. This could be done today with a function hook in @INC...
I guess I don't understand what you have in mind. Here's my @INC. Note there are numerical gaps. Without the explicit list\, how is perl going to know to NOT look in 5.8.7\, but DO look in 5.8.6\, 5.8.5\, etc. (And doing a stat() on 5.8.7 counts as "looking"!) How is it going to find 5.00563 and 5.005 but not bother looking at every possibility in between?
/opt/perl/lib/5.8.8/sun4-solaris /opt/perl/lib/5.8.8 /opt/perl/lib/site_perl/5.8.8/sun4-solaris /opt/perl/lib/site_perl/5.8.8 /opt/perl/lib/site_perl/5.8.6/sun4-solaris /opt/perl/lib/site_perl/5.8.6 /opt/perl/lib/site_perl/5.8.5/sun4-solaris /opt/perl/lib/site_perl/5.8.5 /opt/perl/lib/site_perl/5.8.4/sun4-solaris /opt/perl/lib/site_perl/5.8.4 /opt/perl/lib/site_perl/5.8.3/sun4-solaris /opt/perl/lib/site_perl/5.8.3 /opt/perl/lib/site_perl/5.8.2/sun4-solaris /opt/perl/lib/site_perl/5.8.2 /opt/perl/lib/site_perl/5.8.0/sun4-solaris /opt/perl/lib/site_perl/5.8.0 /opt/perl/lib/site_perl/5.6.1 /opt/perl/lib/site_perl/5.6.0 /opt/perl/lib/site_perl/5.00563 /opt/perl/lib/site_perl/5.005 /opt/perl/lib/site_perl
hmmm... I wonder if I have time to flesh this out as a POC...
That'd probably be useful if you want to pursue this. I'm probably missing something obvious.
In any case\, there are other reasons why some sites may have very large @INC lists\, so even somehow getting rid of this one case won't really help.
Back to the Fedora issue: perhaps Fedora could look into using
sh Configure -Dusesitecustomize
and use that in an rpm post-install step to populate @INC based on what the actual user actually has installed. Never having installed any rpm-based system myself\, I have no idea if that's easy or hard\, but it might be a useful direction for them to start looking in.
-- Andy Dougherty doughera@lafayette.edu
Andy Dougherty wrote:
I guess I don't understand what you have in mind. Here's my @INC. Note there are numerical gaps. Without the explicit list\, how is perl going to know to NOT look in 5.8.7\, but DO look in 5.8.6\, 5.8.5\, etc. (And doing a stat() on 5.8.7 counts as "looking"!) How is it going to find 5.00563 and 5.005 but not bother looking at every possibility in between?
In Perl (obviously the C equivalent would be used):
my @NEWINC = \</path/to/site_perl/*>; foreach my $INC ( reverse sort @NEWINC ) { # search for the module we haven't found yet }
As long as you are in the same PERL_VERSION\, you can consume both pure Perl and binary modules (except that we would have to exclude 5.8.0). Between PERL_VERSION releases\, you can only use pure Perl code. Because you know the path to the current /site_perl/ you can find every other installed version by looking at its parent.
In any case\, there are other reasons why some sites may have very large @INC lists\, so even somehow getting rid of this one case won't really help.
Every time that a new 5.8.x release comes out\, the potential length of @INC grows by two terms\, each of which is constant except for the last segment (which is a waste of space in terms of sum of all of the paths). I just think it would be better to have Perl more intelligently deal with it than loading on more @INC entries.
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Blvd Suite H Lanham\, MD 20706 301-459-3366 x.5010 fax 301-429-5747
On Sat\, 24 Feb 2007\, John Peacock wrote:
Andy Dougherty wrote:
I guess I don't understand what you have in mind. Here's my @INC. Note there are numerical gaps. Without the explicit list\, how is perl going to know to NOT look in 5.8.7\, but DO look in 5.8.6\, 5.8.5\, etc. (And doing a stat() on 5.8.7 counts as "looking"!) How is it going to find 5.00563 and 5.005 but not bother looking at every possibility in between?
In Perl (obviously the C equivalent would be used):
my @​NEWINC = \</path/to/site\_perl/\*>; foreach my $INC \( reverse sort @​NEWINC \) \{ \# search for the module we haven't found yet \}
Ah. I see. That's essentially what Configure does now. (It's rather different from the decrement-and-recurse strategy I thought you were proposing\, and hence my confusion.) Moving that Configure-time calculation to run-time does make more sense for situations where perl is Configured on one system and run on another.
However\, it'd probably be best to do this in a lazy sort of way -- some of the vendor* or site* directories might well be remotely mounted\, and spending time populating @INC even if not needed might not be appropriate.
Every time that a new 5.8.x release comes out\, the potential length of @INC grows by two terms\, each of which is constant except for the last segment (which is a waste of space in terms of sum of all of the paths).
I just think it would be better to have Perl more intelligently deal with it than loading on more @INC entries.
I don't see the "waste of space" as that big a deal. In cases where all those directories exist\, they are all going to show up in @INC anyway\, so this wouldn't save any space.
-- Andy Dougherty doughera@lafayette.edu
Andy Dougherty wrote:
However\, it'd probably be best to do this in a lazy sort of way -- some of the vendor* or site* directories might well be remotely mounted\, and spending time populating @INC even if not needed might not be appropriate.
That's why I was thinking of doing it as a coderef at the tail end of @INC. This way\, you only check the backwards compatible paths iff you didn't find the module requested already. This would increase the length of @INC by a constant value (1). I don't see any point in prepopulating @INC on the offchance that you might need it later.
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Blvd Suite H Lanham\, MD 20706 301-459-3366 x.5010 fax 301-429-5747
John Peacock \jpeacock@​rowman\.com\.devel\.redhat\.com writes:
Andy Dougherty wrote:
However\, it'd probably be best to do this in a lazy sort of way -- some of the vendor* or site* directories might well be remotely mounted\, and spending time populating @INC even if not needed might not be appropriate.
That's why I was thinking of doing it as a coderef at the tail end of @INC. This way\, you only check the backwards compatible paths iff you didn't find the module requested already. This would increase the length of @INC by a constant value (1). I don't see any point in prepopulating @INC on the offchance that you might need it later.
fwiw\, I've submitted RT #41657\, which includes a patch related to this issue as well - it adds a -R flag to *not* include paths this way. Both patches should probably be accepted or rejected as a unit.
Thanks\,
-RN
-- Robin Norwood Red Hat\, Inc.
"The Sage does nothing\, yet nothing remains undone." -Lao Tzu\, Te Tao Ching
On Thu\, 1 Mar 2007\, Robin Norwood wrote:
John Peacock \jpeacock@​rowman\.com\.devel\.redhat\.com writes:
Andy Dougherty wrote:
However\, it'd probably be best to do this in a lazy sort of way -- some of the vendor* or site* directories might well be remotely mounted\, and spending time populating @INC even if not needed might not be appropriate.
That's why I was thinking of doing it as a coderef at the tail end of @INC. This way\, you only check the backwards compatible paths iff you didn't find the module requested already. This would increase the length of @INC by a constant value (1). I don't see any point in prepopulating @INC on the offchance that you might need it later.
fwiw\, I've submitted RT #41657\, which includes a patch related to this issue as well - it adds a -R flag to *not* include paths this way. Both patches should probably be accepted or rejected as a unit.
Thanks. However\, I don't think #41586 is going into the main perl sources (it mostly duplicates the existing inc_version_list functionality and can probably be better done either with the existing inc_version_list functionality or with -Dusesitecustomize). Hence this patch to turn it off probably won't be needed either.
Still\, it's not my call\, and it's good to have this around. I've linked the RT entries together.
-- Andy Dougherty doughera@lafayette.edu
Given that there are other ways to specify additional directories in @INC at Configure time\, I won't apply this patch to bleadperl.
@rgs - Status changed from 'open' to 'rejected'
On Sat\, 24 Feb 2007 15:04:09 -0500\, John Peacock \jpeacock@​rowman\.com wrote:
In any case\, there are other reasons why some sites may have very large @INC lists\, so even somehow getting rid of this one case won't really help.
Every time that a new 5.8.x release comes out\, the potential length of @INC
*potentionally*. I always build my new perl with -Dinc_version_list=none to prevent that. Another good reason to do so\, is that once you've tested your new shiny version of perl\, and decide to make it the production one\, and maybe even package it up for your customers you do not want scripts to start failing when you remove the tree of the old(er) perl(s).
grows by two terms\, each of which is constant except for the last segment (which is a waste of space in terms of sum of all of the paths). I just think it would be better to have Perl more intelligently deal with it than loading on more @INC entries.
-- H.Merijn Brand Amsterdam Perl Mongers (http://amsterdam.pm.org/) using & porting perl 5.6.2\, 5.8.x\, 5.9.x on HP-UX 10.20\, 11.00\, 11.11\, & 11.23\, SuSE 10.0 & 10.2\, AIX 4.3 & 5.2\, and Cygwin. http://qa.perl.org http://mirrors.develooper.com/hpux/ http://www.test-smoke.org http://www.goldmark.org/jeff/stupid-disclaimers/
Migrated from rt.perl.org#41586 (status was 'rejected')
Searchable as RT41586$