Using ExtUtils::Installed may be very slow

[eserte]

Alien::Base::ModuleBuild::alien_refresh_packlist() is using ExtUtils::Installed to fetch the current module's packlist. Unfortunately this may be very slow, especially for large perl installations. The slowness happens already in ExtUtils::Installed->new, which is scanning the whole perl directory for .packlist files, reads them all (no filtering possible), and also parsing the version numbers of all modules.

On my systems, this may take between 15s (for smaller perl installations, and only if everything's already in the system cache), 300s (also a small perl installation, but uncached), and half an hour (large perl installation, nothing in the system cache).

It would be more efficient if the .packlist file would be search like this:

my $mod = "Term::ReadKey"; $mod =~ s{::}{/}g; for (@INC) { my $candidate = "$_/auto/$mod/.packlist"; return $candidate if -r $candidate}'

and the resulting filename provided to ExtUtils::Packlist->new, without using ExtUtils::Installed at all.

[jberger]

I have not seen such a delay.

Further it seems safer to trust the core module to do it's job rather than roll my own, if for no other reason than if things should change I would expect the core module will be updated while our code would not. On May 5, 2014 3:01 PM, "Slaven Rezić" notifications@github.com wrote:

Alien::Base::ModuleBuild::alien_refresh_packlist() is using ExtUtils::Installed to fetch the current module's packlist. Unfortunately this may be very slow, especially for large perl installations. The slowness happens already in ExtUtils::Installed->new, which is scanning the whole perl directory for .packlist files, reads them all (no filtering possible), and also parsing the version numbers of all modules.

On my systems, this may take between 15s (for smaller perl installations, and only if everything's already in the system cache), 300s (also a small perl installation, but uncached), and half an hour (large perl installation, nothing in the system cache).

It would be more efficient if the .packlist file would be search like this:

my $mod = "Term::ReadKey"; $mod =~ s{::}{/}g; for (@INC) { my $candidate = "$_/auto/$mod/.packlist"; return $candidate if -r $candidate}'

and the resulting filename provided to ExtUtils::Packlist->new, without using ExtUtils::Installed at all.

— Reply to this email directly or view it on GitHubhttps://github.com/Perl5-Alien/Alien-Base/issues/38 .

[eserte]

Timings vary depending on system, IO load, size of the site_perl directory, if filesystem cache is empty, filesystem type (local or nfs)...

Here's one of the worse systems, a Debian Linux running within a xen instance, with a quite large site_perl directory (almost 2.5GB). Run time is about 13 minutes:

$ for i in ~/var/ctps/51911linux/install/perl-5.1*/bin/perl; do echo -n "$i: "; time $i -MExtUtils::Installed -e 'ExtUtils::Installed->new'; du -s `$i -MConfig -e 'print $Config{sitelib}'`; done
/home/eserte/var/ctps/51911linux/install/perl-5.18.2/bin/perl: $i -MExtUtils::Installed -e 'ExtUtils::Installed->new'  7.87s user 4.54s system 1% cpu 13:10.29 total
2461624 /home/eserte/var/ctps/51911linux/install/perl-5.18.2/lib/site_perl/5.18.2
/home/eserte/var/ctps/51911linux/install/perl-5.19.11/bin/perl: $i -MExtUtils::Installed -e 'ExtUtils::Installed->new'  8.83s user 4.47s system 1% cpu 13:31.11 total
2358316 /home/eserte/var/ctps/51911linux/install/perl-5.19.11/lib/site_perl/5.19.11

Here's an faster experience. Relatively small site_perl directory, a physical system, and I ran the ExtUtils::Installed command before, so the filesystem cache was filled. Run time less than one second:

/usr/local/bin/perl5.18.2: $i -MExtUtils::Installed -e 'ExtUtils::Installed->new'  0,56s user 0,13s system 99% cpu 0,695 total
155528  /usr/perl5.18.2/lib/site_perl/5.18.2

Actually, the slowness is an issue for me, as it significantly slows down my smoker box, so I consider to skip all Alien-Base-based distributions...

[mohawk2]

1. Why are you comparing a large case without a cache, with a small case with a cache? Why not show how long it would take WITH the cache?
2. Why not work with the author of EU::Installed to improve that, rather than try to not use it AND to reimplement it? Remember: "The First Rule of Program Optimization: Don't do it. The Second Rule of Program Optimization (for experts only!): Don't do it yet." — Michael A. Jackson (And use a benchmark)

[eserte]

mohawk2 notifications@github.com writes:

1. Why are you comparing a large case without a cache, with a small case with a cache? Why not show how long it would take WITH the cache?

This was just an illustration of the both extremes. And I doubt that the filesystem cache is large enough to accomodate all of that large perl directory.

1. Why not work with the author of EU::Installed to improve that, rather than try to not use it AND to reimplement it? Remember: "The First Rule of Program Optimization: Don't do it. The Second Rule of Program Optimization (for experts only!): Don't do it yet." ― Michael A. Jackson (And use a benchmark)

The difference between 13 minutes and a few milliseconds is not covered by this saying.

Slaven Rezic - slaven rezic de BBBike - route planner for cyclists in Berlin WWW version: http://www.bbbike.de Perl/Tk version for Unix and Windows: http://bbbike.sourceforge.net

[jberger]

I think I agree with closing this issue at this point, but I do not agree with @mohawk2 's tone. If slowdowns are seen this can be reopened.

[plicease]

I'd like to see something done to alleviate the burden on cpan testers I am not sure if the solution is here or elsewhere or maybe a little of both.

[jberger]

@plicease most cpantesters (the automated ones) don't have large perl installations. They have a very clean perl and append all required and built modules to an ever growing PERL5LIB. This actually was a huge headache for writing AB, but it means that we won't be stressing the testers over this point at least.

[plicease]

@eserte

Alien::Base::ModuleBuild now respects the create_packlist=0, skipping ExtUtils::Installed completely if it is turned off:

./Build install create_packlist=0

This was added when I was working on DESTDIR support. If this does come up as an issue in the future then you can either use this option, or if it is difficult to wrangle the smoker to use that, we can add an environment variable to assume create_packlist=0 for alien_refresh_packlist.

edit for clarity