RedHat Perl cannot find dependencies

[hatt]

Initially reported in #515 but unrelated to the actual bug in that issue, RedHat's packaging of Perl module directories is incorrect and currently breaks FPM and cpanm --self-contained installs that require 3rd party core modules in RHEL and CentOS 7.

The RedHat packaging for perl separates directories for core, 3rd party, and CPAN installed modules. Some parts of core are developed and maintained by 3rd party developers, and as such get installed into vendorlib. The directories used by cpanm for finding core modules are handled by only::core::lib, and evaluate to privlib and archlib specifically. As an example, ExtUtils::MakeMaker is installed to privlib/vendor_perl, which is not read as a core directory. A separate circular dependency between EU::MM and Data::Dumper prevents installing this into the sandbox, so we rely on core for the modules which it ends up not being able to find.

I believe @satoshi is filing a bug with RedHat concerning this. Essentially they need to update their list of core modules and install the 3rd party core modules as if they were internal core. Until then, cpanm is unable to find modules it needs to function normally and due to the circular dependency of EU::MM and DD, there is no way to run in a sandbox entirely. While inconvenient, I do not think this is a bug on EU::MM and DD's part as modules can be installed manually in this case.

[hatt]

Does anyone know how to handle reporting this? I've only personally confirmed it in CentOS, however the CentOS package code comes from RHEL upstream and Fedora is possibly impacted as well. Would it be better to report to CentOS bug tracker and let them handle upstream comms, or go direct to RedHat even without proof it's a problem there too? I'm unfamiliar with non-Archlinux distro bug reporting.

[satoshi]

That's a good point I haven't thought about. As far as I know, CentOS 6 / 7 / Fedora 20 are impacted. Luckily I have RHEL7 running on Amazon AWS, so let me make sure this issue happens there (I'm 99% sure it does).

[satoshi]

I wonder if the other possibility might be to modify cpanm (or perhaps local::lib?) to have them look at wider range of directories, not just privlib and archlib? If we could do that, then we don't have to nag Red Hat for a major major change.

[hatt]

Possibly. We currently instruct cpanm to be self-contained, so all non-core modules are installed whether we need them or not. This allows us to fatpack, iterate dependencies, and make sure the latest versions of build deps exist for the build run.

Perhaps we can make this optional though, with a flag such as --cpan-core-only to sandbox or to otherwise use all available modules. When I attempted the Parallel::Fork build like this, it still failed elsewhere however. Maybe @jordansissel or @dnbert know why we specifically sandbox but I can think of a few reasons why it would be desirable.

[satoshi]

Just to follow up, RHEL7 suffers the same issue. It was interesting that ruby-devel wasn't found in the yum repo, so I yanked one from CentOS7 and stuck it in there.

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.0 (Maipo)

build.log:
Checking if you have ExtUtils::MakeMaker 6.31 ... No
Checking if you have ExtUtils::Install 1.46 ... Yes (1.58)
==> Found dependencies: ExtUtils::MakeMaker
Searching ExtUtils::MakeMaker on cpanmetadb ...
--> Working on ExtUtils::MakeMaker

$ cpanm --version
cpanm (App::cpanminus) version 1.6922 (/usr/bin/cpanm)
perl version 5.016003 (/usr/bin/perl)

  %Config:
    archname=x86_64-linux-thread-multi
    installsitelib=/usr/local/share/perl5
    installsitebin=/usr/local/bin
    installman1dir=/usr/share/man/man1
    installman3dir=/usr/share/man/man3
    sitelibexp=/usr/local/share/perl5
    archlibexp=/usr/lib64/perl5
    privlibexp=/usr/share/perl5
  %ENV:
  @INC:
    /usr/local/lib64/perl5
    /usr/local/share/perl5
    /usr/lib64/perl5/vendor_perl
    /usr/share/perl5/vendor_perl
    /usr/lib64/perl5
    /usr/share/perl5
    .

[hatt]

Thanks for that. I ran some tests and it seems that if we explicitly disable sandboxing and the system has perl-local-lib installed, we can still get a successful build. I'm thinking that while it's not ideal, we can add a flag to select whether to sandbox or not, and include a node that local::lib should be installed by the user if not doing so. Still interested to know what the original reason for it was though. Will throw a PR later on this week.

[hatt]

Issue tracking with CentOS team here, if necessary I expect they will push it through to RHEL and Fedora as well.

[miked63017]

Personally I kind of like the idea of being able to disable the sandboxing with a flag. I find myself manually adding perl (watever) provides flags, and dependency flags, pretty often anyway. I build alot of perl modules with fpm and never really understood the reason for the sandbox considering there is no option to automatically package them. Just an opinion though, smarter people than me may disagree :) On Aug 6, 2014 1:36 AM, "Matt Sharpe" notifications@github.com wrote:

Thanks for that. I ran some tests and it seems that if we explicitly disable sandboxing and the system has perl-local-lib installed, we can still get a successful build. I'm thinking that while it's not ideal, we can add a flag to select whether to sandbox or not, and include a node that local::lib should be installed by the user if not doing so. Still interested to know what the original reason for it was though. Will throw a PR later on this week.

— Reply to this email directly or view it on GitHub https://github.com/jordansissel/fpm/issues/748#issuecomment-51298816.

[hatt]

Thanks for the feedback @miked63017, as I said I have no idea why it's there, so making it optional seems like a good idea. This is still a bug upstream as there are times where you will want this (packaging an application with all sandboxed dependencies f.e.), but it will still be good to have a flag for it and I'll be submitting a PR soon. Incidentally, this also breaks Carton and cpanm in general.

[satoshi]

@hatt Thank you very much for your incredible work on this issue. I'm an experienced Perl developer, however my exposure to core libraries only goes as far as @INC level. It was interesting to learn different classification of library paths and how they can cause subtle issues like this.

[hatt]

Bug submitted upstream with RedHat at https://bugzilla.redhat.com/show_bug.cgi?id=1127585, if anyone wants to check if this is also an issue in CentOS/RHEL 5, 6, and 6.5 that'd be cool too. I expect they'd backport from 7 to the others anyway, but it's a lot of work so who knows.

@satoshi no problem, I use Carton and cpanm kind of a lot too so I'm a little more aware of weird internal things, even if my usual perl isn't crash hot :)

[satoshi]

@hatt this issue happens on 6.5 and Fedora20 for sure. I can't speak of earlier version of CentOS 6 but I am quite certain it happens as well, since system Perl installation would stay the same.

Thanks for filing the bug with Red Hat. I see that they're indicating this as a cpanm bug now :-)

[hatt]

Well, their points are valid I guess, but I disagree with their entire Perl packaging scheme. The claim that other distributions do it the same way is really limited to Gentoo as well, since it's a non-issue in all Debian-based distros, Arch, Slackware, etc.

Personal thoughts aside, it isn't so much a bug as a maintainer preference. I'm not sure how cpanm would go about fixing it since they override the @INC path which I don't think would let them load individual modules outside that path, but perhaps @miyagawa has some ideas. I'm happy to play with the cpanm code to try and get a solution in, but I don't know if it's actually possible in the first place.

For now, PR #752 should work around the issue once merged although if you do need packaged dependencies then you're out of luck unless everything uses Module::Builder instead of ExtUtils::MakeMaker. @jordansissel any input on your end on how we could maybe work around this inside FPM itself?

[miked63017]

I always thought it was kind of a pain getting the cpan source plugin setup, and wondered why cpanm wasnt bundled. Couldn't we include a bundled cpanm and just use lib 'whatever/dir/we/want'? Which would also nullify the need for local::lib if we go as far as modifying @INC at runtime. Personally before I found this beautiful piece of software, fpm, I did similar when making my own spec files for rpms. On Aug 7, 2014 7:50 PM, "Matt Sharpe" notifications@github.com wrote:

Well, their points are valid I guess, but I disagree with their entire Perl packaging scheme. The claim that other distributions do it the same way is really limited to Gentoo as well, since it's a non-issue in all Debian-based distros, Arch, Slackware, etc.

Personal thoughts aside, it isn't so much a bug as a maintainer preference. I'm not sure how cpanm would go about fixing it since they override the @INC path which I don't think would let them load individual modules outside that path, but perhaps @miyagawa https://github.com/miyagawa has some ideas. I'm happy to play with the cpanm code to try and get a solution in, but I don't know if it's actually possible in the first place.

For now, PR #752 https://github.com/jordansissel/fpm/pull/752 should work around the issue once merged although if you do need packaged dependencies then you're out of luck unless everything uses Module::Builder instead of ExtUtils::MakeMaker. @jordansissel https://github.com/jordansissel any input on your end on how we could maybe work around this inside FPM itself?

— Reply to this email directly or view it on GitHub https://github.com/jordansissel/fpm/issues/748#issuecomment-51551643.

[satoshi]

@hatt awesome, for my use case it is perfectly fine to simply package with system dependencies. I also like that you caught the subtle bug on the cpan_test flag :-)

[hatt]

Thanks for the input @miked63017. As far as I'm aware, we can add paths to the lib @INC but we can't add individual modules. Since not all of vendor_perl is core, we don't want to just blanket add it. Maybe we could do some nasty hack with a temporary core modules directory and symlink modules directly but that seems kinda gross. What do you mean by bundle cpanm?

@satoshi glad that helped, noticed it while working on the line above haha

[miked63017]

I was thinking include cpanm with fpm. Symlinks do sound kind of hacky, but since modules are downloaded into our sandbox anyway what would be different if we used findbin and just copied them into the sandbox. And by sandbox, I mean one we would setup at runtime/compile time in our includes cpanm by modifying @INC and skipping local::lib. Please do point out any flaws, I am shooting from the hip but so far it feels doable. I might be able to spend some time hacking on it this weekend, since I am on call and not doing anything anyway, depending on the wife's consumption of my time. On Aug 8, 2014 3:26 AM, "Matt Sharpe" notifications@github.com wrote:

Thanks for the input @miked63017 https://github.com/miked63017. As far as I'm aware, we can add paths to the lib @INC but we can't add individual modules. Since not all of vendor_perl is core, we don't want to just blanket add it. Maybe we could do some nasty hack with a temporary core modules directory and symlink modules directly but that seems kinda gross. What do you mean by bundle cpanm?

@satoshi https://github.com/satoshi glad that helped, noticed it while working on the line above haha

— Reply to this email directly or view it on GitHub https://github.com/jordansissel/fpm/issues/748#issuecomment-51575611.

[hatt]

It feels weird to package a Perl app in a Ruby gem. If it was pure git clone code for most users I could understand it but the reality is people install it as a gem. Some modules are quite large too so it would be weird to copy them directly, but I think symlinking while nasty could work. Using local::lib isn't really a problem if we pass it the right module directory to use. I'm not sure how we'd modify @INC from Ruby either, seeing as iirc the Perl cli options for lib paths are added, not replaced. I'm open to reviewing code snippets/ideas since I really can't think of a clean solution. @jordansissel, do you have any opinions on this?

[miked63017]

I could be wrong, but I think you can pop and push onto @INC but not sure if its at compile time or runtime. I would agree, it would feel weird packaging a perl app in a gem. On Aug 10, 2014 8:26 PM, "Matt Sharpe" notifications@github.com wrote:

It feels weird to package a Perl app in a Ruby gem. If it was pure git clone code for most users I could understand it but the reality is people install it as a gem. Some modules are quite large too so it would be weird to copy them directly, but I think symlinking while nasty could work. Using local::lib isn't really a problem if we pass it the right module directory to use. I'm not sure how we'd modify @INC from Ruby either, seeing as iirc the Perl cli options for lib paths are added, not replaced. I'm open to reviewing code snippets/ideas since I really can't think of a clean solution. @jordansissel https://github.com/jordansissel, do you have any opinions on this?

— Reply to this email directly or view it on GitHub https://github.com/jordansissel/fpm/issues/748#issuecomment-51734327.

[hatt]

We can add/remove things from @INC but we can't change the privlib and archlib paths as they're compiled into the Perl. If we want to fatpack, we need to install all non-core modules into a specific directory. What we'd need to do then would be to link all core modules to one path, have an empty directory for the other dependencies, and add both directories to @INC I think. Then we can install all non-core deps to the secondary path. That SHOULD work.

[satoshi]

Not sure if it helps, but you could also export PERL5LIB at runtime if modifying @INC seems difficult. Iirc tweaking @INC is only possible from Perl scripts whereas PERL5LIB is an environment variable in shell.

[satoshi]

@jordansissel any chance PR #752 would be merged into master?

[satoshi]

I believe this issue is fixed after #752 has been merged and fpm-1.4.0 is released. Thanks for this excellent patch and close out this issue whoever that can do so.