Request: Allow a pp-ed executable to run in debug mode

[LouisStrous]

I have a script.pl that runs fine and finishes quickly. If I turn it into a Windows 10 executable script.exe using pp and then run script.exe then it seems to hang. I would like to be able to have pp produce an executable that runs the perl script in the perl debugger, so I can figure out conveniently where the difference comes from. I.e., I would like to be able to make the pp-ed executable act like perl -d script.pl instead of perl script.pl.

I've tried adding BEGIN { require 'perl5db.pl' } at the start of script.pl (inspiration taken from perldebguts) but that didn't have the desired effect. I've tried beginning my script with #!/bin/perl -d but then script.exe fails with a "mkdir" error about an "Invalid argument" involving AppData\Local\par-HASH\cache-HASH. I've tried hacking the PAR/pp source code by looking for the places where I think perl.exe gets started and then injecting "-d", but to no avail.

[rschupp]

This can't be done without invasive changes to the current source and perhaps not at all.

Even if it could be done there's a good chance it wouldn't help with your problem. Better check the usual suspects:

• did you use pp --gui ...?
• does your script use fork (either explicitly or via some parallelizing module)?
• does your script use unusual BEGINconstructs?

Try some old-fashioned print debugging.

[LouisStrous]

Thank you for responding. It is unfortunate that my request turns out to be too difficult. I'll have to debug my problem in a less convenient manner.

You do not need to respond to the following, unless you feel like it. I do not understand why you think that "there's a good chance it wouldn't help with [my] problem". That sounds like saying that the perl debugger is not useful. Perhaps you mean that it is unlikely that you could get the pp-ed script to act exactly as if it were called via "perl -d".

FYI: I did not use pp --gui. My script doesn't use fork or any BEGIN constructs, but perhaps some of the third-party modules do. I suspect that the problem has to do with the interaction with one of the third-party modules used by my script. Maybe some such module is not included or somehow configured differently in the pp-ed version, but I don't know which one or why. I may have to inject print statements into those third-party modules to find out.

Regards, Louis Strous

Roderich Schupp schreef op 2020-08-09 18:04:

This can't be done without invasive changes to the current source and perhaps not at all.

Even if it could be done there's a good chance it wouldn't help with your problem. Better check the usual suspects:

• did you use pp --gui ...?
• does your script use fork (either explicitly or via some parallelizing module)?
• does your script use unusual BEGINconstructs?

Try some old-fashioned print debugging.

-- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/rschupp/PAR-Packer/issues/24#issuecomment-671069654 [2] https://github.com/notifications/unsubscribe-auth/ABJRVQOKJIUCT5KA7Z52DPTR73CIVANCNFSM4PXTXCMA

[rschupp]

@LouisStrous

script.exe then it seems to hang

Add something like print "starting\n"as the first line of your script. If you don't see this message, then it's most likely that loading of some module failed (esp. non-pure Perl modules). In that case add the following as the first line which should tell you the failing module

BEGIN { unshift @INC, sub { print STDERR "loading $_[1]...\n"; return; } }

[rschupp]

Even if it could be done there's a good chance it wouldn't help with your problem

I did a quick hack (that might have broken other things) that will drop you into the Perl debugger. But at the debugger's initial prompt you are deep in the prologue that runs before finally executing the original script. You would have to either single step through lots of arcane code or know the magic incatation (breaking twice at PAR::_run_member plus a couple of ns until you reach do 'main';) to get to start of your script. The easiest solution is to stick a $DB::single = 1; at the start of your script, repack it and then simply c at the initial debugger prompt.

[LouisStrous]

Thank you for your efforts. A hack-ish method to achieve the goal would already be much better than no method. I had in the meantime bitten the bullet and located the culprit using print-style debugging. It was less work than I feared. I identified the third-party module that was missing from the pp-ed version, and added it in via "pp -M".

I tried your unshift @INC trick and it is very useful. It showed which module was the last one loaded before my pp-ed script hung, and by comparing the stderr output from script.exe and script.pl I could tell which was the first module loaded by script.pl after the point where script.exe hung. It turns out that that wasn't the exact module that I needed to add via "pp -M", but it was part of the same interlocking group of modules as the one that I needed to add. The comparison showed quite a few more modules that are loaded when I run script.pl but not when I run script.exe, but they (so far) don't seem to affect the proper operation of script.exe.

I wanted to try your "quick hack" but couldn't find the code.

[shawnlaffan]

Out of interest, which modules needed to be loaded using -M? (if they are on CPAN).

You could also try packing with the -x flag as static scanning can often miss runtime loaded modules.

[LouisStrous]

For my current problem I had to add Net::FTP via -M. In the past, for other scripts, I've had to add

* Template::Context
* Template::Plugins
* Template::Filters
* Template::Stash::XS

Packing with the -x flag is a bit problematic in this case because I've automated the packing (via a make.pl script that processes many scripts that might be updated and need (re)packing) and the script that gave me trouble requires an ftp server address and credentials to be passed in. The ftp server might not always be reachable, and credentials shouldn't be hardcoded. Nevertheless, I could have tried packing manually using -x as part of my analysis. Thank you for the suggestion. I'll use it when I run into a similar problem again.

Regards, Louis Strous

shawnlaffan schreef op 2020-08-11 09:52:

Out of interest, which modules needed to be loaded using -M? (if they are on CPAN).

You could also try packing with the -x flag as static scanning can often miss runtime loaded modules.

-- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/rschupp/PAR-Packer/issues/24#issuecomment-671789455 [2] https://github.com/notifications/unsubscribe-auth/ABJRVQPFRIFDRIM2E5WIX5TSAD2D3ANCNFSM4PXTXCMA

[rschupp]

@LouisStrous

For my current problem I had to add Net::FTP via -M. In the past, for other scripts, I've had to add

• Template::Context
• Template::Plugins
• Template::Filters
• Template::Stash::XS

Module::ScanDeps is used by PAR::Packer to figure out the modules used (directly or transitively) by your script. Which version of Module::ScanDeps do you have installed?

Can you show the exact statement with which your script loads Net::FTP?

All Template::** modules should automagically be included once use Template; has been seen by Module::ScanDeps or do you use something else to load the Template module?

[LouisStrous]

cpan -D Module::ScanDeps Loading internal logger. Log::Log4perl recommended for better logging CPAN: CPAN::SQLite loaded ok (v0.212) CPAN: LWP::UserAgent loaded ok (v6.36) Fetching with LWP: http://cpan.strawberryperl.com/authors/01mailrc.txt.gz CPAN: YAML::XS loaded ok (v0.75) Fetching with LWP: http://cpan.strawberryperl.com/modules/02packages.details.txt.gz Fetching with LWP: http://cpan.strawberryperl.com/modules/03modlist.data.gz Creating database file ... Done! Module::ScanDeps

(no description) R/RS/RSCHUPP/Module-ScanDeps-1.28.tar.gz C:\Strawberry\perl\site\lib\Module\ScanDeps.pm Installed: 1.27 CPAN: 1.28 Not up to date Roderich Schupp (RSCHUPP) rschupp@cpan.org

It looks like I don't have the latest version of Module::ScanDeps. I'll upgrade.

I load Net::FTP using

use Net::FTP;

and I load Template using

use Template;

Roderich Schupp schreef op 2020-08-11 11:04:

@LouisStrous [1]

For my current problem I had to add Net::FTP via -M. In the past, for other scripts, I've had to add

• Template::Context
• Template::Plugins
• Template::Filters
• Template::Stash::XS

Module::ScanDeps is used by PAR::Packer to figure out the modules used (directly or transitively) by your script. Which version of Module::ScanDeps do you have installed?

Can you show the exact statement with which your script loads Net::FTP?

All Template::** modules should automagically be included once use Template; has been seen by Module::ScanDeps or do you use something else to load the Template module?

-- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [2], or unsubscribe [3].

Links:

[1] https://github.com/LouisStrous [2] https://github.com/rschupp/PAR-Packer/issues/24#issuecomment-671825662 [3] https://github.com/notifications/unsubscribe-auth/ABJRVQLEZXGCSQI5VEZ2QQDSAECRJANCNFSM4PXTXCMA

[shawnlaffan]

Running scandeps -e "use Net::FTP" under strawberry perl 5.28 does not list any of the Net::FTP::* packages. Adding the -x flag makes no difference in this case.

Looking through the source code, there is this line that Module::ScanDeps will not catch:
https://metacpan.org/source/SHAY/libnet-3.11/lib/Net/FTP.pm#L1036

eval "require " . $pkg ## no critic (BuiltinFunctions::ProhibitStringyEval)
    or croak("cannot load $pkg required for type ".$ftp->type);

There is a preload rule for Net::FTP, but it might not be working as needed?

https://github.com/rschupp/Module-ScanDeps/blob/79ffe2af993a023d7d2fb6fe5b4f2dcd5ed52547/lib/Module/ScanDeps.pm#L422

[shawnlaffan]

Update: scandeps -B -e "use Net::FTP" lists the Net::FTP sub-modules.

I am using Module::ScanDeps 1.27.

[rschupp]

@shawnlaffan Yeah, scandeps -B should really be the default. Even the long name of the option, "--bundle", does only make sense in the context of PAR::Packer.

@LouisStrous I just checked that Module::ScanDeps does the right thing wrt Net::FTP and Template. Can you run scandeps -B script.pl | grep Net::FTP on the script.pl where you needed to add -M Net::FTP to the pp command line? Also, is the line use Net::FTP; in the main script.pl? If it's in another script, how is that loaded?

[LouisStrous]

scandeps -B script.pl | grep Net::FTP yields no output (after updating all modules using cpan-outdated). scandeps -B script.pl yields a long list of 'Module' => 'version' lines.

The main script is schematically like this:

use Modern::Perl; use FindBin; use lib "$FindBin::Bin/Modules"; use ModuleA; my $ftp = ModuleA->new('hostname');

Modules/ModuleA.pm is schematically like this:

package ModuleA; use Modern::Perl; use Net::Ftp; sub new { my ($class, $hostname) = @_; my $ftp = Net::FTP->new($hostname); return bless { backend => $ftp }, $class; } 1;

Regards, Louis Strous

Roderich Schupp schreef op 2020-08-12 09:38:

@shawnlaffan [1] Yeah, scandeps -B should really be the default. Even the long name of the option, "--bundle", does only make sense in the context of PAR::Packer.

@LouisStrous [2] I just checked that Module::ScanDeps does the right thing wrt Net::FTP and Template. Can you run scandeps -B script.pl | grep Net::FTP on the script.pl where you needed to add -M Net::FTP to the pp command line? Also, is the line use Net::FTP; in the main script.pl? If it's in another script, how is that loaded?

-- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [3], or unsubscribe [4].

Links:

[1] https://github.com/shawnlaffan [2] https://github.com/LouisStrous [3] https://github.com/rschupp/PAR-Packer/issues/24#issuecomment-672693456 [4] https://github.com/notifications/unsubscribe-auth/ABJRVQKCVAGRZIXHIGKTY43SAJBGXANCNFSM4PXTXCMA

[shawnlaffan]

I don't think pp has a chance of finding the directory referred to by this line under static scanning:

use lib "$FindBin::Bin/Modules";

If you add the path to the pp call using the -I flag then it should work. e.g.:

pp -I .\bin\Modules -o somename.exe script.pl

[rschupp]

Indeed, a construct like

use FindBin;
use lib "$FindBin::Bin/Modules";

can't be correctly interpreted (by Module::ScanDeps) with "static scanning", i.e. just by looking at the source code. Module::ScanDeps will see the following use ModuleA but isn't able to find ModuleA.pm. It doesn't complain, because there are lots of scenarios where this is alright (meaning: your script works nonetheless). use lib "$FindBin::Bin/Modules" works at runtime (or compile time, to be precise), that's what pp options -c and -x are for.

In your case pp -c ... would have done the trick. Or use Shawn's solution, as it would also work if you used require ModuleA instead of use ModuleA (because the former happens at runtime, while the latter at compile time).

[LouisStrous]

I did already use pp -c to pack my script, yet it didn't include Net::FTP.

But Shawn's solution (-I path_to_extra_modules_folder) works. Then I don't have to -M Net::Ftp anymore, so that should also take care of any other problematic third-party modules that are pulled in via the extra modules folder.

Thank you!

Louis Strous

Roderich Schupp schreef op 2020-08-13 12:37:

Indeed, a construct like

use FindBin; use lib "$FindBin::Bin/Modules";

can't be correctly interpreted (by Module::ScanDeps) with "static scanning", i.e. just by looking at the source code. Module::ScanDeps will see the following use ModuleA but isn't able to find ModuleA.pm. It doesn't complain, because there are lots of scenarios where this is alright (meaning: your script works nonetheless). use lib "$FindBin::Bin/Modules" works at runtime (or compile time, to be precise), that's what pp options -c and -x are for.

In your case pp -c ... would have done the trick. Or use Shawn's solution, as it would also work if you used require ModuleA instead of use ModuleA (because the former happens at runtime, while the latter at compile time).

-- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/rschupp/PAR-Packer/issues/24#issuecomment-673401627 [2] https://github.com/notifications/unsubscribe-auth/ABJRVQPFSUFV5DZUJ7P6Y53SAO65ZANCNFSM4PXTXCMA

[rschupp]

@LouisStrous FYI: I just released Module::ScanDeps 1.29 that should Do What I Mean for stuff like

use FindBin;
use lib "$FindBin::Bin/Modules";

without the need for any additional options to pp.

[LouisStrous]

It works for me without the need for any additional options to pp.

Thank you! Louis Strous

Roderich Schupp schreef op 2020-08-16 18:13:

@LouisStrous [1] FYI: I just released Module::ScanDeps 1.29 that should Do What I Mean for stuff like

use FindBin; use lib "$FindBin::Bin/Modules";

without the need for any additional options to pp.

-- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [2], or unsubscribe [3].

Links:

[1] https://github.com/LouisStrous [2] https://github.com/rschupp/PAR-Packer/issues/24#issuecomment-674545682 [3] https://github.com/notifications/unsubscribe-auth/ABJRVQOBWDPJAHD3O2RMZWDSBAAQ7ANCNFSM4PXTXCMA

[thoke]

Even if it could be done there's a good chance it wouldn't help with your problem

I did a quick hack (that might have broken other things) that will drop you into the Perl debugger. But at the debugger's initial prompt you are deep in the prologue that runs before finally executing the original script. You would have to either single step through lots of arcane code or know the magic incatation (breaking twice at PAR::_run_member plus a couple of ns until you reach do 'main';) to get to start of your script. The easiest solution is to stick a $DB::single = 1; at the start of your script, repack it and then simply c at the initial debugger prompt.

Can you share this quick hack? I'm fine with having to insert $DB::single = 1; at the beginning of my script and using c to get to it.

[thoke]

As an alternative to the "quick hack", is there a method to use parl or par.pl to invoke the debugger against a .par file?

[rschupp]

@thoke Sorry, I didn't bother to keep my quick hack and I see no way to get parl to invoke the debugger.

What is the problem you're trying to solve?

[thoke]

@rschupp From time to time I can't reproduce a problem I see with my par-packed exe vs running directly from the perl script. In those scenarios it'd be nice to run the exe (or the par) via the debugger.

After sleeping on my current issue last night, I may have found a solution to the issue I'm seeing. So this will probably sit again until I feel like I need it in the future.

FWIW, I feel like a number of years ago I was able to manipulate par.pl to invoke the debugger and actually be able to run my par file in the debugger... But I cannot remember how I did that, nor can I get it to work.