search

evalEmpire / perl5i

A single module to fix as much of Perl 5 as possible in one go
http://search.cpan.org/perldoc?perl5i
Other
156 stars 42 forks source link

UTF-8 and threads #271

Open Michael-Adams opened 11 years ago

Michael-Adams commented 11 years ago

I haven't had any real issues from doing a "force install perl5i" show up yet. But I notice that all the Windows smoke test results on CPAN Testers share the failures listed below.

Please note that Child-0.009 installed without a hitch, not sure why it's failing your test in t/Child.t... I've attached (at the end) output of "perl -V". I'm using Citrus Perl 5.16018 and the mingw64 package they provide.

Thank you and I have been (so far) enjoying perl5i!

cpan[15]> test perl5i Running test for module 'perl5i' Running make for M/MS/MSCHWERN/perl5i-v2.12.0.tar.gz Checksum for C:\Opt\bin\perl\cpan\sources\authors\id\M\MS\MSCHWERN\perl5i-v2.12.0.tar.gz ok

CPAN.pm: Building M/MS/MSCHWERN/perl5i-v2.12.0.tar.gz

Created MYMETA.yml and MYMETA.json Creating new 'Build' script for 'perl5i' version 'v2.12.0' Building perl5i

-->8-->8--snipped out some pod linking issue stuff-->8-->8---

MSCHWERN/perl5i-v2.12.0.tar.gz C:\Opt\bin\perl\bin\perl.exe ./Build -- OK Running Build test t/ARGV.t ..................... ok t/ARGV_twice.t ............... ok t/CLASS.t .................... ok t/Child.t .................... Dubious, test returned 5 (wstat 1280, 0x500) All 1 subtests passed t/English.t .................. ok t/File-stat.t ................ ok t/List-MoreUtils/all.t ....... ok t/List-MoreUtils/any.t ....... ok t/List-MoreUtils/false.t ..... ok t/List-MoreUtils/mesh.t ...... ok t/List-MoreUtils/minmax.t .... ok t/List-MoreUtils/none.t ...... ok t/List-MoreUtils/true.t ...... ok t/List-MoreUtils/uniq.t ...... ok t/List-Util/first.t .......... ok t/List-Util/max.t ............ ok t/List-Util/maxstr.t ......... ok t/List-Util/min.t ............ ok t/List-Util/minstr.t ......... ok t/List-Util/reduce.t ......... ok t/List-Util/shuffle.t ........ ok t/List-Util/sum.t ............ ok t/Meta/ISA.t ................. ok t/Meta/checksum.t ............ ok t/Meta/class.t ............... ok t/Meta/id.t .................. ok t/Meta/is-equal.t ............ ok t/Meta/linear_isa.t .......... ok t/Meta/methods.t ............. ok t/Meta/reftype.t ............. ok t/Meta/super.t ............... ok t/Meta/symbol_table.t ........ ok t/Want.t ..................... ok t/alias.t .................... ok t/as_hash.t .................. ok t/autobox.t .................. ok t/autodie.t .................. ok t/autovivification.t ......... ok t/caller.t ................... ok t/can.t ...................... ok t/capture.t .................. ok t/carp.t ..................... ok t/center.t ................... ok t/chdir.t .................... ok t/command_line_wrapper.t ..... 1/? "blib\script\perl5i.bat" unexpectedly returned exit value 255 at (eval 95) line 13. at t/command_line_wrapper.t line 25

Tests were run but no plan was declared and done_testing() was not seen.

t/command_line_wrapper.t ..... Dubious, test returned 255 (wstat 65280, 0xff00) All 2 subtests passed t/commify.t .................. ok t/datetime.t ................. ok t/die.t ...................... ok t/diff.t ..................... ok t/dump/array.t ............... ok t/dump/code.t ................ ok t/dump/formats.t ............. ok t/dump/hash.t ................ ok t/dump/obj.t ................. ok t/dump/scalar.t .............. ok t/each.t ..................... eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15. t/each.t ..................... 1/? eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15. eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15. eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15. eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15. eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15. eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15. eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15. t/each.t ..................... ok t/equal.t .................... ok t/everything_is_an_object.t .. ok t/flip.t ..................... ok t/foreach.t .................. ok t/github164.t ................ ok t/grep.t ..................... ok t/hash-diff.t ................ ok t/hash-intersect.t ........... ok t/hash-merge.t ............... ok t/intersect.t ................ ok t/io-handle.t ................ ok t/is_module_name.t ........... ok t/lexical.t .................. ok t/list-trim.t ................ ok t/list.t ..................... ok t/load_together.t ............ ok t/map.t ...................... ok t/method_leaking.t ........... ok t/modern_perl.t .............. ok t/module2path.t .............. ok t/no_indirect.t .............. ok t/number.t ................... ok t/perl5i.t ................... 1/? "C:\Opt\bin\perl\bin\perl.exe" unexpectedly returned exit value 255 at (eval 95) line 13. at t/perl5i.t line 14

Tests were run but no plan was declared and done_testing() was not seen.

t/perl5i.t ................... Dubious, test returned 255 (wstat 65280, 0xff00) All 3 subtests passed t/pick.t ..................... ok t/popn.t ..................... ok t/require.t .................. ok t/require_message.t .......... ok t/say.t ...................... ok t/scalar.t ................... ok t/shiftn.t ................... ok t/signature.t ................ ok t/signatures.t ............... ok t/skip.t ..................... ok t/taint.t .................... ok t/time_compat.t .............. ok t/true.t ..................... ok t/try-tiny.t ................. ok t/uniq.t ..................... ok t/utf8.t ..................... ok t/version_0/00_compile.t ..... skipped: Needs Time::y2038 t/version_1/00_compile.t ..... skipped: Needs Time::y2038 t/vs_listmoreutils.t ......... ok t/wrap.t ..................... ok t/y2038.t .................... ok

Test Summary Report

t/Child.t (Wstat: 1280 Tests: 1 Failed: 0) Non-zero exit status: 5 Parse errors: No plan found in TAP output t/command_line_wrapper.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: No plan found in TAP output t/perl5i.t (Wstat: 65280 Tests: 3 Failed: 0) Non-zero exit status: 255 Parse errors: No plan found in TAP output Files=100, Tests=1177, 59 wallclock secs ( 0.50 usr + 0.31 sys = 0.81 CPU) Result: FAIL Failed 3/100 test programs. 0/1177 subtests failed. MSCHWERN/perl5i-v2.12.0.tar.gz C:\Opt\bin\perl\bin\perl.exe ./Build test -- NOT OK //hint// to see the cpan-testers results for installing this module, try: reports MSCHWERN/perl5i-v2.12.0.tar.gz Failed during this command: MSCHWERN/perl5i-v2.12.0.tar.gz : make_test NO

Summary of my perl5 (revision 5 version 16 subversion 3) configuration:

Platform: osname=MSWin32, osvers=6.0, archname=MSWin32-x64-multi-thread uname='' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='gcc', ccflags =' -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_RELOCATABLE_INCPUSH -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -fno-strict-aliasing -mms-bitfields', optimize='-s -O2', cppflags='-DWIN32' ccversion='', gccversion='4.6.3', gccosandvers='' intsize=4, longsize=4, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='long long', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='g++', ldflags ='-s -L"C:\Opt\bin\perl\lib\CORE" -L"C:\Opt\bin\perl\mingw64\x86_64-w64-mingw32\lib"' libpth=C:\Opt\bin\perl\mingw64\x86_64-w64-mingw32\lib libs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32 perllibs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32 libc=, so=dll, useshrplib=true, libperl=libperl516.a gnulibc_version='' Dynamic Linking: dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-mdll -s -L"C:\Opt\bin\perl\lib\CORE" -L"C:\Opt\bin\perl\mingw64\x86_64-w64-mingw32\lib"'

Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES HAVE_INTERP_INTERN MULTIPLICITY PERLIO_LAYERS PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS PERL_MALLOC_WRAP PERL_PRESERVE_IVUV PERL_RELOCATABLE_INCPUSH PL_OP_SLAB_ALLOC USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF USE_SITECUSTOMIZE Built under MSWin32 Compiled at Apr 3 2013 00:54:02 @INC: C:/Opt/bin/perl/site/lib C:/Opt/bin/perl/vendor/lib C:/Opt/bin/perl/lib .

exodist commented 11 years ago

I will try to answer some of these as best I can:

On Sat, Jul 6, 2013 at 8:09 PM, Michael-Adams notifications@github.comwrote:

t/Child.t .................... Dubious, test returned 5 (wstat 1280, 0x500)

Fork on win32 is a farce. It is fragile and broken and occomplished using threads, not actual process forking. Last time this was brought up I believe it was traced to some utf8/unicode stuff in perl5i breaking forking which in turn broke child.pm, this is why child.pm works fine in win32, but not in perl5i-win32

t/command_line_wrapper.t ..... 1/? "blib\script\perl5i.bat" unexpectedly returned exit value 255 at (eval 95) line 13. at t/command_line_wrapper.t line 25

Tests were run but no plan was declared and done_testing() was not seen.

t/command_line_wrapper.t ..... Dubious, test returned 255 (wstat 65280, 0xff00) All 2 subtests passed

No idea, needs investigation

t/each.t ..................... eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15. t/each.t ..................... 1/? eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15. eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15. eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15. eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15. eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15. eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15. eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15.

These warnings are fixed as of yesterday or the day before in git, but not in any release. The warnings are caused by an API change in Hash::StoredIterator which perl5i uses under the hood. The newest revisions of perl5i use the latest Hash::StoredIterator, and do not use the deprecated functions.

t/perl5i.t ................... 1/? "C:\Opt\bin\perl\bin\perl.exe" unexpectedly returned exit value 255 at (eval 95) line 13. at t/perl5i.t line 14

Tests were run but no plan was declared and done_testing() was not seen.

t/perl5i.t ................... Dubious, test returned 255 (wstat 65280, 0xff00) All 3 subtests passed

No idea on these, needs investigation.

So of the issues 1 is known, but not fixed (utf8/fork issues) Not sure if it is fixable Another is known and fixed The others are not known (to me)

Michael-Adams commented 11 years ago

Ok, so...

On 2013-07-06 23:58, Chad Granum wrote:

I will try to answer some of these as best I can:

On Sat, Jul 6, 2013 at 8:09 PM, Michael-Adams notifications@github.comwrote:

t/Child.t .................... Dubious, test returned 5 (wstat 1280, 0x500)

Fork on win32 is a farce. It is fragile and broken and occomplished using threads, not actual process forking. Last time this was brought up I believe it was traced to some utf8/unicode stuff in perl5i breaking forking which in turn broke child.pm, this is why child.pm works fine in win32, but not in perl5i-win32

Understood, I keep tripping over the rather awful utf8 issues in perl myself. Not that /any/ programming language really does unicode well, especially considering that Windows unicode is all utf-16 internally.

t/command_line_wrapper.t ..... 1/? "blib\script\perl5i.bat" unexpectedly returned exit value 255 at (eval 95) line 13. at t/command_line_wrapper.t line 25

Tests were run but no plan was declared and done_testing() was not

seen. t/command_line_wrapper.t ..... Dubious, test returned 255 (wstat 65280, 0xff00) All 2 subtests passed

No idea, needs investigation

t/each.t .....................

These warnings are fixed as of yesterday or the day before in git, but not in any release. The warnings are caused by an API change in Hash::StoredIterator which perl5i uses under the hood. The newest revisions of perl5i use the latest Hash::StoredIterator, and do not use the deprecated functions.

Will await the new release, and they're just depreciation warnings, so I'm not personally to worried about them...

t/perl5i.t ................... 1/? "C:\Opt\bin\perl\bin\perl.exe" unexpectedly returned exit value 255 at (eval 95) line 13. at t/perl5i.t line 14

Tests were run but no plan was declared and done_testing() was not

seen. t/perl5i.t ................... Dubious, test returned 255 (wstat 65280, 0xff00) All 3 subtests passed

No idea on these, needs investigation.

So of the issues 1 is known, but not fixed (utf8/fork issues) Not sure if it is fixable Another is known and fixed The others are not known (to me)

— Reply to this email directly or view it on GitHub https://github.com/schwern/perl5i/issues/271#issuecomment-20565758.

Thanks! If I can help by testing or supplying any other info for any of these 3 outstanding issues, just let me know.

Michael Adams SQL/Perl/VBA/C??/Java

exodist commented 11 years ago

On Sat, Jul 6, 2013 at 10:22 PM, Michael Adams notifications@github.comwrote:

Thanks! If I can help by testing or supplying any other info for any of these 3 outstanding issues, just let me know.

I am not sure if perl5i has any windows dev's devoting any effort to it. If you wanted to to try and debug/fix these issues I am sure it would be appreciated by everyone involved.

Michael-Adams commented 11 years ago

On 2013-07-07 17:04, Chad Granum wrote:

On Sat, Jul 6, 2013 at 10:22 PM, Michael Adams notifications@github.comwrote:

Thanks! If I can help by testing or supplying any other info for any of these 3 outstanding issues, just let me know.

I am not sure if perl5i has any windows dev's devoting any effort to it. If you wanted to to try and debug/fix these issues I am sure it would be appreciated by everyone involved.

— Reply to this email directly or view it on GitHub https://github.com/schwern/perl5i/issues/271#issuecomment-20578487.

Well, I've started on the t/Child.t issue, and it /is/ related to utf8 processing:

use strict; use warnings; use utf8::all; use Child;

my $child = Child->new( sub { my $self = shift; $self->say( "Have self" ); $self->say( "parent: " . $self->pid ); my $in = $self->read(); $self->say( $in ); }, pipe => 1 );

my $proc = $child->start;

It dies in Child.pm on line 61: if ( my $pid = fork() ) {

If one comments out the "use utf8::all;" line, it goes to completion.

Not quite sure where to go from here... It's looking like maybe a p5p issue as utf8::all breaks fork on win32? Especially as I don't see where (in utf8::all source) it would have a bad interaction.

Michael Adams SQL/Perl/VBA/C??/Java

exodist commented 11 years ago

http://www.nntp.perl.org/group/perl.perl5.porters/2012/12/msg196821.html

On Sun, Jul 7, 2013 at 5:08 PM, Michael Adams notifications@github.comwrote:

On 2013-07-07 17:04, Chad Granum wrote:

On Sat, Jul 6, 2013 at 10:22 PM, Michael Adams notifications@github.comwrote:

Thanks! If I can help by testing or supplying any other info for any of these 3 outstanding issues, just let me know.

I am not sure if perl5i has any windows dev's devoting any effort to it. If you wanted to to try and debug/fix these issues I am sure it would be appreciated by everyone involved.

— Reply to this email directly or view it on GitHub https://github.com/schwern/perl5i/issues/271#issuecomment-20578487.

Well, I've started on the t/Child.t issue, and it /is/ related to utf8 processing:

|use strict; use warnings; use utf8::all; use Child;

my $child = Child->new( sub { my $self = shift; $self->say( "Have self" ); $self->say( "parent: " . $self->pid ); my $in = $self->read(); $self->say( $in ); }, pipe => 1 );

my $proc = $child->start;

|It dies in Child.pm on line 61: | if ( my $pid = fork() ) {

If one comments out the "use utf8::all;" line, it goes to completion. | Not quite sure where to go from here... It's looking like maybe a p5p issue as utf8::all breaks fork on win32? Especially as I don't see where (in utf8::all source) it would have a bad interaction.

Michael Adams SQL/Perl/VBA/C??/Java

— Reply to this email directly or view it on GitHubhttps://github.com/schwern/perl5i/issues/271#issuecomment-20580327 .

exodist commented 11 years ago

https://rt.perl.org/rt3//Public/Bug/Display.html?id=31923

On Sun, Jul 7, 2013 at 5:19 PM, Chad Granum exodist7@gmail.com wrote:

http://www.nntp.perl.org/group/perl.perl5.porters/2012/12/msg196821.html

On Sun, Jul 7, 2013 at 5:08 PM, Michael Adams notifications@github.comwrote:

On 2013-07-07 17:04, Chad Granum wrote:

On Sat, Jul 6, 2013 at 10:22 PM, Michael Adams notifications@github.comwrote:

Thanks! If I can help by testing or supplying any other info for any of these 3 outstanding issues, just let me know.

I am not sure if perl5i has any windows dev's devoting any effort to it. If you wanted to to try and debug/fix these issues I am sure it would be appreciated by everyone involved.

— Reply to this email directly or view it on GitHub https://github.com/schwern/perl5i/issues/271#issuecomment-20578487.

Well, I've started on the t/Child.t issue, and it /is/ related to utf8 processing:

|use strict; use warnings; use utf8::all; use Child;

my $child = Child->new( sub { my $self = shift; $self->say( "Have self" ); $self->say( "parent: " . $self->pid ); my $in = $self->read(); $self->say( $in ); }, pipe => 1 );

my $proc = $child->start;

|It dies in Child.pm on line 61: | if ( my $pid = fork() ) {

If one comments out the "use utf8::all;" line, it goes to completion. | Not quite sure where to go from here... It's looking like maybe a p5p issue as utf8::all breaks fork on win32? Especially as I don't see where (in utf8::all source) it would have a bad interaction.

Michael Adams SQL/Perl/VBA/C??/Java

— Reply to this email directly or view it on GitHubhttps://github.com/schwern/perl5i/issues/271#issuecomment-20580327 .

exodist commented 11 years ago

This really is not a perl5i bug. It is a bug with threads and utf8. It is not a problem on linux (usually) because people rarely use perl threads in linux. However on windows, forking is emulated using threads.

Perl5i has a choice between utf8-all and broken fork, or working fork w/o utf8.

However, If utf8all is scoped, it might be possible to have Child.pm actively turn it off before calling fork()....

On Sun, Jul 7, 2013 at 5:22 PM, Chad Granum exodist7@gmail.com wrote:

https://rt.perl.org/rt3//Public/Bug/Display.html?id=31923

On Sun, Jul 7, 2013 at 5:19 PM, Chad Granum exodist7@gmail.com wrote:

http://www.nntp.perl.org/group/perl.perl5.porters/2012/12/msg196821.html

On Sun, Jul 7, 2013 at 5:08 PM, Michael Adams notifications@github.comwrote:

On 2013-07-07 17:04, Chad Granum wrote:

On Sat, Jul 6, 2013 at 10:22 PM, Michael Adams notifications@github.comwrote:

Thanks! If I can help by testing or supplying any other info for any of these 3 outstanding issues, just let me know.

I am not sure if perl5i has any windows dev's devoting any effort to it. If you wanted to to try and debug/fix these issues I am sure it would be appreciated by everyone involved.

— Reply to this email directly or view it on GitHub https://github.com/schwern/perl5i/issues/271#issuecomment-20578487.

Well, I've started on the t/Child.t issue, and it /is/ related to utf8 processing:

|use strict; use warnings; use utf8::all; use Child;

my $child = Child->new( sub { my $self = shift; $self->say( "Have self" ); $self->say( "parent: " . $self->pid ); my $in = $self->read(); $self->say( $in ); }, pipe => 1 );

my $proc = $child->start;

|It dies in Child.pm on line 61: | if ( my $pid = fork() ) {

If one comments out the "use utf8::all;" line, it goes to completion. | Not quite sure where to go from here... It's looking like maybe a p5p issue as utf8::all breaks fork on win32? Especially as I don't see where (in utf8::all source) it would have a bad interaction.

Michael Adams SQL/Perl/VBA/C??/Java

— Reply to this email directly or view it on GitHubhttps://github.com/schwern/perl5i/issues/271#issuecomment-20580327 .

exodist commented 11 years ago

Looks like it is lexical, I am going to see if I can find a solution.

On Sun, Jul 7, 2013 at 5:26 PM, Chad Granum exodist7@gmail.com wrote:

This really is not a perl5i bug. It is a bug with threads and utf8. It is not a problem on linux (usually) because people rarely use perl threads in linux. However on windows, forking is emulated using threads.

Perl5i has a choice between utf8-all and broken fork, or working fork w/o utf8.

However, If utf8all is scoped, it might be possible to have Child.pm actively turn it off before calling fork()....

On Sun, Jul 7, 2013 at 5:22 PM, Chad Granum exodist7@gmail.com wrote:

https://rt.perl.org/rt3//Public/Bug/Display.html?id=31923

On Sun, Jul 7, 2013 at 5:19 PM, Chad Granum exodist7@gmail.com wrote:

http://www.nntp.perl.org/group/perl.perl5.porters/2012/12/msg196821.html

On Sun, Jul 7, 2013 at 5:08 PM, Michael Adams notifications@github.comwrote:

On 2013-07-07 17:04, Chad Granum wrote:

On Sat, Jul 6, 2013 at 10:22 PM, Michael Adams notifications@github.comwrote:

Thanks! If I can help by testing or supplying any other info for any of these 3 outstanding issues, just let me know.

I am not sure if perl5i has any windows dev's devoting any effort to it. If you wanted to to try and debug/fix these issues I am sure it would be appreciated by everyone involved.

— Reply to this email directly or view it on GitHub https://github.com/schwern/perl5i/issues/271#issuecomment-20578487.

Well, I've started on the t/Child.t issue, and it /is/ related to utf8 processing:

|use strict; use warnings; use utf8::all; use Child;

my $child = Child->new( sub { my $self = shift; $self->say( "Have self" ); $self->say( "parent: " . $self->pid ); my $in = $self->read(); $self->say( $in ); }, pipe => 1 );

my $proc = $child->start;

|It dies in Child.pm on line 61: | if ( my $pid = fork() ) {

If one comments out the "use utf8::all;" line, it goes to completion. | Not quite sure where to go from here... It's looking like maybe a p5p issue as utf8::all breaks fork on win32? Especially as I don't see where (in utf8::all source) it would have a bad interaction.

Michael Adams SQL/Perl/VBA/C??/Java

— Reply to this email directly or view it on GitHubhttps://github.com/schwern/perl5i/issues/271#issuecomment-20580327 .

Michael-Adams commented 11 years ago

On 2013-07-07 19:29, Chad Granum wrote:

Looks like it is lexical, I am going to see if I can find a solution.

On Sun, Jul 7, 2013 at 5:26 PM, Chad Granum exodist7@gmail.com wrote:

This really is not a perl5i bug. It is a bug with threads and utf8. It is not a problem on linux (usually) because people rarely use perl threads in linux. However on windows, forking is emulated using threads.

Perl5i has a choice between utf8-all and broken fork, or working fork w/o utf8.

However, If utf8all is scoped, it might be possible to have Child.pm actively turn it off before calling fork().... Just spent a fair amount of time researching it as well based on your first hint to me.

Traipsed through the Perl source code involved, took Ibuprofen...

If you can turn the utf8 off for Child.pm, it may be the only fix until p5p figures out how to make utf8 and threads play nice together. But since the bug is open and was registered against v5.8.3!, I'm not sure they /can/ fix it without a huge amount of work as that section of Perl source code is rather a mess -- It makes me think, just looking at it, that Perl's threads aren't quite "production" ready -- even after all this time.

There seems to be more than one possibly dodgy stack manipulation involved in emulating fork(), along with possibly more than one initialization sequencing issue of the catch-22 variety.

Later on this week I'll try digging into the other 2 items.

Michael Adams SQL/Perl/VBA/C??/Java

exodist commented 11 years ago

I was not able to find a way to lexically tun it off in Child. I was able to reproduce the failure, but not eliminate it :-(

I think perl5i should probably not include utf8 on windows. I say this because people can always load utf8 if they need it, but they cannot turn it off. So if anyone needs to use forking on windows and wants to use perl5i they cannot.

At the same time I am hesitant to suggest we remove utf8-all because if anyone uses forking on win32 they are already making a mistake.

We could also skip the child.pm tests in win32 under perl5i, but then people will not know it is broken and will report the issue again and again.

On Sun, Jul 7, 2013 at 6:14 PM, Michael Adams notifications@github.comwrote:

On 2013-07-07 19:29, Chad Granum wrote:

Looks like it is lexical, I am going to see if I can find a solution.

On Sun, Jul 7, 2013 at 5:26 PM, Chad Granum exodist7@gmail.com wrote:

This really is not a perl5i bug. It is a bug with threads and utf8. It is not a problem on linux (usually) because people rarely use perl threads in linux. However on windows, forking is emulated using threads.

Perl5i has a choice between utf8-all and broken fork, or working fork w/o utf8.

However, If utf8all is scoped, it might be possible to have Child.pm actively turn it off before calling fork().... Just spent a fair amount of time researching it as well based on your first hint to me.

Traipsed through the Perl source code involved, took Ibuprofen...

If you can turn the fut8 off for Child.pm, it may be the only fix until p5p figures out how to make fut8 and threads play nice together. But since the bug is open and was registered against v5.83!, I'm not sure they /can/ fix it without a huge amount of work as that section of Perl source code is rather a mess -- It makes me think, just looking at it, that Perl's threads aren't quite "production" ready -- even after all this time. There seems to be more than one possibly dodgy stack manipulation involved in emulating fork(), along with possibly more than one initialization sequencing issue of the catch-22 variety.

Later on this week I'll try digging into the other 2 items.

Michael Adams SQL/Perl/VBA/C??/Java

— Reply to this email directly or view it on GitHubhttps://github.com/schwern/perl5i/issues/271#issuecomment-20581442 .

Michael-Adams commented 11 years ago

So the possibilities are a "$^O" test in perl5i startup code that :

exodist commented 11 years ago

Actually. Maybe a better solution in perl5i would be to override CORE::GLOBAL::Fork to throw an exception explaining the situation and that perl5i breaks fork on windows. This will catch uses by Child.pm (though Child() can also be overriden). The Child tests for perl5i can skip in win32.

On Sun, Jul 7, 2013 at 7:01 PM, Michael Adams notifications@github.comwrote:

So the possibilities are a "$^O" test in perl5i startup code that :

-

Disables utf8 on windows with a strong note in a "CAVEATS" section of the POD (not "BUGS" as it's not a perl5i bug) noting that utf8 is not turned on by perl5i due to this issue and also mentioning that fork() is not really recommended on windows.

Or keeps utf8 on windows but not Child (as fork() is contraindicated on win32)... with a similar strong note, unless it's just fork() that's borked by utf8, not threads per se... (need to test that...) and Child on windows can be reworked to do its own explicit fork() emulation via threads (most probably way too much work for too little gain...).

— Reply to this email directly or view it on GitHubhttps://github.com/schwern/perl5i/issues/271#issuecomment-20582343 .

Michael-Adams commented 11 years ago

I think I'd +1 the CORE::GLOBAL::Fork solution for win32 installs of perl5i. And of course I must'a been all goofy from reading "sv.c"; threads and utf8 are contraindicated on seemingly all systems...

exodist commented 11 years ago

I would prefer to have Schwern and others weigh in on this, I don't think it is a decision you and I should make :-)

On Sun, Jul 7, 2013 at 7:29 PM, Michael Adams notifications@github.comwrote:

I think I'd +1 the CORE::GLOBAL::Fork solution for win32 installs of perl5i. And of course I must'a been all goofy from reading "sv.c"; threads and utf8 are contraindicated on seemingly all systems...

— Reply to this email directly or view it on GitHubhttps://github.com/schwern/perl5i/issues/271#issuecomment-20582868 .

Michael-Adams commented 11 years ago

On 2013-07-07 22:13, Chad Granum wrote:

I would prefer to have Schwern and others weigh in on this, I don't think it is a decision you and I should make :-)

On Sun, Jul 7, 2013 at 7:29 PM, Michael Adams notifications@github.comwrote:

I think I'd +1 the CORE::GLOBAL::Fork solution for win32 installs of perl5i. And of course I must'a been all goofy from reading "sv.c"; threads and utf8 are contraindicated on seemingly all systems...

— Reply to this email directly or view it on GitHubhttps://github.com/schwern/perl5i/issues/271#issuecomment-20582868 .

— Reply to this email directly or view it on GitHub https://github.com/schwern/perl5i/issues/271#issuecomment-20583770.

True, thus the /"I t//hink I'd +1"/part:) and anyway Schwern being more experienced than, well me definitely, may have an even better solution. Good night and have a good day!

Michael Adams SQL/Perl/VBA/C??/Java

schwern commented 8 years ago

I do not relish having some features that only work on some platforms.

I chose Child for perl5i not just because it provides a nice interface to fork, but because it provides a nice abstraction for interprocess work. Looking at the Child interface, I wonder if it can be rewritten on Windows to use threads (which hopefully are less buggy when not trying to be fork) or native Windows IPC.

exodist commented 8 years ago

Making Child use threads on windows would be sane, and would solve many problems. That said it will not solve this problem :-(

https://rt.perl.org/Public/Bug/Display.html?id=31923 is the rt ticket that covers the problem in perl itself. Basically "PerlIO::encoding isn't thread safe."

There is a workaround I use in Test2: https://metacpan.org/pod/Test2::Plugin::UTF8#NOTES

Which uses binmode(HANDLE, ':utf8'); This is not as good as :encoding(utf-8) which is more strict, but on the other hand it actually works, which I consider a bonus.

Note that linux also has this problem, the reason we only notice on windows is because fork on windows is actually threads under the hood. If you used perl5i with threads you would likely encounter the same issue on linux,

exodist commented 8 years ago

Hmm, another option would be to require a newer version of perl, it appears the issue was fixed in blead last October, but I think that means depending on 5.24.0+.

schwern commented 8 years ago

If threads + :encoding(utf8) is as broken as that perlbug thread says, I think I'll have perl5i use just :utf8.

exodist commented 8 years ago

@schwern I agree that is probably the most sane option.

schwern commented 8 years ago

https://github.com/doherty/utf8-all/pull/43 should fix this problem. Once it's in utf8::all we'll depend on the new version.

schwern commented 8 years ago

Damn, that didn't fix it. But it got us further along. https://ci.appveyor.com/project/schwern/perl5i/build/7

Free to wrong pool 32180a0 not 859b10 at C:/strawberry/perl/site/lib/Child.pm line 86.
t/Child.t .................... 
Dubious, test returned 5 (wstat 1280, 0x500)
All 5 subtests passed

I've seen that "free to wrong pool" before when trying to free memory using Perl's free() that was allocated using the system's malloc(). http://www.perlmonks.org/?node_id=717224