perlfunc/perlpacktut pack w documentation

[p5pRT]

Migrated from rt.perl.org#117661 (status was 'open')

Searchable as RT117661$

[p5pRT]

From schmorp@schmorp.de

Created by schmorp@schmorp.de

perldoc -f pack tells us​:

  w A BER compressed integer (not an ASN.1 BER\, see perlpacktut for details).

Ok\, it's not an ASN.1 BER integer\, still\, it's an ASN.1 BER subidentifier (and is happily used to decode these in e.g. Net​::SNMP).

Anyways\, heading to perlpacktut to see why there is strong negation of it's ASN.1 BER qualities\, I find... nothing.

The only thing I can find is "Another Portable Binary Encoding"\, which tells us​:

  (Details can be found at \<http​://Casbah.org/>\, the Scarab project.)

Apparently\, that domain fell off the internet in 2008. Looking at it in the wayback machine still shows that there are no details at that URL. At all. If there ever was\, it must have been buried deep inside that domain at some other URL. It looks more like some shameless plug for some project\, and in any case\, perl now refers to some kind of recycling shop or so.

Furthermore​:

  There is no size limit to BER encoding\, but Perl won't go to extremes.

Well\, last I looked\, perl went very well to extremes when decoding these gems\, as it implements it's own multi-precision multiplication function so it can decode BER integers of arbitrary lentgh into decimal string form. Of course\, whether that's going to extremes is another question\, but to me\, that's quite surprising\, and\, if officially supported\, quite a nice touch.

Here are my suggestions to improve this situation​:

1. drop the "not an ASN.1 BER" comment\, or at least qualify it\, like   "like an ASN.1 BER subidentifier\, not a BER integer".

  At the very least the description of BER subidentifier encoding in   X.209​:198811 6.22 is much more concise than any I found in the perl   documentation\, so saying it isn't an ASN.1 BER is more confusing than   helpful\, because by all means it is.

2. remove the url in perlpacktut.

3. document the fact that perl will go to multiprecision arithmetic when   decoding these values (but not when encoding them)\, and that treating   these decoded values as numbers in perl might reduce their precision.

Greetings\,

Perl Info

``` Flags: category=core severity=none Site configuration information for perl 5.16.3: Configured by Marc Lehmann at Wed Mar 13 11:30:24 CET 2013. Summary of my perl5 (revision 5 version 16 subversion 3) configuration: Platform: osname=linux, osvers=3.2.0-4-amd64, archname=x86_64-linux uname='linux cerebro 3.2.0-4-amd64 #1 smp debian 3.2.35-2 x86_64 gnulinux ' config_args='-Duselargefiles -Duse64bitint -Dusemymalloc=n -Dstatic_ext=Fcntl -Dcc=gcc -Dccflags=-DPERL_DISABLE_PMC -DPERL_ARENA_SIZE=16376 -USITEARCH_EXP -USITELIB_EXP -UARCHLIB_EXP -D_GNU_SOURCE -I/opt/include -ggdb -gdwarf-2 -g3 -Doptimize=-O6 -fno-asynchronous-unwind-tables -fno-strict-aliasing -Dcccdlflags=-fPIC -Dldflags=-L/opt/perl/lib -L/opt/lib -Dlibs=-ldl -lm -lcrypt -Dprefix=/opt/perl -Dprivlib=/opt/perl/lib/perl5 -Darchlib=/opt/perl/lib/perl5 -Uusevendorprefix -Dsiteprefix=/opt/perl -Dsitelib=/opt/perl/lib/perl5 -Dsitearch=/opt/perl/lib/perl5 -Dsitebin=/opt/perl/bin -Dman1dir=/opt/perl/man/man1 -Dman3dir=/opt/perl/man/man3 -Dsiteman1dir=/opt/perl/man/man1 -Dsiteman3dir=/opt/perl/man/man3 -Dman1ext=1 -Dman3ext=3 -Dpager=/usr/bin/less -Uafs -Uusesfio -Uusenm -Uuseshrplib -Ud_dosuid -Dusethreads=undef -Duse5005threads=undef -Duseithreads=undef -Dusemultiplicity=undef -Demail=perl-binary@plan9.de -Dcf_email=perl-binary@plan9.de -Dcf_by=Marc Lehmann -Dlocincpth=/opt/perl/include /opt/include -Dmyhostname=localhost -Dmultiarch=undef -Dbin=/opt/perl/bin -Dxxxusedevel -DxxxDEBUGGING -Dxxxuse_debugging_perl -Dxxxuse_debugmalloc -dEs' hint=recommended, useposix=true, d_sigaction=define useithreads=undef, usemultiplicity=undef useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='gcc', ccflags ='-DPERL_DISABLE_PMC -DPERL_ARENA_SIZE=16376 -USITEARCH_EXP -USITELIB_EXP -UARCHLIB_EXP -D_GNU_SOURCE -I/opt/include -ggdb -gdwarf-2 -g3 -fno-strict-aliasing -pipe -I/opt/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O6 -fno-asynchronous-unwind-tables -fno-strict-aliasing', cppflags='-DPERL_DISABLE_PMC -DPERL_ARENA_SIZE=16376 -USITEARCH_EXP -USITELIB_EXP -UARCHLIB_EXP -D_GNU_SOURCE -I/opt/include -ggdb -gdwarf-2 -g3 -fno-strict-aliasing -pipe -I/opt/include' ccversion='', gccversion='4.7.2', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='gcc', ldflags ='-L/opt/perl/lib -L/opt/lib -L/usr/local/lib' libpth=/usr/local/lib /lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib libs=-ldl -lm -lcrypt perllibs=-ldl -lm -lcrypt libc=, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='2.13' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O6 -fno-asynchronous-unwind-tables -fno-strict-aliasing -L/opt/perl/lib -L/opt/lib -L/usr/local/lib' Locally applied patches: @INC for perl 5.16.3: /root/src/sex /root/pserv/lib/perl5 /opt/perl/lib/perl5 /opt/perl/lib/perl5 . Environment for perl 5.16.3: HOME=/root LANG (unset) LANGUAGE (unset) LC_CTYPE=en_US.UTF-8 LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/root/s2:/root/s:/opt/bin:/opt/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11/bin:/usr/games:/usr/local/bin:/usr/local/sbin:/root/pserv:. PERL5LIB=/root/src/sex:/root/pserv/lib/perl5 PERL5_CPANPLUS_CONFIG=/root/.cpanplus/config PERLDB_OPTS=ornaments=0 PERL_ANYEVENT_DBI_TESTS=1 PERL_ANYEVENT_LOOP_TESTS=1 PERL_ANYEVENT_NET_TESTS=1 PERL_BADLANG (unset) PERL_UNICODE=E SHELL=/bin/bash ```

[p5pRT]

From @tonycoz

On Thu\, Apr 18\, 2013 at 08​:04​:39AM -0700\, schmorp@​schmorp.de (via RT) wrote​:

[Please describe your issue here]

perldoc -f pack tells us​:

w A BER compressed integer (not an ASN.1 BER\, see perlpacktut for details).

Ok\, it's not an ASN.1 BER integer\, still\, it's an ASN.1 BER subidentifier (and is happily used to decode these in e.g. Net​::SNMP).

Anyways\, heading to perlpacktut to see why there is strong negation of it's ASN.1 BER qualities\, I find... nothing.

The only thing I can find is "Another Portable Binary Encoding"\, which tells us​:

(Details can be found at \<http​://Casbah.org/>\, the Scarab project.)

Apparently\, that domain fell off the internet in 2008. Looking at it in the wayback machine still shows that there are no details at that URL. At all. If there ever was\, it must have been buried deep inside that domain at some other URL. It looks more like some shameless plug for some project\, and in any case\, perl now refers to some kind of recycling shop or so.

I suspect it refers to this\, or a similar definition​:

https://github.com/mworks-project/mw_scarab/blob/master/Scarab-0.1.00d19/doc/binary-serialization.txt#L201

There's another note in the same file complaining about the usage​:

https://github.com/mworks-project/mw_scarab/blob/master/Scarab-0.1.00d19/doc/binary-serialization.txt#L151

Here are my suggestions to improve this situation​:

1. drop the "not an ASN.1 BER" comment\, or at least qualify it\, like "like an ASN.1 BER subidentifier\, not a BER integer".

At the very least the description of BER subidentifier encoding in X.209​:198811 6.22 is much more concise than any I found in the perl documentation\, so saying it isn't an ASN.1 BER is more confusing than helpful\, because by all means it is.

"the integer encoding used for ASN.1 subidentifiers"

2. remove the url in perlpacktut.

I wasn't able to find a good linkable reference either.

http​://luca.ntop.org/Teaching/Appunti/asn1.html

provides a good description\, but I didn't see any fragment ids.

Tony

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From schmorp@schmorp.de

On Thu\, Apr 18\, 2013 at 04​:00​:00PM -0700\, Tony Cook via RT \perlbug\-followup@&#8203;perl\.org wrote​:

1. drop the "not an ASN.1 BER" comment\, or at least qualify it\, like "like an ASN.1 BER subidentifier\, not a BER integer".

At the very least the description of BER subidentifier encoding in X.209​:198811 6.22 is much more concise than any I found in the perl documentation\, so saying it isn't an ASN.1 BER is more confusing than helpful\, because by all means it is.

"the integer encoding used for ASN.1 subidentifiers"

The BER encoding\, not the integer encoding. I don't think there strictly is something like an integer encoding for ASN.1 subidentifiers - subidentifiers *are* integers\, and BER is the encoding used by perl (there are many encodings for ASN.1) to encode such subidentifiers (or any other non-negative integer within range).

There also is a BER integer encoding\, but it is indeed different (and can represent negative integers as well).

It's also quite clear by the references you dug out that these integers are in fact supposed to be BER encoded\, and the "BER" term used by scarab is the asn.1 ber\, just that the author of the document was confused.

So\, saying it is the ASN.1 BER subidentifier encoding might be very helpful for some people\, and saying it isn't the BER intger encoding

2. remove the url in perlpacktut.

I wasn't able to find a good linkable reference either.

I don't think there needs to be one\, or even should one. I think it would be very bad for perl to refer to external documents to describe it's functionality. If it's built in\, perl should document it.

And I think it is documented reasonably well. What's not documented is why it isn't ASN.1 BER (when it is)\, and the fix for that would likely be to just drop the errornous statement.

http​://luca.ntop.org/Teaching/Appunti/asn1.html

provides a good description\, but I didn't see any fragment ids.

Well\, X.209 is freely downloadable from http​://www.itu.int/rec/T-REC-X.209-198811-W/en

However\, BER subidentifier coding is so trivial that having any external reference would be like hitting the perl user into the face with "yeah\, we could have described it in two sentences\, but it's so much more fun to force you to go online and wade through big documents".

Really\, the integers is just grouped into 7-bit groups from most significant to least signficant and all but the last have the 8th bit set.

It's so trivial it's not even neccessary to explain that this is asn.1 ber subidentifier coding\, it would just be a nice touch for those people who need to decode them\, but frankly\, that's probably just me and two other people or so\, and even I use "w" a lot more for other uses (such as length prefixes or a compact integer encoding).

But saying it isn't asn.1 ber encoding *is* confusing\, because I kept wondering about it\, because I couldn't see why not.

:)

--   The choice of a Deliantra\, the free code+content MORPG   -----==- _GNU_ http​://www.deliantra.net   ----==-- _ generation   ---==---(_)__ __ ____ __ Marc Lehmann   --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de   -=====/_/_//_/\_\,_/ /_/\_\

[p5pRT]

From @rurban

Marc\,

Can you please prepare a patch with the proposed changes. Thanks

[p5pRT]

From schmorp@schmorp.de

On Fri\, Apr 19\, 2013 at 02​:21​:40PM -0700\, Reini Urban via RT \perlbug\-followup@&#8203;perl\.org wrote​:

Marc\,

Can you please prepare a patch with the proposed changes.

I was under the impression I already gave one in my initial bug report(?)

--   The choice of a Deliantra\, the free code+content MORPG   -----==- _GNU_ http​://www.deliantra.net   ----==-- _ generation   ---==---(_)__ __ ____ __ Marc Lehmann   --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de   -=====/_/_//_/\_\,_/ /_/\_\

[p5pRT]

From @rurban

On Sat Apr 20 01​:43​:31 2013\, schmorp@​schmorp.de wrote​:

On Fri\, Apr 19\, 2013 at 02​:21​:40PM -0700\, Reini Urban via RT \<perlbug- followup@​perl.org> wrote​:

Marc\,

Can you please prepare a patch with the proposed changes.

I was under the impression I already gave one in my initial bug report(?)

No sorry. No attachments appeared at https://rt-archive.perl.org/perl5/Ticket/Display.html? id=117661 -- Reini Urban

[p5pRT]

From @rjbs

I have applied your suggestions in a branch here​: http​://perl5.git.perl.org/perl.git/commitdiff/e2b02bb

If they are acceptable to all\, I will apply them to blead soon.

-- rjbs