Make native type and ref_to_native_type equivalent where ref_to_native_type is not a legal value

[Astara]

Make Explicit Sigil and Ref to Same, Equivalent (where refs to a type are not legal values)

• Wherever the language currently requires a value of type Array or a value of type Hash will allow a reference to the same type.

• Any place where the language allows more than one type, will allow a reference to one of those allowed types.

Note: *This feature request excludes indeterminate behaviors. Also, this not the same as the previous "autoderef" feature, as it had been modified to allow overload operators associated with blessed packages. The original autoderef feature did not allow blessed packages and neither would this one. More than once I've asked when the change to allow blessed packages was inserted into the code (as it was the reason this feature was declared invalid), yet no one seemed to be able to point to the discussion where changing this to allow such references existed.

In any event, since this feature doesn't allow blessed packages, it cannot be used with overloading.

[leonerd]

I'm finding it quite hard to read through the wording in this request, to work out what feature you're actually asking for.

Could you perhaps provide some small perl code snippets (ideally in the form of a Test::More-based unit test or similar) that demonstrate what it is you want to do, but which isn't currently possible?

Maybe that would gain more traction.

[jkeenan]

Make Explicit Sigil and Ref to Same, Equivalent (where refs to a type are not legal values)

* Wherever the language currently requires a value of type Array or a value of type Hash will allow a reference to the same type.

* Any place where the language allows more than one type, will allow a reference to one of those allowed types.

Note: *This feature request excludes indeterminate behaviors. Also, this not the same as the previous "autoderef" feature, as it had been modified to allow overload operators associated with blessed packages. The original autoderef feature did not allow blessed packages and neither would this one. More than once I've asked when the change to allow blessed packages was inserted into the code (as it was the reason this feature was declared invalid), yet no one seemed to be able to point to the discussion where changing this to allow such references existed.

In any event, since this feature doesn't allow blessed packages, it cannot be used with overloading.

I, too, found this feature request confusing. I grepped the core documentation and could find no instance of ref_to_native_type.

Thank you very much. Jim Keenan

[leonerd]

Also, this not the same as the previous "autoderef" feature, as it had been modified to allow overload operators associated with blessed packages. The original autoderef feature did not allow blessed packages and neither would this one. More than once I've asked when the change to allow blessed packages was inserted into the code (as it was the reason this feature was declared invalid), yet no one seemed to be able to point to the discussion where changing this to allow such references existed.

I'm also not sure where this assertion comes from. At no point in the entire version history for which autoderef was a thing, has it ever accepted blessed refs.

$ perl5.14.4 -E 'my $obj = bless {one=>1, two=>2}, "Object"; say for sort keys $obj'
Type of argument to keys on reference must be unblessed hashref or arrayref at -e line 1.

$ perl5.16.3 -E 'my $obj = bless {one=>1, two=>2}, "Object"; say for sort keys $obj'
Type of argument to keys on reference must be unblessed hashref or arrayref at -e line 1.

$ perl5.18.2 -E 'my $obj = bless {one=>1, two=>2}, "Object"; say for sort keys $obj'
Type of argument to keys on reference must be unblessed hashref or arrayref at -e line 1.

$ perl5.20.3 -E 'my $obj = bless {one=>1, two=>2}, "Object"; say for sort keys $obj'
keys on reference is experimental at -e line 1.
Type of argument to keys on reference must be unblessed hashref or arrayref at -e line 1.

$ perl5.22.2 -E 'my $obj = bless {one=>1, two=>2}, "Object"; say for sort keys $obj'
keys on reference is experimental at -e line 1.
Type of argument to keys on reference must be unblessed hashref or arrayref at -e line 1.

$ perl5.24.4 -E 'my $obj = bless {one=>1, two=>2}, "Object"; say for sort keys $obj'
Experimental keys on scalar is now forbidden at -e line 1.

It was removed at 5.24 because it was felt the postfix-deref syntax was better. This works unambiguously on the actual container type, so is fine for both unblessed and blessed references alike.

$ perl5.24.4 -E 'my $href = {one=>1, two=>2}; say for sort keys $href->%*'
one
two

$ perl5.24.4 -E 'my $obj = bless {one=>1, two=>2}, "Object"; say for sort keys $obj->%*'
one
two

[karenetheridge]

No one has linked to the previous closed issue on this subject -- https://github.com/Perl/perl5/issues/18518 -- which I think is necessary for context and background information.

also the p5p thread: https://www.nntp.perl.org/group/perl.perl5.porters/2021/02/msg258964.html

[Astara]

@Leonard -- the claim was made that the previous syntax allowed use of overloaded references. How is that possible unless the references were blessed? That the previous (or this) syntax did not work and had ambiguities was the reason stated for removing the previously working syntax.

The old syntax could be used, more concisely and with fewer characters than the new syntax. Ex:

> perl5.16.3 -MP -e'my $href = {one=>1, two=>2}; P for sort keys %$href;'
one
two

perl5.16.3 -MP -e'my $obj = bless {one=>1, two=>2}, "Object"; P for sort keys %$obj;'
one
two

What it doesn't work for is the simplest case:

> perl5.16.3 -MP -E'my $href = {one=>1, two=>2}; P for sort keys $href;'
 one
 two

This simple case only works for unblessed references, which was the original intention.

For those who don't understand that 'native_type' is a "native type" -- specifically ARRAY or HASH in this proposal, and that 'ref_to_native_type' is a "reference to [the same] 'native_type'", use this as a guide:

• 'native_type' is a "native type" -- specifically ARRAY or HASH in this proposal
• 'ref_to_native_type' is a "reference to [the same] 'native_type'"

The intent of this syntax was to simplify such references like:

• $href (a hash ref) being used in places where %$href is now required and
• $aref (an array ref) being used in places where \@$aref is now required.

The type of a ref is already known by perl by using the building ''''ref'''' operator which returns ARRAY or HASH depending on the type of ref. It will not work (nor is it intended to work) on a blessed reference.

As near as I can tell there is no "failure" case as some have stated, and the "autoderef" experiment never included cases that failed (like allowing use of overloaded operators), because overloaded operators can only be used in the context of a package with a blessed reference.

The existing syntax allowed since 5.14 through 5.22 doesn't appear to have any problems with ambiguity. So far, no one has presented any evidence or reasons why it was removed, nor given any reason why post-ref operators should be added to a language that contains too many sigils when no other modern language supports or uses such syntax. Why introduce a more complicated syntax when the less complicated, simpler syntax worked fine?

As near as I can tell, the postfixsigil feature was implemented because, the autoderef syntax had problems, yet no one is able to state what or how those problems occurred or wouldn't, for that manner, exist in the postfix syntax.

[leonerd]

@leonard -- the claim was made that the previous syntax allowed use of overloaded references. How is that possible unless the references were blessed? That the previous (or this) syntax did not work and had ambiguities was the reason stated for removing the previously working syntax.

First off it's "@leonerd", not what you wrote.

Secondly, where has that claim been previously made? I don't recall making it myself, and I don't see anyone else having made it. Do you have a reference?

Have you checked the list threads I mentioned in reply on the mailing list? Specifically:

• https://www.nntp.perl.org/group/perl.perl5.porters/2015/06/msg228318.html
• https://www.nntp.perl.org/group/perl.perl5.porters/2013/08/msg206464.html

Those all talked about the fact that postfix deref was deemed a better solution.

[leonerd]

For those who don't understand that 'native_type' is a "native type" -- specifically ARRAY or HASH in this proposal, and that 'ref_to_native_type' is a "reference to [the same] 'native_type'", use this as a guide:

'native_type' is a "native type" -- specifically ARRAY or HASH in this proposal 'ref_to_native_type' is a "reference to [the same] 'native_type'"

Some interesting notation. Did you document that previously where anyone else would be able to see it?

[leonerd]

The existing syntax allowed since 5.14 through 5.22 doesn't appear to have any problems with ambiguity. So far, no one has presented any evidence or reasons why it was removed, nor given any reason why post-ref operators should be added to a language that contains too many sigils when no other modern language supports or uses such syntax. Why introduce a more complicated syntax when the less complicated, simpler syntax worked fine?

As near as I can tell, the postfixsigil feature was implemented because, the autoderef syntax had problems, yet no one is able to state what or how those problems occurred or wouldn't, for that manner, exist in the postfix syntax.

Again I suggest going to read the mailing list archives. It was discussed there. I wasn't around at the time so I have no personal insight into the details; all I can do is read and quote things from public sources.

In any case, that all happened 6 or 8 years ago now.

[leonerd]

Ah; further discussion also on another mailing list thread linked from those ones, that I missed on my initial search because it has hyphenated "auto-deref":

https://www.nntp.perl.org/group/perl.perl5.porters/2013/11/msg209170.html

It would also appear I was around for at least some of the discussion, though I don't recall having much influence in the matter all the way back then.

[Grinnz]

On Fri, Feb 5, 2021 at 1:32 PM Paul Evans notifications@github.com wrote:

Ah; further discussion also on another mailing list thread linked from those ones, that I missed on my initial search because it has hyphenated "auto-deref":

https://www.nntp.perl.org/group/perl.perl5.porters/2013/11/msg209170.html

I would draw your attention specifically to https://www.nntp.perl.org/group/perl.perl5.porters/2013/11/msg209444.html as well as https://www.nntp.perl.org/group/perl.perl5.porters/2013/11/msg209460.html.

-Dan

[Astara]

It wasn't removed until 5.24, which didn't hit distros till at most, a few years ago. Time is funny, what you see as happening long ago, I see as a current issue, but like many in the outside world, I have been using earlier versions of perl. Regardless of the notes you provide from earlier discussions, be aware I was removed from the list for seven years. I could not be responsible or respond to any changes that occurred. Others who disagreed with the oligarch's also were censured and in some cases started supporting earlier versions or alternate versions of perl.

I don't put much stock in a perl developed in a closed environment or anything from 5.18 on. 5.18 added the forced warnings turned on, while the perl community, at the time, went against sound software practice in all other languages where fatal warnings were turned on by default. Warnings are a signal that something is wrong and that indeterminate behavior may result. They are a signal that the language has changed that may result in unpredictable behavior then or in the future. It's a security bug and disk-format potential all-in-one.

I remember being an advocate for simpler options for syntax, and to reduce unnecessary non-working cases to produce something useful.

As for the portions you quote, I see Ricardo claiming that the more complicated syntax was somehow better. By the measures of complexity and past understanding the new syntax was not better. Saying it is better without defining why or how it is better doesn't make it so. Ricardo was the pumpking and as such had dictator power to enact anything he wanted. No one spoke against him who wasn't told or forced to leave the community.

Someone else suggested a new syntax for "something" and one of the perl oligarch's asked what other languages had used such a construct that it should be adopted. I ask the same of the post-sigil syntax. Perl had a working syntax before that wasn't ambivalent, and was, most of all, simpler.

In the same way allowing printf FILE,"fmt", [vars] to work rather than emitting an error was suggested, as well as sprintf working with an array or list instead of a forced 'scalar', array. In both cases, perl produced or produces and error or something that doesn't work or is "almost never useful". -later I asked for cases to be included in the documentation where it was useful -- no one could think of any working code where it was used or useful. But I was told a warning message would be put in.

There were other examples where I asked for simplifications and one case where I suggested something that would have created an incompatibility, though no one could think of how it would arise. I later found it myself in my own testing -- but giving evidence to my premise that it was rarely used or done.

Besides those, I had other improvements, like removing the need for sigils where a variable of a unique name had been previously declared. So if you don't create the same variable name of different types, and you had them declared such that perl knew about them, you would be able to leave off sigils in a wide number of cases -- perhaps all cases where nothing fancy was being done. Javascript, which was derived from perl, demonstrates a sigil-less syntax that would be comparable in many cases.

Syntax that adds more sigils is not better by any computer language definition. Brevity and conciseness are hallmarks of a languages power -- allowing user to express more in less space, more quickly and with less work.

[Astara]

@leonerd wrote:

For those who don't understand that 'native_type' is a "native type" -- specifically ARRAY or HASH in this proposal, and that 'ref_to_native_type' is a "reference to [the same] 'native_type'", use this as a guide:

'native_type' is a "native type" -- specifically ARRAY or HASH in this proposal 'ref_to_native_type' is a "reference to [the same] 'native_type'"

Some interesting notation. Did you document that previously where anyone else would be able to see it?

Funny. I documented it as soon as someone claimed not to understand the notation, not before. I didn't realize how complicated such notation was until questioned about it.

[karenetheridge]

I didn't realize how complicated such notation was until questioned about it.

It's always more complicated when you get into details. When digging deeper, one encounters edge cases that were not initially obvious.

We await some code demonstrating the effects of changes you request, as per https://github.com/Perl/perl5/issues/18532#issuecomment-774153300.

[Astara]

` perl5.16.3 -MP -E'my $href = {one=>1, two=>2}; P for sort keys $href;' one two

` Doesn't work anymore. Changes I request would make it work as it did in 5.16.3.

[Grinnz]

How do you propose to make this consistent with all keywords that accept arrays and all forms of arrayrefs not just unblessed ones, as laid out in the autoderef discussion I highlighted here: https://github.com/Perl/perl5/issues/18532#issuecomment-774219380

[Astara]

The only array ref is one that perl return 'ARRAY' when used with 'ref'. You are talking about objects. You and others don't seem to get what objects are. They are opaque. Neither I nor perl can make assumptions about what is inside the object. To do so violates object integrity. To have perl "peer" inside an object and operate on it based on some underlying fundamental data layout or arrangement.

I'm not sure how to say this gently, but what you are asking violates the most basic principles of object oriented design.

Even if an object appeared to use a native ARRAY or HASH as a basis, can you guarantee that all such objects would behave, in all ways as perl Arrays or Hashes? You can't. An object makes no guarantees to the world outside of the object. To try to come up with something outside of the object that makes assumptions about the internal arrangement of the object is guaranteed to fail.

Suppose I want base my object on a HASH, and tomorrow I want my HASH to be ordered. That's entirely legal and doable inside an object, but perl only supports unordered HASHes natively. Worse, suppose my ordering is based on a user-defined sort algorithm.

Nothing can be assumed about the interior of an object by users or by the perl interpreter or language. It breaks object integrity to have things outside the object make assumptions about the object. The object has the freedom to use any representation the user wants to create -- but that will never mean that those representations are native types. C++ has native data types, but once you encapsulate a data type in an object the compiler makes no assumptions about the internal layout of the object. As such, you can't apply functions or subroutines designed for native, 0 based arrays or ones designed for perl's hash's to an object (anything that is blessed).

This is not a generic change to apply to all references, because perl knows nothing about data types other than its own native data types. Objects/blessed refs, are not interchangeable with native arrays or hashes. Trying to makes refs to 'native types' solve an unsolvable problem would be like putting a requirement on division that it handle divide by zero, or that that the 'log' function in perl handle negative numbers. Those are constraints set by the algorithms.

In the same way objects preclude arbitrary deconstruction that assumes they are equivalent to some other type (like an array or a hash). That's not how objects in objected oriented programming works. Not understanding that division cannot support division by zero doesn't mean you throw out division. Similarly you can't force logarithms or that root functions work with negative operands in the domain of real numbers as supported by most modern computer languages.

Functions operate on a 'domain' of inputs. That doesn't make the functions "wrong" or "inadequate". It is a natural constraint on the types of inputs allowed.

The only places where I am proposing allowances are ones where perl already knows the type of the reference -- where the 'ref' keyword returns the same type as the type required by perl, namely functions for that require an ARRAY or HASH, like delete, each, exists, keys, pop, push, shift, splice, unshift, values and keys. In no case does this permit an undefined value be dereferenced by perl into an ARRAY or HASH.

As an aside, I don't see where one could use a postfix notation to specify something that isn't already permitted with @{} or %{}, but such an example is not entirely germane to the utility of this proposal, but that doesn't mean I'm curious as to why a less familiar and more complex syntax was deemed to be "better". Certainly, such syntax would seem to violate the desire for perl operators to make learning perl easier for those familiar with C (familiar != same) (as stated in perlop).

So I don't propose to make a perl language feature violate object integrity. It would be poor software practice, IMO.

[Grinnz]

The topic referenced was using object refs as arrayrefs that are designed to be used as arrayrefs, explicitly so as to not to violate their encapsulation. You have just written an entire argument against a strawman.

[Astara]

And what is the syntax in perl that indicates such a design?

[Grinnz]

It is not related to syntax but to language design - the ability to pass a reference to, say, a Mojo::Collection or IO::All object to anything expecting an array reference.

[Astara]

But a ref to Mojo::Collection or IO::All does not return 'ARRAY', thus neither is an ARRAY references. Overriding an object that returns something other than ARRAY when 'ref' is used on it, isn't an ARRAY. You may think that it looks like an ARRAY, but unless applying 'ref' to it yields ARRAY, how can perl know? Overriding it by either \@{} or any other syntax doesn't make it an ARRAY, it just violates object integrity.

[Astara]

@leonerd

re: autoderef... In any case, that all happened 6 or 8 years ago now.

it was released in perl5.24 in May 2016. It wouldn't have been included in released distros until at least a year or two later, so maybe 2018, and depending on when you installed a distro upgrade, you are looking at 2019-2020 timeframe.

Companies updating would even be slower, with some on releases prior to 5.12, Though if you were there and commented on the issue (as you mentioned) and didn't even remember doing so, expecting others to know about it in the same timeframe seems a bit optimistic. I first found out about it about a year ago and tried to have it put in when issues were still being discussed on p5p. The pumpking at the time was sounding way too stressed to deal with the issue and wanted it dropped until after that upcoming release.

The perldelta in 5.24 said

The experimental C<autoderef> feature (which allowed calling C<push>,
C<pop>, C<shift>, C<unshift>, C<splice>, C<keys>, C<values>, and C<each> on
a scalar argument) has been deemed unsuccessful. It has now been removed;

It gave no details about why it was "unsuccessful" and even saying it was unsuccessful was no more true than calling "division" unsuccessful because it doesn't handle division by zero. Some complained about it because it didn't handle blessed objects. As I have pointed out, treating an opaque object the same as a native array or hash, cannot be done safely. It violates object integrity (even though some claim it is their intention that some objects behave the same as their underlying native types). The problem is that perl, of course, cannot know the developer's intention, and just because it might work for some specific use case, doesn't mean it would work for objects in general. I.e. some gave cases where blessed references were designed to be like the built-in types, but others pointed out that blessed references also support operator overloading which prevent any assurance of how a blessed object behaves (regardless of how it was designed to behave).

I have more than one module that implemented objects with a HASH, that later had the underlying datatype become a type of ordered HASH and in another case as an ARRAY. How the data type is implemented in an object is something perl doesn't know and cannot know in regard to blessed-references.

[leonerd]

It is rather difficult to hold a conversation with someone who doesn't read and follow the references in replies.

It gave no details about why it was "unsuccessful"

It most certainly did. The main discussions happened in November 2013, a link to that thread I have already posted, but here again:

https://www.nntp.perl.org/group/perl.perl5.porters/2013/11/msg209170.html

For the sake of posterity and letting third-parties follow this at least, I shall briefly summarize the discussion. There are two separate reasons explained here.

Reason 1

While it does touch on subjects of "blessed objects" it centres around the fact that, elsewhere in perl, we have many operator overloads that allow objects (regardless of their internal representation) to be able to appear like numbers, strings, arrays or hashes. For example, well known are the things like Math::BigRat or various vector/matrix objects which can take part in all the numerical operators like + and sqrt and so on, and thus feel like they are just as numerical as a native number. Similarly, we have operator overloads that allow objects to appear to behave like arrays or hashes, but perhaps are implemented as something else inside (maybe a gateway to some database table or whatever). Thus, when we e.g.

push @{$myarr}, 123, 456;

push $myarr->@*, 123, 456;   # 5.18 onwards

we might be pushing into a plain perl array, or we might be activating the operator overloads on some object and end up INSERTing rows into a database, or whatever. In the latter case perl does not care about the internal representation type of the $myarr object instance, only that it can claim to appear to be an array-like thing if asked to be.

We could even consider some sort of "OrderedHash"-type object, which can appear to be both a hash, and an array, if it provides both kinds of overloading. Quite what it does for a key string when someone pushes a new value I can't imagine offhand (maybe a UUID or timestamp?), but that is irrelevant to this discussion. Point is, objects can appear to be both.

Now, without autoderef this is all fine. We can always tell what kind of overloading we want at any one time.

push @$orderedhash, "new val";   # arraylike
$orderedhash->{newkey} = "new val";  # hashlike

push $orderedhash->@*, "new val";  # arraylike on 5.18

say for $orderedhash->@*;  # arraylike
say for keys $orderedhash->%*;  # hashlike

But because of the decision back in 5.12 to allow keys, values and each to operate on arrays as well as hashes, we have a problem. We now don't know which of the overloaded operators to invoke on:

say for keys $orderedhash;

In summary: autoderef is problematic because a) Perl wants to allow any object class to appear to behave like any core type b) Objects classes can choose to appear like more than one object type c) keys, values, each can operate on either hashes or arrays, d) autoderef has to guess which

This problem could have been solved by removing any of those four parts. The chosen solution was d, because it was newest and still experimental. (Personally in hindsight, I think I would have removed c - but it's rather too late for that now).

I will again emphasise this has nothing to do with the representation type of $orderedhash, and is all about what core-like interface(s) it claims to have via the overload pragma. I therefore find that any further discussions that only mention "the representation type of a blessed object" as motivation are not relevant and offtopic, and I shall downvote any that I see.

Reason 2

The other reason, totally unrelated to any of the above, is also mentioned in the mail thread.

This centres around the problem of foreach. It makes the argument that if we allow keys $arr, values $arr, etc.. to appear to operate on the list of values contained within that referenced array, then it becomes somewhat reasonable to expect that

say foreach $arr;

to operate on, not the singleton list containing this single $arr reference, but instead every element of the array so referenced. This way quickly leads to all sorts of problems for the language. Some of the awkwardness this would result in can be seen in Python, which makes a similar choice and I explained further

https://www.nntp.perl.org/group/perl.perl5.porters/2013/11/msg209463.html

It is therefore easiest and most consistent to explain this in terms of parts of syntax that appear to be equivalent to a plain array or hash variable, which the prefixing @{...} or postfix ...->@* operators provide:

say foreach $arr->@*;

my @items = $arr->@*;
say foreach @items;

say keys $hash->%*;

my @keys = keys $hash->%*;
say @keys;

Again, this reason is entirely unconnected with the internal representation type of blessed objects; and thus similar replies on that theme will be downvoted and ignored.

[leonerd]

it was released in perl5.24 in May 2016. It wouldn't have been included in released distros until at least a year or two later, so maybe 2018, and depending on when you installed a distro upgrade, you are looking at 2019-2020 timeframe.

Yes; it is regrettable that some folks find themselves inside a longer, slower feedback loop than would be useful, when getting into deep design discussions on an open source developed project.

However, there are many options available to you to shorten that cycle from years, into months, weeks... maybe even days. Of course it is by no means expected that every perl user be running the latest code developed this week - but anyone who wants to get into deep and involved discussions on the fundamentals of core language features should at least be expected to be familiar with the latest stable release (5.32 at time of writing) within a few months or so.

We cannot hold features hostage across a decade simply to wait for everyone to have updated to a version via their slow vendor channels and can now see it. We have to find some sort of compromise position that allows a decent velocity of forward progress.

[Astara]

@leonerd wrote (https://github.com/Perl/perl5/issues/18532#issuecomment-774658783):

> (> It is rather difficult to hold a conversation with someone who doesn't read and follow the references in replies.
> > The perldelta in 5.24 said:
> > ...
> > It gave no details about why it was "unsuccessful"
> 
> It most certainly did. 

Please do not quote out of context

If you are quoting For the sake of posterity and letting third-parties follow this at least attribute the source that was quoted, so that others can look at the original for themselves.

I stated that the text I wrote, was directly quoted from "5.24's official 'CHANGES' (perldelta.pod) document" (also, on the perl.org website, at https://perldoc.perl.org/perl5240delta .

It was the first item stated under section heading:

Incompatible Changes

NOTE: [an "aside" that I ran into while searching for an online copy of the page I quoted] The perldelta in version 5.24.0 (sadly not available via the archival/searchable name of "perl5240delta"[pod] within the release's "pod" documentation for "perl 5.24.0" that was(/is) included with the perl-5.24.0 release (as documented in github Perl/perl5/issue/15621 )), wasn't searchable via the "perl5240delta" tag until releases that followed the actual release.

@leonerd continues:

> The main discussions happened in November 2013, a link to that thread I have already posted,  ...

Links to a private email list (I was not allowed access at the time) are not something linked or cross-referenced when reading the Official Perl Documentation on perl.org's website nor in a non-hyperlinked environment as is available where the "perldeta" documentation is included in an unwrapped release-tarball or extracted, github src tree.

I'll attempt to digest the rest of your post, but it does not change the original proposal, nor the fact that the replacing post-typeID syntax you are referring to will be unfamiliar to most people familiar with other, mainstream languages or standard Perl syntax from the early 90's for the next 25 years.

Neither does it change the fact the added post-typeID operator is syntactically more complicated than the preexisting syntax** and more complicated than the original (pre perl v5.10+v5.12), still existing syntax.

**-note the removal of the earlier, simplified syntax was noted under "Incompatible Changes" in the perl5240delta document).

[Astara]

@leonerd

> It is rather difficult to hold a conversation with someone who doesn't read
> and follow the references in replies.
> @Astara:
> > The perldelta in 5.24 said:
> > It gave no details about why it was "unsuccessful"

I looked at the refs that were mentioned. None of them explained what 'failed' with the auto-deref that had been working since 5.14.

You seem to give reasons, but you are comparing old perl syntax with an extra sigil in front to adjust type, and the longer syntax with 4 extra chars on the tail to do the same.

Neither of those is about being able to shortcut syntax where perl requires a real ARRAY or real HASH ("real" as quoted from perlfunc).

> ## Reason 1
> While it does touch on subjects of "blessed objects" it centres around the fact that, elsewhere in perl, we have many operator overloads that allow objects ...
> 
> We could even consider some sort of "OrderedHash"-type object, which can appear to be both a hash, and an array, if it provides both kinds of overloading.
> Now, without autoderef this is all fine. We can always tell what kind of overloading we want at any one time.

None of the stuff you are quoting would be changed by autoderef, as it doesn't apply to any of the example or cases you have mentioned so far.

> ```perl
> push @$orderedhash, "new val";   # arraylike
> $orderedhash->{newkey} = "new val";  # hashlike
> push $orderedhash->@*, "new val";  # arraylike on 5.18
> say for $orderedhash->@*;  # arraylike
> say for keys $orderedhash->%*;  # hashlike
> ```
> But because of the decision back in 5.12 to allow `keys`, `values` and `each` to operate on arrays as well as hashes, we have a problem. We now don't know which of the overloaded operators to invoke on:

Who is this "we" -- I don't recall having this problem.

> In summary: autoderef is problematic because

Ignoring that you have not demonstrated a problem, but are saying autoderef is problematic in your summary. ??? But you haven't proved this yet.

> a) Perl wants to allow any object class to appear to behave like any core type

Perl wants?  Perl doesn't have wants.  It has a design.  Please say that you want

something if it is what you want, but you are not perl.

> b) Objects classes can choose to appear like more than one object type

What do you mean "appear"? Please state it in precise terms. How does an object class choose anything? Again, it is a class, without a mind, it can't choose. Someone can try to implement a class that behaves more than one way, that is fine. But Perl is an object oriented language -- not a mind-reading language. It cannot know what is inside a reference if the reference is to an object. If the reference (cf. use of 'ref' operator), is of type ARRAY or HASH, perl knows what type it is, but if it is a user-built type, it doesn't know what is inside.

> c) `keys`, `values`, `each` can operate on either hashes or arrays,
> d) autoderef has to guess which

Autoref should never have been implemented to "guess". If it was, that was a bug in that it is not defined behavior. I have no example where autoref ever "guessed". Do you have the location (filename+line numbers) in the perl source, say for 5.16, where this guessing was implemented?

> This problem could have been solved by removing any of those four parts. The 
> chosen solution was d, because it was newest and still experimental. (Personally 
> in hindsight, I think I would have removed c - but it's rather too late for that now).

You are citing a "strawman" argument that never existed.  If you want to claim it was a problem, you need to provide code that failed under 5.16.  This is the crux of the issue.

I assert there was no problem or place where autoderef would have to "guess".

If you have such code that someone can run, then you will enlighten me.  I only need

objective facts. I've changed my mind before, and am willing to again if it is proven to be a problem, but at this point no one has been able to demonstrate how a '$ref' that is an ARRAY or 'HASH' cannot be taken as equivalent to \@$ref or %$ref .

> ## Reason 2
> The other reason, totally unrelated to any of the above, is also mentioned in the mail thread.
> This centres around the problem of `foreach`. It makes the argument that if we allow `keys
> $arr`, `values $arr`, etc.. to appear to operate on the list of values...
> contained within that referenced array, then it becomes somewhat reasonable to expect that
> ```perl 
> say foreach $arr;
> ```

keys/values of $arr are "pluralized" (made into a list) by the keywords. Use of they ref without they keyword drops the pluralizing effect. HOWEVER, you can't expand a bare ref inside of foreach because a bare ref in a for[/each]() is already allowed. As such, a for loop cannot autoderef an ARRAY or HASH ref. Only values and keys can auto-expand their operand

to operate on, not the singleton list containing this single $arr reference, but instead every element of the array so referenced. This way quickly leads to all sorts of problems for the language. Some of the awkwardness this would result in can be seen in Python, which makes a similar choice and I explained further

My proposal didn't allow you to expand if the presence of the ref was valid. Please look at the title of the original proposal:

Make type and ref_to_type equivalent where ref_to_type is not [currently] a legal value

> It is therefore easiest and most consistent to explain this in terms of parts of
> syntax that appear to be equivalent to a plain array or hash variable, which the 
> prefixing `@{...}` or postfix `...->@*` operators provide.

The autoderef can only be done where perl is looking at it's own native types for ARRAY or HASH. Second, this is about simplifying -- reducing the number of extra characters needed to accomplish a task. If a user is accessing a class, they have to do it with the extra sigils. The autoderef only works for the simply, built-in perl data types of ARRAY and HASH. Other places they may have to spell things out.

Autoderef is not a panacea -- it works only where perl already is CERTAIN about the type of the reference (ARRAY or HASH). IF there is ANY ambiguity, it cannot be used.

The only reason not to allow it is to screw the users who might want a more simplified form for their work. Let me reiterate: it doesn't provide any new functionality. It allows the saving of a small amount of typing -- and only where perl can use 'ref' and get back ARRAY or HASH.

One more comment, in a foreach(xxx) statement, if you see a ref by itself or not in the company of 'keys' or similar, then it won't be expanded. If you see it after a keyword like keys, each or values, then it must be a ref to a native perl type.

Does this answer anyone's question(s)?

[philiprbrenan]

use Test::More qw(no_plan);

my $a = {a=>1};

ok 1 if $a and keys %$a; #AAAA

ok 1 if keys $a;         #BBBB Experimental keys on scalar is now forbidden

=pod

Please provide a pragma:

use autoderef;

that makes #BBBB possible as it is difficult to explain to new programmers why they should code #AAAA rather than #BBBB.

=cut

[Grinnz]

Would it not be even more difficult to explain that you can't do "BBBB" if the variable happens to reference an object with a hash dereference interface, despite functioning normally as a hashref in any other part of the language?

[Astara]

Not really, as the object designer chose not to provide such an interface in their object.

The author of the object class might choose to make the class based on an array dereference interface at some point in the future. But an object doesn't specific which is used by default.

Certainly, I would not see why the object doesn't identify its interface to perl if it wanted to be accessed via the ARRAY or HASH interface.

Since currently printing a CLass will show "Class=ARRAY(0x1234) (which certainly isn't the same as Class[0x1234] or Class{0x1234}, Maybe "Class" claiming @ISA=(HASH) would provide a hint from the class creator as to which way it would be dereferened, but without the Class creator providing a hint that they want the class to be treated as a HASH or an ARRAY, it would be a random-choice made by someone that breaks object opacity.

[Grinnz]

The author of the object class can do that, by overloading hash/array dereference or documenting access to its internals. That is what we are talking about. Since you said this interface should be excluded from this behavior, it would be a special case if a hashref doesn't work in this operation if it happens to be an object with a hashref interface, violating the encapsulation of the object's ability to be used as a normal hashref.

[Astara]

The author can do what, exactly, I don't understand. A document would not provide an auto-derefernce hint to perl on how to handle an object reference that is not a native type. If you want to make the case that providing an "ISA" method of 'ARRAY' or 'HASH' be a hint for how perl should auto-dereference a reference to that class, I would think that would be a great idea. But without some official perl mechanism, to tell perl how it should treated, then there is no way it could (or should) be auto-dereferenced.
But requiring the author of the class to "ok" the treatment of the class having ISA-ARRAY or ISA-HASH properties would make it fine in my view.

[Grinnz]

That there is no mechanism by which code can tell whether a reference to an object should be interpreted as an arrayref or hashref is rather the problem.

[Grinnz]

I commonly use Mojo::DOM and Mojo::Collection as example objects for this design question. I don't recall if I've done so in this discussion so far, so I'll list them again for reference.

Mojo::DOM is an object that provides both hashref and arrayref overloads as part of its documented interface. It is meant to be usable as either, for different purposes.

Mojo::Collection is an arrayref-based object that does not provide any overloads, but documents access to its internal array as an interface.

[Astara]

What I'm saying, though, is if a class put the string 'ARRAY' or 'HASH' in its @ISA array, that would be an indicator of which type of object perl would be able to auto-dereference /treat it as.

The point of auto-deref is that it is automatic. If the reference doesn't declare itself to be ARRAY or HASH compatible by some peril-language means (like using ISA), then perl can not do runtime-autodereferencing of a reference automatically. That's the way it currently stands without any auto-deref feature, there is no way to provide such a decision for a class designer who refuses to add such information to a Class. Auto-deref can only occur for items that are known to perl to be ARRAYs or HASHs. I'm saying that providing for run-time auto-dereference via checking a class's ISA would be once such method.

[Grinnz]

It's not the way it currently stands because the autoderef feature does not currently exist. For this specific reason. Providing a method to opt in doesn't solve the problem of determining how to treat object classes that haven't opted in. Disallowing them is not a solution.

[Astara]

On 2021/04/06 17:06, Dan Book wrote:

It's not the way it currently stands because the autoderef feature does not currently exist. For this specific reason. Providing a method to opt in doesn't solve the problem of determining how to treat object classes that haven't opted in. Disallowing them is not a solution.

But since that feature has never been provided for objects -- only for native ARRAYS and HASHES, why would you think the addition of such a feature should apply to such objects?

It's not like we are breaking existing compatibility. We aren't providing compatibility, automatically with a new feature.

The autoderef feature works where '@' or '%' are currently required.

In the places where '@' and '%' are required, no class is allowed, so adding auto-deref for '@' and '%' does not remove any functionality from Class.

Now I agree to providing a way for a Class to specify auto-deref treatment via the class's @ISA, and you want to create an auto-dereference for those objects that refused to specify a type?

By not specifying a type they are not enabling compatibility.

Disallowing them IS a solution: that is currently what we have. So you cannot claim it is not a solution.

That a new feature doesn't provide a solution for something it cannot, means it is the status-quo. It's a bug in the new feature, but that those classes cannot take advantage of a feature -- thus they demand the feature not be implemented. Users, rather should demand those classes not be given special treatment.

You don't block floating point enhancement to a number format because some existing class uses a different number system like Roman numerals.

The suggested feature provides benefit to most users but it can't be everything to every user. That's not reasonable, but absurd.

[Grinnz]

But since that feature has never been provided for objects -- only for native ARRAYS and HASHES, why would you think the addition of such a feature should apply to such objects?

Because it is imperative that we consider consistent language design for new features. Consider: Someone attempts to use this feature with a hashref and it doesn't work. At best it provides an error message telling them they passed an object reference. But the reference being to an object was not relevant to any of their other usage, so they didn't have to know or care until now. This is why it violates encapsulation to implement this feature this way.

[shadowcat-mst]

Others who disagreed with the oligarch's also were censured and in some cases started supporting earlier versions or alternate versions of perl.

Grammatic errors aside, I don't consider this statement by @Astara to be in keeping with either the letter or the spirit of the code of conduct that covers such conversations.

I'd ask you to be significantly more precise in your proposal including example code for all of the cases that have already been raised as questionable by people - I'm aware that's a trifle annoying, but such is the nature of proposing core language features.

[Astara]

Because it is imperative that we consider consistent language design for new features. Consider: Someone attempts to use this feature with a hashref and it doesn't work. At best it provides an error message telling them they passed an object reference.

If they get an error, it is because it isn't a hashref, that means they were given incorrect information. An Object is not a hashref or an arrayref. There is nothing in Perl, right now, that defines an object to be such. I propose accepting the idea of providing a mechanism, but that mechanism is separate. You are requiring every new feature to handle everything. That's a straight-jacket that the language doesn't support.

The perlfunc page defines Functions for real @ARRAYS, for real %HASHES and list data. With 5.14 and above it also included ARRAYrefs and HASHrefs. There was nothing about those functions working with opaque objects.

You don't have a "hashref" until you have notified the language through some mechanism (which we've discussed)

But the reference being to an object was not relevant to any of their other usage, so they didn't have to know or care until now.

The reference to it being an object not being known by them was due to inadequate documentation, as it is obviously not the same as a hashref from perl's point of view. Perl figures out the 'type' of something by using the 'ref' keyword. ref doesn't return 'HASH' for an OBJECT. You are attempting to support a construct that the language doesn't support and calling it a violation-of-encapsulation if the language doesn't work the way you want on the new object. This is why, in earlier discussions I asked you how your class tells perl (not a user of the class, as that's just deluded or wishful convincing) it should be treated the same as a HASH or ARRAY. Currently perl does not support this construct.

Claiming that autoderef on native ARRAY and HASH references doesn't work because the language doesn't support assigning other characteristics to non-native types isn't a failing of autoderef, but of the perl language. You can't put a Regex-ref in those positions either -- yet a Regex might act as an array or hash too .. you completely ignore the fact that you are in a defined group that isn't equated with ARRAYs or HASHs.

[Grinnz]

No, these objects are usable as hashrefs in every other part of the language. That already works. A function that takes a hash variable can also take an object reference dereferenced as a hash. The fact that a reference to some objects can be treated identically throughout the language as a reference to a hash or array is a feature, not a restriction. This is similar to the concept of polymorphism. If an API returns a hashref, it can start returning a reference to an object that can be used as a hash, and nothing will break - this is used across CPAN and core. Another example (though not relevant to this discussion) is exception objects that transparently work as strings when used as such, without any part of the system having to know it was an object.

Perl does not figure out the type of something with ref - it figures out the type of something by trying to use it, and either succeeding or failing.

[Astara]

Others who disagreed with the oligarchs' also were censured and in some cases started supporting earlier versions or alternate versions of perl.

Grammatic errors aside, I don't consider this statement by @Astara to be in keeping with either the letter or the spirit of the code of conduct that covers such conversations.

Grammatically, the possessive on oligarchs' was of the incorrect number. Given your confusion about this statement's alignment with with letter or spirit of a code of conduct, it might be prudent to make sure your understanding of this statement is clear. Oligarchies are groups-of-people who are ruled by a few individuals. If it is really only one individual, it might be better termed a monarchy, but perl, over its history from before ~5-6 years ago had often been referred to as being run (ruled) by a cabal which, I adjudged to be a synonymous usage of a group of oligarchs. Internally, the cabal deferred to a pumpking, though like congress, the actual power sometimes varies between the congress and the president. Usually a pumpking presides over 1 perl release, though from around perl 5.18 up to perl in the early to mid 5.20's it was presided over by 1 person. I suppose some might argue that it really was a monarchy during that period, but I hardly see referring to that as being a violation of some code of conduct (more than one would apply, being posted here on github). Perhaps you could clarify exactly how this violates some code of conduct? Otherwise it seems like an unsubstantiated accusation to my detriment. If it is your intent to do so, please be a bit forth-coming as to how.

I'd ask you to be significantly more precise in your proposal including example code for all of the cases that have already been raised as questionable by people - I'm aware that's a trifle annoying, but such is the nature of proposing core language features.

Given that my proposal isn't new, but is meant to restore a feature recently removed, the details would be a part of its original inclusion. I would prefer not to get things wrong or leave things out, as I'm simply trying to restore that already implemented functionality. If you could point me to where it was proposed/defined, I can make sure to include the precise language from the original proposal to clearly state the intentions in restoring that feature.

[Astara]

But since that feature has never been provided for objects -- only for native ARRAYS and HASHES, why would you think the addition of such a feature should apply to such objects?

Because it is imperative that we consider consistent language design for new features. Consider: Someone attempts to use this feature with a hashref and it doesn't work. At best it provides an error message telling them they passed an object reference. But the reference being to an object was not relevant to any of their other usage, so they didn't have to know or care until now. This is why it violates encapsulation to implement this feature this way.

One cannot consider all language designs for new features. The number of incompatible, new features is innumerable. Thus it cannot be considered in the adoption of new features that benefit the existing language as auto-dereferences did for for the few existing items that REQUIRED a native array, '@', or a native hash '%'. Future features need to find ways to become compatible -- and, indeed, a way has been suggested where by the perl interpreter could auto dereference types for which $typeref->isa(ARRAY) or $typeref->isa(HASH) is true, as defined by placing 'ARRAY' or 'HASH' into the class's @ISA array beforehand. Saddling the existing language with requirements with future features that don't currently have working counterparts in the language is also unreasonable and a way to prevent logical extension of existing features to support a more generally useful perl rather than one limited by constraints and needs of advanced research topics that have special needs for compatibility that they refuse to implement to not be a burden on existing perl's needs.

[Grinnz]

But since that feature has never been provided for objects -- only for native ARRAYS and HASHES, why would you think the addition of such a feature should apply to such objects?

Because it is imperative that we consider consistent language design for new features. Consider: Someone attempts to use this feature with a hashref and it doesn't work. At best it provides an error message telling them they passed an object reference. But the reference being to an object was not relevant to any of their other usage, so they didn't have to know or care until now. This is why it violates encapsulation to implement this feature this way.

One cannot consider all language designs for new features. The number of incompatible, new features is innumerable. Thus it cannot be considered in the adoption of new features that benefit the existing language as auto-dereferences did for for the few existing items that REQUIRED a native array, '@', or a native hash '%'. Future features need to find ways to become compatible -- and, indeed, a way has been suggested where by the perl interpreter could auto dereference types for which $typeref->isa(ARRAY) or $typeref->isa(HASH) is true, as defined by placing 'ARRAY' or 'HASH' into the class's @isa array beforehand. Saddling the existing language with requirements with future features that don't currently have working counterparts in the language is also unreasonable and a way to prevent logical extension of existing features to support a more generally useful perl rather than one limited by constraints and needs of advanced research topics that have special needs for compatibility that they refuse to implement to not be a burden on existing perl's needs.

This is a strawman. No one is suggesting Perl must be a perfectly spherical cow. Every feature must be weighed in its contribution to Perl's design and its risks. The difficulty of doing so does not obviate the requirement. The method you are proposing to implement this feature is inconsistent with Perl's design, in that consideration of such was exactly why this feature was already removed from the language.

[philiprbrenan]

Yet NeilB has kindly explained that the use feature capability does indeed allow us to introduce new, incompatible features for just this kind of experimentation. If a user wishes to experiment then I say that we should be encouraging them to do so rather than telling them that it would be bad for them otherwise they will go and find a language which does strive to meet their needs more effectively than Perl.

At the heart of the objection is perhaps the notion that "we do not have the time and resources to create/support/document/test such capabilities"

• this might be true but if it is, are we then just planning to fade away gracefully into the sunset and hope no-one notices?

Tomorrow, and tomorrow, and tomorrow, Creeps in this petty pace from day to day, To the last syllable of recorded time; And all our yesterdays have lighted fools The way to dusty death. Out, out, brief candle! Life's but a walking shadow, a poor player, That struts and frets his hour upon the stage, And then is heard no more. It is a tale Told by an idiot, full of sound and fury, Signifying nothing.

On Sun, Apr 11, 2021 at 2:39 AM Dan Book @.***> wrote:

But since that feature has never been provided for objects -- only for native ARRAYS and HASHES, why would you think the addition of such a feature should apply to such objects?

Because it is imperative that we consider consistent language design for new features. Consider: Someone attempts to use this feature with a hashref and it doesn't work. At best it provides an error message telling them they passed an object reference. But the reference being to an object was not relevant to any of their other usage, so they didn't have to know or care until now. This is why it violates encapsulation to implement this feature this way.

One cannot consider all language designs for new features. The number of incompatible, new features is innumerable. Thus it cannot be considered in the adoption of new features that benefit the existing language as auto-dereferences did for for the few existing items that REQUIRED a native array, '@', or a native hash '%'. Future features need to find ways to become compatible -- and, indeed, a way has been suggested where by the perl interpreter could auto dereference types for which $typeref->isa(ARRAY) or $typeref->isa(HASH) is true, as defined by placing 'ARRAY' or 'HASH' into the class's @isa https://github.com/isa array beforehand. Saddling the existing language with requirements with future features that don't currently have working counterparts in the language is also unreasonable and a way to prevent logical extension of existing features to support a more generally useful perl rather than one limited by constraints and needs of advanced research topics that have special needs for compatibility that they refuse to implement to not be a burden on existing perl's needs.

This is a strawman. No one is suggesting Perl must be a perfectly spherical cow. Every feature must be weighed in its contribution to Perl's design and its risks. The difficulty of doing so does not obviate the requirement. The method you are proposing to implement this feature is inconsistent with Perl's design, in that consideration of such was exactly why this feature was already removed from the language.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Perl/perl5/issues/18532#issuecomment-817232311, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZS3WQFNJMNTXJNUX6VLYTTID4TZANCNFSM4XFB5U4A .

-- Thanks,

Phil https://metacpan.org/author/PRBRENAN

Philip R Brenan https://metacpan.org/author/PRBRENAN