Add `cons` pair to `hy.contrib`

[brandonwillard]

How does everyone feel about (re)-adding a cons pair class—and complementary car, cdr—to hy.contrib?

The cons semantics are mostly necessary for work on adderall and hydiomatic, and extremely useful for adapting code from other Lisps, but its implementation isn't large enough to warrant a separate package, so hy.contrib seems like the perfect place.

If people are good with that, I can put in a PR right now.

[Kodiologist]

I made the case against HyCons in #1576. The warts I mentioned could be removed in a fresh implementation, but it's still not clear to me if there are any circumstances where cons cells are more convenient than all the other data structures Python already has. The justifying example can be just as easily accomplished with ~@.

[brandonwillard]

This issue isn't a rehash of #1576; removing cons as a built-in Hy object and syntax support for the dot notation is perfectly fine. We're talking about adding a cons pair and cons, car and cdr functions as an optional feature to support code written with their semantics.

@Kodiologist, your statements—and parts of #1576—are actually asking about the meaning and/or purpose of cons, car and cdr. Explanations/answers for those are largely context dependent and already well covered elsewhere, but one obvious stumbling block here appears to be the misconception that cons, car and cdr only server to (immediately) produce and/or manipulate a list. The fundamental idea of "partially constructed sequences" and the process of constructing sequences with cons has been overlooked (or outright dismissed), and, in #1576, point 2. iv. very clearly demonstrates this.

For example, in adderall and hydiomatic, the cons pair construct is used to represent such a partially constructed sequence. The cdr can be a "variable" object—or a structure containing such objects—that might later be replaced by an actual list, and, when that occurs, the same operators are used to succinctly produce a "completed" list. Basically, the same cons function handles both cases; plus, the "incomplete" list (i.e. a cons pair) serves to identify it as such and exactly how/where one can "complete" it. Without the semantics and constructs of cons, car and cdr, the logic underlying this process is considerably less succinct or ultimately reinvented in a way that speaks to much fewer people. In other words, recommendations to use ~@, +, extend, #*, etc., completely miss the point.

Regardless, the semantics and usage of cons, car and cdr do not have a one-to-one mapping via any combination of builtin Python or Hy data structures and functions (that I'm aware of, at least), so there is a need for an implementation when the semantics are used. Finally, if Hy is intended to serve as a Lisp, these basic Lisp constructs and semantics shouldn't be dismissed simply because some people don't use or understand them.

[Kodiologist]

your statements—and parts of #1576—are actually asking about the meaning and/or purpose of cons, car and cdr. Explanations/answers for those are largely context dependent and already well covered elsewhere

It's clear to me what cons cells and the cons, car, and cdr functions are good for in Emacs Lisp and Common Lisp. It's not clear to me what they would be good for in Hy.

For example

Can you provide an example that's concrete? You say that "recommendations to use ~@, +, extend, #*, etc., completely miss the point", but the one concrete example I've seen has a one-to-one reimplentation with ~@.

[brandonwillard]

It's clear to me what cons cells and the cons, car, and cdr functions are good for in Emacs Lisp and Common Lisp. It's not clear to me what they would be good for in Hy.

Emacs and Common Lisp both have list construction and manipulations functions nearly equivalent to Python's/Hy's, so what exactly are they "good for" in those languages? More importantly, by what measure are you judging Lisp conventions to be "good for Hy"?

Can you provide an example that's concrete? ...

adderall and hydiomatic are excellent concrete examples; please, do not be so unproductively dismissive.

You say that "recommendations to use ~@, +, extend, #*, etc., completely miss the point", but the one concrete example I've seen has a one-to-one reimplentation with ~@.

What example are you referring to?
If it's the one in #1576 recommending that (~?fn ~?name ~?params . ~?body) be replaced with (~fn ~?name ~?params ~@?body), then there's definitely no one-to-one reimplementation.

As I said,

For example, in adderall andhydiomatic, the cons pair construct is used to represent such a partially constructed sequence.

The point of the expression (~?fn ~?name ~?params . ~?body) is that it results in a partially constructed sequence (and later a proper sequence). hydiomatic and adderall operate on such objects and produce iterations of viable completions. This use of cons is an abstraction similar to delayed evaluation and macros in Lisp.

Such approaches are fundamental to symbolic computation (and even historically, Lisp itself), where cons, car and cdr are used to succinctly represent relevant algebras, structures built from them, and their properties.

[Kodiologist]

Emacs and Common Lisp both have list construction and manipulations functions nearly equivalent to Python's/Hy's, so what exactly are they "good for" in those languages?

Probably the clearest example is that in those Lisps, expressions are made up of cons cells. So, you need to deal with cons cells in order to do much metaprogramming at all. In Hy, by contrast, expressions are implemented as Python lists.

More importantly, by what measure are you judging Lisp conventions to be "good for Hy"?

What I mean by a feature being "good for something" is just that there's some circumstance or problem where it would be helpful. Having a practical purpose of some sort, in other words.

Can you provide an example that's concrete? ...

adderall and hydiomatic are excellent concrete examples

Those are entire libraries. Can't you be more specific?

please, do not be so unproductively dismissive.

Dude, I'm just asking for a concrete example. What's your problem?

What example are you referring to? If it's the one in #1576 recommending that (~?fn ~?name ~?params . ~?body) be replaced with (~fn ~?name ~?params ~@?body), then there's definitely no one-to-one reimplementation.

Yes, that's the one. Am I wrong in thinking that `(~a ~b . ~c) is equivalent to `(~a ~b ~@c)? If so, what exactly does `(~a ~b . ~c) return? What is a "partially constructed sequence" in terms of Python objects?

[Kodiologist]

At least in Hy 736426fc, right before #1580, `(~a ~b . ~c) differs from `(~a ~b ~@c) in terms of boxing: given (setv a 1 b 2 c [3]), it returns [HyInteger(1), HyInteger(2), 3] instead of HyExpression([1, 2, 3]). I'm not sure if that's the behavior you would want, or indeed why one would want it.

[brandonwillard]

Probably the clearest example is that in those Lisps, expressions are made up of cons cells. So, you need to deal with cons cells in order to do much metaprogramming at all. In Hy, by contrast, expressions are implemented as Python lists.

They might be made from cons cells, but when/if you're only ever manipulating or producing proper lists, there's effectively no need to use or deal with cons cells directly. Still, you've been implying that this cons stuff isn't necessary or necessarily advantageous, but now you're implying that they sometimes are—even in the presence of Python-equivalent list construction and manipulation functions?
And what exactly is the "metaprogramming" you're referring to? Is it something that can just as easily be done using Python lists? Is it related to the improper list/cons pair use-cases I've been talking about? If it is, then you're starting to answer your own questions.

Otherwise, if you can clearly identify those instances in which they are needed—outside of the construction of those aforementioned Python-equivalent list functions—you might also find some other reasons for using cons pairs. By the way, even if some of those reasons are convention/language-based, we are ultimately talking about a Lisp-like programming language (i.e. Hy), so we can't simply dismiss them.

As I said, there are quite a few existing resources demonstrating the non-list relevance for the combined use of cons pairs, cons, car and cdr. Here's a fairly apt set of examples. You can even look to the non-list examples in nearly any classic Lisp-based book or paper (e.g. SICP).

Those are entire libraries. Can't you be more specific?

Sure, start by try to rewrite this function without some form of cons pairs and car, cdr abstractions. It can definitely be done, but it's not likely to be more concise or as easily relatable to people working with these concepts and structures.

Furthermore, it's not apparent to me that all, more, or even some current hy.contrib offerings surpass the demands for justification and relevance leveled against these cons elements. Also, by removing cons functionality, two projects that very clearly demonstrate the power of Hy—hydiomatic and adderall—through their ability to bring non-trivial Lisp abstractions directly into the Python world, were completely disabled and made considerably more difficult to repair.

Out of respect for those projects alone, limited cons functionality could've simply been moved to hy.contrib. That approach also seems much more in-line with the deprecation process of most other mature software projects. However, if one assumes that the problems involved with not removing all aspects of cons clearly outweighed the value of those two projects, then, sure, but #1576 did not make that particular case.

[Kodiologist]

you've been implying that this cons stuff isn't necessary or necessarily advantageous, but now you're implying that they sometimes are—even in the presence of Python-equivalent list construction and manipulation functions?

I don't mean to imply that Emacs Lisp or Common Lisp would be worse off using their other data structures for their expression type. Maybe they'd be fine. But currently, they use cons cells.

And what exactly is the "metaprogramming" you're referring to? Is it something that can just as easily be done using Python lists?

Yes, since in Hy, expressions are Python lists.

Furthermore, it's not apparent to me that all, more, or even some current hy.contrib offerings surpass the demands for justification and relevance leveled against these cons elements.

That's fair. I weeded it once (#1189) and more weeding could conceivably happen. Certainly, any module that has no practical purpose whatsoever has no place in our repo. (Lookin' at you, botsbuiltbots.) I'm too much of a Perl programmer at heart to set more than a slight bar to new features.

Sure, start by trying to rewrite this function without some form of cons pairs and car, cdr abstractions.

All right, thanks. What exactly is the function intended to do? (unify ['a 'b 'c] ['a (LVar 'b) (LVar 'c)] (tuple)) is given as an example call, but what should that return? I presume that LVar is a generic encapsulating operator.

That approach also seems much more in-line with the deprecation process of most other mature software projects.

We're still 0.x, so we don't deprecate stuff, we just burn it.

[Kodiologist]

In other words, we are not a mature software project. Yet.

[brandonwillard]

Version numbers aren't what make a project meaningfully mature, nor are they justification for enforcing the preferences of individual project members, and especially not at an unnecessary cost to the development community. They exist as a service to those communities, and only sometimes as a means of justifying destructive changes when they serve an agreeable purpose—usually one that's equal to or greater than the things they destroy.

Community-based projects tend to become mature by actively respecting people's efforts and continually adopting and improving fair, transparent, easy-to-follow standards and practices, so that discussions like these become less and less frequent and necessary.
When that happens, it's easier for others to get—and stay—involved, or for existing contributors to confidently work toward larger improvements. That's how version numbers meaningfully increase.

Alternatively, without those elements of maturity, projects end up being driven by random individuals. Unless those individuals are consistently driving all the version-defining, large-scale improvements, contributions will mostly likely consist of minor improvements, and those version numbers won't reasonably increase.

[Kodiologist]

We're still unstable and still have many major bugs. That's why I say we're not mature; it has nothing to do with respecting efforts or fairness or standards or transparency. So long as we're still unstable and have many major bugs, I think that worrying about backwards compatibility is premature. I'm sorry this is something you wanted that we haven't done, but this is an all-volunteer project, so things only happen when someone is personally interested in doing them, especially unglamorous things like maintaining backwards compatibility.

Certainly I've never intended the version numbers of the Hy releases I've done to serve any communities. I've never served a community in my life, and I don't intend to start now! :)

[Kodiologist]

I'm closing this for lack of response, but let me know if you want to pick it up again and I'll reopen it.