The second PEP draft

[gvanrossum]

Once within the authors group we have consensus on the major issues (primarily #90) I would like to do a thorough rewrite of the PEP, esp. the rationale and motivation, to lay out the argument much more clearly. We've had a bunch of unfocused negative feedback that comes down to people not understanding the PEP or getting distracted by poorly reasoned arguments and silly examples.

There has also been discussion about an "Anti-PEP" -- this was discouraged by the SC, but basically the people who oppose the PEP want their voices to be heard and I think we should summarize (or ask them to summarize) their main arguments and place them in a separate section of the PEP.

Also, I think the traditional structure of PEPs makes it hard to lay out clearly the arguments for design decisions (by putting everything in a "Rejected Ideas" section). In other languages (or proposals?) I've seen a better structure, where the description of each feature of the specification is immediately followed by a discussion of what the design choices were, how the feature interacts with other features, and and what the pros and cons of various choices are.

Who wants to help with this (ambitious!) second draft of the PEP?

[brandtbucher]

I agree that a rewrite is in order.

I’m going to focus the bulk of my time on the implementation, but I’m happy to play a part in the rewrite as well (especially reviewing to make sure the implementation is consistent with the spec).

[Tobias-Kohn]

I am happy to help with the second draft. Is there something in particular that I should take on?

[viridia]

I can pitch in depending on available time...

[dmoisset]

I'm quite interested in taking this, under the assumption that there's some rough consensus

[gvanrossum]

Unfortunately we're still stuck getting rough consensus. There are things that could be done before that's reached, such as rewriting the motivation, summarizing common objections, or coming up with a better set of examples. Honestly, we could use two separate sets of examples: one set to support the motivation, another set to support the specification. Possibly a third set to provide an introduction for people just interested in learning the basics.

[brandtbucher]

Is the general idea for the PEP to shrink or grow? Or do we not have a rough size target, relative to the current draft?

[gvanrossum]

I don't care if it grows -- I want it to be accepted. It would of course be nice if it shrunk, but I don't know what we could get rid of. (We could remove rejected ideas that haven't been brought up in the discussion, though perhaps that's because we clearly explained why they were rejected.)

There are two pieces of feedback from core devs I'd like to take especially serious: a message (or several?) from Greg Smith where he shows that the PEP confused him (a very experienced Python and C developer) big time; and Mark Shannon's thread about an "Anti-PEP" -- the conclusion of that thread seems to be that opponents might be happier if their view was represented fairly in the PEP.

[gvanrossum]

We also need a list of acknowledgements. I'd include:

• Daniel Moisset
• Taine Zhao
• Nate Lust?

Anyone I'm forgetting?

[dmoisset]

I'm sharing a first draft, but I'll work more on these today

[dmoisset]

If you check my draft PR, you'll see that it's getting in shape, I'm happy to receive reviews. Mainly what I did is:

• Rewrote the abstract + motivation trying to focus on our "deconstructing data" story
• Added a description of what a pattern is, some definitions, and an informal summary of all the pattern types at the top, before starting the discussion on syntax and semantics.
• Added "simplified syntax" to all of the pattern types that didn't have it (the constant value is still left TBD until a decision is reached).
• Fix some minor typos

I know the PEP is already huge, but I'd like to have an examples section (even if it's one or two, with a link to an external document) to drive the point that a lot of code can be much prettier using this, and describe some concrete applications beyond "walking ASTs" :).

[dmoisset]

OK, a couple of status updates on this:

• I played a bit how would it look to write some pseudocode semantics, thinking of the requests for pseudocode for the class matching we saw in python-dev. It ended up in a document that's accurate, but a bit long/dry to go in the PEP, so I saved my notes elsewhere: https://github.com/dmoisset/notebook/blob/master/python/pep622/semantic-specs.md ; you can ignore it, or keep it as a resource to send to the mailing list if people say that this is underspecified or confusing, or perhaps grab little bits of it that you like or think are useful.
• I'm collecting examples to include. I have the app in the examples directory of these repo, the (somewhat outdated) EXAMPLES.md, are there any others that have been gathered by you, or should I start hunting for example snippets?

[natelust]

It's going to be a busy weekend for me, but I have a few toy examples I have been playing with. I will try to get them cleaned up and open a pull request. They might not be worth putting in the pep directly, a link to the examples in this repo might suffice. Does anyone have objections to data files in the examples directory? I have an example in mind that would work well with a small sqlite file.

[gvanrossum]

FYI, this message from Mark Shannon pointed out significant problems with how the introduction is written. I don't believe Daniel has rewritten this part yet.

[gvanrossum]

This message from Greg P Smith also worries me (especially since I respect Greg so highly). It presents the POV of someone who knows Python (and C, in Greg's case) but doesn't know match from any of the other languages.

[brandtbucher]

Agreed, I think the PEP misses a great opportunity to explain how match-case can be used to introduce pattern matching (and related concepts) to those who've never seen it before... and I think that if we couple this with the motivation, it could effectively set the stage for the proposal at the same time. Python is regularly employed as a teaching language, so I think there is a natural flow from:

• We build data structures for convenience.
• We often want to break them back down again for use.
• The simplest example of this is unpacking, which allows us to conveniently iterate over or return more than one value. It's not very expressive, but it saves us a lot of manual work pulling apart data that takes very simple, specific forms.
• Pattern-matching allows us to not only pull apart data, but also decide how we want to handle data based on the way it's structured, or the properties it may have.

The beginner may not even draw any connection to switch-case / load semantics, when introduced in this way. I still love the Point examples for their expressiveness and simplicity, and I think we should definitely lean on ones like them more in our examples (I think anyone writing node visitors can make the obvious connection).

[stereobutter]

@dmoisset I like both the UI event and fizzbuzz example in https://github.com/gvanrossum/patma/issues/49. I also posted an example there for a recursive function (that ist not about AST transformation).

[viridia]

I also have been reading some of the reactions to the PEP in the python-dev archive. Here are some of my reactions:

1) The fact that there's a fully functional implementation does not mean that the PEP is set in stone. The reason for having a working prototype is to help people explore and assess the proposal. Because adding a new statement to Python is a high-impact change, it's hard to mentally simulate all of the potential consequences of the change in an abstract way.

We have never claimed that the working prototype will (or will not) be used as the final implementation.

2) Similarly, the fact that the PEP appears fully formed, like Athena springing from Zeus's forehead, is not an indication that it is "finished" or that it is not subject to criticism and revision. We are in the process of making several major changes to the PEP in response to community feedback.

3) I think one of the hardest sticking points is the idea that match patterns are a different syntactical and behavioral world than the rest of Python. Some folks were complaining about the fact that constructor matchers looked too much like constructors, which was intentional on our part. Inside of a match pattern, everything works opposite from normal (what I referred to as "the mirror of construction") - what looks like putting a tuple together is actually taking a tuple apart.

For us, it's a fairly simple mental leap to say "everything between this symbol and that symbol is the reverse of what you would normally expect - destructuring instead of construction." But for someone who hasn't made that mental leap, the fact that everything suddenly behaves the opposite of what they would expect creates confusion and uncertainty. Thus, you see folks calling for extra syntax to signal this different behavior.

[dmoisset]

OK, thanks for the awesome discussion. I'll see if I can cover your points better

[gvanrossum]

I did some drafting myself. I started with an Abstract. I am using Dropbox Paper which is similar to Google Docs but understands markdown. If you have an account you can edit and leave comments.

https://paper.dropbox.com/doc/PEP-622-new-intro--A3Izq1j0KV0bBEGm9Fuz6KdBAg-f9Cb9yaOV4olvbeDWG1ML

[dmoisset]

I like some of what I see there, @gvanrossum . I stole some of it and added it to my version, including a new section right under the abstract titled Informal overview. The goal of that section is giving the "elevator pitch" and a simple example of the idea, without making the abstract that long. I think it aims directly to tell "this is what's about, and how you read it", likely answering people like Greg Smith that considered this confusing. It also stresses very early on, as @viridia suggested, that patterns are "reverse constructors".

[dmoisset]

Going over Mark Shannon's email I think many of the things he didn't like about presentation have been addressed. I still see the following of his points that I haven't covered in my version:

1. Why couple the choice part (a sort of enhanced elif) with destructing (a sort of enhanced unpacking)? We could have a "switch" statement that chooses according to value, and we could have "destructuring" that pulls values apart. Why do they need to be coupled?
2. An explanation is needed of why "destructuring" needs to be so tightly coupled with matching by class or value.
3. Without modifying Node or Leaf, the matching code will need to access attributes. You should at least mention side effects and exceptions. E.g. matching on ORM objects might be problematic.
4. Python's support for OOP provides an alternative to ADTs. For example, by adding a simple "matches" method to Node and Leaf, is_tuple can be rewritten...
5. Should [typing.sealed] be in a separate PEP? It seems only loosely related

What I'd respond to each of this is...:

1. Patterns could be used as a standalone syntactic construct in several other places in the language. Someone suggested just that a few days ago regarding assignment, and I emailed Guido about other things like this. I believe the short answer is "this opens a huge can of worms, let's start with something self-contained", and if you have to do one thing with patterns, it will be a matching statement (the smallest thing maybe would be a matching operator, I have no arguments regarding that). I could put all of this right after saying what a pattern is, before introducing match: "There are many possible language constructs that could benefit with patterns, but to minimize the design impact this PEP only propose the single most useful one" I could use some of your help here.
2. I think I cover some of these in my rationale, although not very directly (and with a fuzzy "destructuring data of any kind is sorta important") what do you think?
3. I can add a small note to the semantics section.
4. I understand that the argument here is that we can provide a universal feature rather than asking each class to implement it, and this also allows us to apply the feature for classes that we don't "own".
5. What's your answer for this? Should we do it? I understand that this could go in another PEP, it's probably easy to pass than one through (I think PEPs that just adds stuff to typing without runtime effect don't have that much bikeshedding), and if it makes PEP-622 easier to digest to people, I'm all for it.

[viridia]

I'd like to address item 4 on your list.

The strict OOP philosophy is that you put logic that deals with the state of an object as a method on the object itself. However, the last several decades have shown a weakness in this approach, which is that it requires objects to understand every possible way in which that object will be used. I have written on this topic in much more depth in my essay, "The Rise and Fall of Object-Oriented Programming".

A great example of this is ASTs. AST node types shouldn't need to understand all of the various compiler passes or other visitors that walk the AST graph. For compilers, it turns out that it is architecturally much cleaner to put code for each pass together, separate from the data objects that the pass operates on.

It is true that Python has always been a language where OOP was the dominant paradigm, but that doesn't necessarily exclude other paradigms. The match statement is a prime example of a flow of control which turns OOP on its head - instead of having each object decide what to do, you have one central location that decides based on object type and shape.

I think part of the motivation for wanting match in Python is a desire to explore some of these other non-OOP styles. My expr.py sample program is just such an exploration - there's hardly any classic OOP in it, and yet the result is (IMHO) well structured and easy to comprehend.

Part of the hurdle that we have to overcome is the fact that people are used to thinking in an OOP way, and many college courses and textbooks dating from the 90s taught the rules of OOP in a very strict and dogmatic way. When people see code that is written in a way that violates the rules that they were taught, they might judge it to be a bad design or an anti-pattern.

[dmoisset]

@viridia, I agree with everything you just said :) What I'm still not sure is how to put this succinctly and decently argued without turning the PEP into an essay on the comparative benefits of different programming paradigms (writing is hard... ). I'll see what I can do

[gvanrossum]

@dmoisset

I stole some of it and added it to my version [...]

Sure (although I wasn't quite done yet -- I was just at a reasonable stopping point to ask for critique). In any case, I will probably stop writing for a while -- my tendonitis is getting worse.

[gvanrossum]

PS. How I wish we were using Dropbox Paper for the finessing of the text. I've already found a few questionable issues when skimming the rendered version of Daniel's rewrite, but it's too much work to track them down in his PR, so I hope I'll remember then when my elbow is better.

[dmoisset]

I could try moving my content into paper... then we can see how to merge it. I'll have to guess some process with Paper (if I don't like something do I just edit? is there a way to "suggest" that can be approved? I just did a fork because the git/PR is a model I'm familiar with :) ).

I'll try moving the "Informal overview" section for now, I think it's important and overlaps a bit with the abstract

[gvanrossum]

Paper emails me whenever someone makes a change (with some delays). If you see a typo or a slight grammatical issue just fix it; if it looks at all like different might want to say the same thing differently, select the text and add a comment with your suggestion.

[dmoisset]

OK, I moved most of my notes here. There was some other stuff in that PR that I'll submit separately

[gvanrossum]

Also, feel free to just add your work to my Paper doc rather than starting (yet) another one.

[natelust]

I started a write up on how I would introduce this topic to students with some python knowledge. Notably the order I introduce things is different from that outlined in the PEP. I dont know if the lecture style fits well inside the PEP or not, and I didnt want to interrupt others work, so I created it as a stand alone document. If it would be better to be put in the document Guido linked at the bottom, I would be happy to move it there as well. @dmoisset If any of it is useful, you are free to take if for the pep, or it may be useful to link to. I am not finished with it yet, but you can find it here: https://www.dropbox.com/scl/fi/u14yp0msiy9bfmw3mr01m/Introduction-To-Pattern-Matching.paper?dl=0&rlkey=v56lbedzcaafx9xfr4d4vntmg

[gvanrossum]

Thanks! I would certainly expect that the way to introduce this to students is different from the PEP -- the PEP needs to make the strongest possible argument that we should add this feature and that our design is the best one (especially in the light of the feedback received on python-dev); while the students need the easiest path to understanding how it works (and that path may vary depending on what other parts of Python they already know when you start teaching this topic).

[gvanrossum]

@dmoisset I did a thorough review of your work in the Paper doc and left you a ton of comments. I also made some small changes directly in the text and used ~strikethrough~ to delete occasional words I think could be struck. After you work through my comments we should be able to turn this back into a PR.

[DimitrisJim]

The Class pattern is definitely going to be a limitless source of confusion due to the strong association between 'name followed by (args)' and calls. (the only association maybe? I can't think of cases where a name(args) expression doesn't mean "I'm making a call"). I see two main points raised by Greg P.Smith's email:

1. Implicit assignments (appearance of magic)
2. Class patterns just look like calls.

For 1. I think the rebuttal is easier: Python already has implicit assignments. In some cases as is used to signify that some assignment will happen (except ... as err, with open(...) as fp) in others we generally accept the implicit assignment due to it being ubiquitous in many other languages (for loops, function calls). Finally, decorators are probably the most magical and closely fit with what troubled Greg, I wouldn't be surprised if when decorators were added, people new with the concept raised similar concerns about how magical and confusing it is.

I believe the Runtime Specification should cover this point. It should eventually talk about the when and how of bindings since that's clearly a sticking point for some people.

For 2. the situation is trickier. If class destructuring was a thing (as in Rust and, I believe, JavaScript) I'm sure the mental switch between calls and destructuring would be easier. Looking through the PEP (I'm looking at Daniel's branch, maybe I missed it) I couldn't see the rationale behind () being chosen neither a Rejected Ideas section on why other forms where rejected. I personally don't think another form (e.g Greg's suggestions on using {} instead of () or @ between the name and the parentheses) would be a sensible approach since that breaks the, as viridia puts it, "mirror of construction" which I see as a good mental guide for match.

Though the PEP is getting quite large, a Rejected Ideas on a different pattern for classes might help in making it clear why () was chosen.

[gvanrossum]

What else looks just like a call but isn’t....?

A function definition. (Also a class definition.)

[DimitrisJim]

Well, yes, but those are statements, I was talking about expressions in that specific sentence. And though patterns should not be thought of as expressions I can see how it might be hard for someone to initially make that distinction.

[gvanrossum]

The top level pattern is preceded by a clear syntax marker, ‘case’.

Also, please stop, we have enough people here.

[gvanrossum]

After https://github.com/python/peps/pull/1500 I propose to post another version to python-dev. I’ll also write a cover letter explaining our response to the feedback.

Did I miss any tasks?

[brandtbucher]

I don't think so. Great effort this week, everybody!

[dmoisset]

After python/peps#1500 I propose to post another version to python-dev. I’ll also write a cover letter explaining our response to the feedback.

Did I miss any tasks?

Thanks for all the help and review during the process.

There are a few things that I thing could help make the PEP slightly stronger, which I can submit as a few extra PRs after this land. Some of these perhaps can be saved for emails during the discussion, where presenting ideas is probably as important as "getting the letter of the PEP right".

To summarize here what I had in mind:

• [x] A "tour" over patterns, described in #126 ; essentially this section https://github.com/dmoisset/peps/blob/motivation/pep-0622.rst#syntax-and-semantics
• [ ] A small appendix with "use cases". I don't think the PEP would be served by a long list of actual examples but an expanded with a slightly expanded version of "we found pattern matching useful when walking ASTs, processing UI events, implementing overloaded signatures", could support the motivation and people "get" where would this feature be used.
• [ ] I'd like to have some doc with good, up-to-date examples (I'm drafting that) so I can point people to that during the discussion (even if it's not part of the PEP, it would be useful to have this handy before submitting it)
• [x] A rejected/deferred ideas item referring to selecting match as our application of pattern matching and not other things (this addresses one of Mark Shannon's concerns). Something on the line of "There are many possible language constructs that could benefit with patterns, but to minimize the design impact this PEP only propose the single most useful one"
• [x] A small note in the semantics section specifying possible side effects (also addressing Mark's email)
• [ ] A rejected idea titled "let's not do this and use OOP dispatch instead" again addressing Mark's email. I think @viridia can argue quite eloquently in that direction.
• [x] Something minor but I added a "simplified syntax" to my version of the document (see below each "XXX pattern" header after https://github.com/dmoisset/peps/blob/motivation/pep-0622.rst#literal-patterns )... I found that made much easier for me to read the PEP on the original read rather than going back and forth to the appendix. I'e, if you're looking at what the class pattern does, you have a few lines of syntax neraby describing what it looks like

All of these should be smaller PRs that should be easy to accept or shoot down (so I don't think there are a big delay). I don't want to spam (more) this tracker, but if you give me a thumbs up I can create all of these as issues so we can track them/reject them.

[gvanrossum]

Thumbs up on all of those, either directly as PRs for the PEP or as iddues in this tracker. If it holds up the posting of the next version, that’s fine.

[viridia]

The hard part is going to be distilling it down to just a few paragraphs :)

[natelust]

@dmoisset . The guide to introduce and motivate for those unfamiliar with matching may be useful for your point 3 (either to save you some work or as a secondary link that explains things from a different perspective). I should be finished with a draft what I was planning to write late tonight. I'll let you know so you can decide if it's useful for what you were thinking.

[dmoisset]

I added a PR for some of those points in python/peps#1501 . Will continuo on later today

[dmoisset]

Added python/peps#1502 and python/peps#1503 ; With @viridia's note and @natelust work on examples, I think I can give a shot to the only other remaining item later today

[gvanrossum]

Thanks! Now waiting for @viridia to write his piece.

[gvanrossum]

Landed python/peps#1504. I'm trying to write a piece I can post with the announcement. I'll link it here when I'm ready for review.

[natelust]

@dmoisset I apologize I didn't get as much time as I thought I would to work on the tutorial paper I linked above. I am traveling tomorrow (prepping for travel in this era took up the time I thought I had) but will be available later in the week to keep working on it (and I'm going to squeeze in time now). You are welcome to take anything from that, or add something to it on your own if you wish.

Given my track record on not knowing the best approach to creating examples for this repo I have not opened up a pull request for one of the examples I do have made. If it is useful, you can find an example of using pattern matching to construct objects from rows in database queries here. This example also uses matching for signaling from a function call, somewhat similar to what you would see with returning stateful enums in rust (though they are just classes here). It is based on the chinook example database, and is an application that allows users to list all the artists in the db, or albums and tracts given by a given artist. There is a bit of boiler plate to support the signaling returns, getting, and working with the database, but the match syntax is the heart of it.

If this is the kind of thing that belongs here I can take any suggestions and open a pull request. If not perhaps it will be inspirational to someone here.

[gvanrossum]

This is out: https://mail.python.org/archives/list/python-dev@python.org/thread/LOXEATGFKLYODO5Y4JLSLAFXKIAMJVK5/

[gvanrossum]

Nate, I like your sql example and if you sent a PR I'd add it.

[natelust]

PR opened, thank you.