Deprecation of flatmap is foggy and debatable

Leon Timmermans directed my attention to the situation of flatmap. If you visit the documentation of it, you can see that Any.flatmap appears as "deprecated as of v6.d and to be removed in v6.e" while List.flatmap only appears as discouraged. The issue is about the current situation, process, intentions and reasons. There are two goals of this issue:

to reach a clear situation with flatmap
to potentially keep flatmap from being deprecated and removed

1. The current Rakudo behavior (Rakudo v2022.07 on MoarVM 2022.07)

raku -e '"asd".flatmap: { $_, "Lol" } andthen .say' # (asd Lol)
raku -e 'use v6.e.PREVIEW; "asd".flatmap: { $_, "Lol" } andthen .say' # (asd Lol)

As we can see, there are no hints of Any.flatmap being deprecated, let alone removed.

2. Roast

flatmap appears at two places in the Roast repository:

at the announcement of language release v6.d
at the nodality test of hyper metaoperator

The former is about Any.flatmap while the latter uses List.flatmap. By the way, the latter appearance predates the former by a lot so I already couldn't decide if the distinction was intentional or not. However, the former also seems to be rather low for a "footprint" of a language change. Is there a way to define removed features in Roast at all?

3. `Any.flatmap` vs `List.flatmap`

For this, I think we need to go back in time: RT archive for deprecation discussion about the hint in the docs.

What I see... or well, better said, what I don't see is a clear intent that explains the apparently different fate of Any.flatmap and List.flatmap. I barely even see awareness of these two distinct things in these discussions. I couldn't decide if they were supposed to have the same fate (and Zoffix just missed it for v6.d), or there is a reason to remove it from Any but not List.

For me personally, it seems an odd choice to keep it and kindly discourage it at one place and ditch it at another, generally less common place. Also, I think List methods that are defined for Any in order to support "listy" behavior for as many types as possible. Is this separation common that something is deliberately only defined for Lists? Anyway, here, I don't see a clear and trackable decision.

Anyway, I think there should be at least reassurance if that was indeed the intention (to keep List.flatmap and drop Any.flatmap).

4. Is the deprecation right?

I will refer to the former links again because that's all I know and have: RT archive for deprecation discussion about the hint in the docs.

As the question implies, I'm not sure about the (supposedly predetermined) answer. It could be that we lack proper archives. (I must be honest, it's emotionally damaging to read IRC logs from a certain era.) Eventually, I couldn't see any additional information on the deprecation of flatmap so I'm left with the same feelings as Leon was in 2018: it does seem like basically a handful of people cared and made benefit of others' apathy / lack of communication. This would also explain the highly incomplete procedure of the deprecation - those people who facilitated this change aren't really present/active anymore.

The way I understand the situation:

flatmap was originally highly regarded because of Perl 5 legacy (map behavior in Perl 5)
after the Great List Refactor, this behavior became less relevant and also easy to mimic using .map and .flat
mainly because of the naming scheme of deepmap and duckmap (and an additional documentation issue around flatmap), a rather limited set of people decided that flatmap does more harm than good
this wasn't properly communicated and proponents left anyway so we ended up in this foggy situation with semi-deprecated flatmap that is absolutely valid for contemporary Rakudo still

I think we need to resolve the situation in a consistent manner, preferably addressing all main points.

My stance:

I think the deprecation should be revoked (for both Any.flatmap and List.flatmap) for the following reasons:

I think the arguments made against flatmap misrepresented the weight of certain points:
- the feature was there and its presence was completely aligned with Raku's deliberate redundance; it would take serious arguments to deprecate and remove something that has no apparent, objective issues. Rendering a method as "yet another thing to maintain, yet another possible name collision etc." in a batteries included++ language seems way off.
- I don't think doing stuff differently from either Perl or any other language is a virtue by itself. The fact that flatmap is a familiar name for a familiar thing for many people coming from different languages does matter. It's not everything but it's definitely more than nothing.
- similarly, I think if this feature is commonly used enough across very different languages - and has its legitimate uses in Raku as well - that in itself justified its existence just enough to not be deliberately removed
- I find deepmap and especially duckmap very confusing and arguably less useful, so I don't empathize with the "what about those people who think flatmap is like deepmap"; exactly because of the very different use, I don't think that's necessarily the concern of a user of one of these things what the other thing does
- for the previous reasons, I completely disagree with the sentiment that presented people who have wrong assumptions about flatmap as "Perl 6 experts". Raku is regularly proven to be the language that one single person cannot use to the greatest extent because its size, and this is by design. A Raku expert doesn't have a reductionist approach to mistake an otherwise widely known paradigm to some peculiar Raku utility functions.
- the argument that you can do flat followed by map seems very weak to me
- I welcome Leon's sentiment that it's tidy to keep the callback at the end rather than chain an additional method call after
- doing subroutine style flat map ... with no parens might look clever but avoiding parens scales worse with composing expressions than even method chaining does. Trust me, as an andthen chain advocate following the footsteps of Jan Krňávek, I have experience with the gotchas it can cause.
- the flat .map(...) example was particularly ironic: not only Zoffix was confused by it but even right now, I thought "nope, that wouldn't work after allowing white spaces in method chaining". The participants talked a lot about expertise - I think I have decent knowledge of Raku by now and they provided an alternative that caught me even fully aware of the topic (while it also tripped Zoffix Znet at first)
- and anyway, a reiteration: this in itself surely wouldn't justify removing a fully working thing from Raku of all languages?
I'm generally against "silent disapproval" like the case of List.flatmap - since it's even in Roast, it seems a bit like mocking our own language (for what it's worth, I generally dislike "traps" as well, for the same reason)
since it was added/modified in Roast after the Great List Refactor, I think it's just authoritative enough. If it's a legitimate part of the language, what is the point of "outcasting" the similar method of Any that follows a usual pattern of supporting List features as long as something can be meaningfully coerced into a List?
as this deprecation hasn't been actually implemented in any way, it's not a damaging "change of plans" to revoke it

I find this deprecation entirely baffling. I find myself flatmapping quite often, and I've repeatedly seen use-cases in other people's code too.

I would also like to note that this method is present in several other modern languages such as Javascript, Rust, Kotlin and Scala (and probably many others); I'm genuinely puzzled by the suggestion that it's confusing when they don't find it confusing.

It's particularly annoying because the alternatives are all harder to read. @foo.map({ ... }).flat is hard to read because of end-weight issues, especially if the map block contains multiple lines. flat @foo.map({ ... }) contains a reversal of order (left to right for the method call and but right to left for function call), a break of the method chain and has subtle precedence issues. flatmap is usually the most readable option available in my experience.

But most of all I find the process here disheartening. It was removed with very little communication with the wider community, and even when objections were raised soon after these were not taken seriously; telling people a feature is "virtually useless" when several people have said they do find it useful rather feels like talking to a wall. This is the sort of feature that is either incredibly useful and obvious or entirely useless depending on one's programming style; it rather feels like the alternatives were proposed by people who weren't using this feature in the first place and hence didn't understand why the stated alternatives were dissatisfying.

I don't think .flatmap is actually deprecated in the core???

And it's used everywhere in the core?

@lizmat do I understand you correctly: this deprecation policy was never in actual effect for Rakudo-related content in the first place?

If that really is the case, I would say that's one less reason to stick to it...

What I'm saying is that using flatmap in your code never actually gave a DEPRECATED warning. And nobody reads documentation, so nobody knew about any flatmap deprecation.

So I think this is mostly a documentation issue.

See my issue headings 1 and 2. Seems like nobody reads issues... :)

As a sidenote: I don't think the sole fact that something hasn't been implemented in Rakudo is a decisive language design argument. It's clear that there has been intention to deprecate and remove flatmap. Therefore, you can say out of respect for the supposed design process, I presented this deprecation as a real but unimplemented part of the design.

So overall I agree with the conclusion despite disagreeing with the implications ("if it's not in Rakudo, it doesn't exist").

Raku / problem-solving