headless anonymous function (->) syntax

[rapus95]

Edit 3: Still sold on the original simple idea with optional extensions, as in the previous edits, or combined with https://github.com/JuliaLang/julia/pull/53946. Previous edits highlight and explore different ideas to stretch into, all adding their own value to different parts of the ecosystem. For a glimpse on the simple approach, have a look at the description after all the edited cross-references.

---

Edit 2: again newly progressed state of this proposal: https://github.com/JuliaLang/julia/issues/38713#issuecomment-1436118670

---

Edit 1: current state of this proposal: https://github.com/JuliaLang/julia/issues/38713#issuecomment-1188977419

---

Since https://github.com/JuliaLang/julia/pull/24990 stalls on the question of what the right amount of tight capturing is

Idea

I want to propose a headless -> variant which has the same scoping mechanics as (args...)-> but automatically collects all not-yet-captured underscores into an argument list. EDIT: Nesting will follow the same rules as variable shadowing, that is, the underscore binds to the tightest headless -> it can find.

Before	After
lfold((x,y)->x+2y, A)	lfold(->_+2_,A)
lfold((x,y)->sin(x)-cos(y), A)	lfold(->sin(_)-cos(_), A)
map(x->5x+2, A)	map(->5_+2,A)
map(x->f(x.a), A)	map(->f(_.a),A)

Advantage(s)

In small anonymous functions underscores as variables can increase the readability since they stand out a lot more than ordinary letters. For multiple argument cases like anonymous functions for reduce/lfold it can even save a decent amount of characters. Overall it reads very intuitively as start here and whatever arguments you get, just drop them into the slots from left to right

      -> ---| -----|
            V      V
lfold(->sin(_)-cos(_), A)

Sure, some more complex options like reordering ((x,y)->(y,x)), ellipsing ((x...)->x) and probably some other cases won't be possible but if everything would be possible in the headless variant we wouldn't have introduced the head in the first place.

Feasibility

1) Both, a leading -> and an _ as the right hand side (value side) error on 1.5 so that shouldn't be breaking. 2) Since it uses the well-defined scoping of the ordinary anonymous functions it should be easy to 2a) switch between both variants mentally 2b) reuse most of the current parser code and just extend it to collect/replace underscores

Compatibility with #24990

It shouldn't clash with the result of #24990 because that focuses more on ~tight single argument~ very tight argument cases. And even if you are in a situation where the headless -> consumes an underscore from #24990 unintentionally, it's enough to just put 2 more characters (->) in the right place to make that underscore once again standalone.

[StefanKarpinski]

After that long, inconclusive debate, I think I've also come to the conclusion that having an explicit marker is better. Headless -> is the "natural" marker for this, so there you have it — it's simple and unambiguous. It even leaves room for letting Array{_, 3} be a shorthand for Array{<:Any, 3}. We would have to decide what to do in cases like -> (_, Array{_, 3}), but I would argue that it would probably be best to just make that an error and not allow _ in type parameter position inside of a headless lambda, which would need to be decided when implementing this, since otherwise making it an error after this feature is implemented would be a breaking change.

[StefanKarpinski]

To clarify, the question is which of these -> (_, Array{_, 3}) would mean:

• x -> (x, Array{<:Any, 3})
• (x, y) -> (x, Array{y, 3})

Both could potentially make sense. My suggestion is to make it an error and force the user to either use a normal lambda or not use _ as a type parameter. Alternatively, we could say that _ always "binds" to the tightest thing it could. That's also a consideration in the presence of nested headless lambdas. For example, it seems only sensible to interpret -> (_, -> _ + _) as meaning x -> (x, (y, z) -> y + z), so you could make the same case for -> (_, Array{_, 3}) that the _ as an argument to Array should mean Array{<:Any} since it's innermost.

[yurivish]

Sure, some more complex options like reordering ((x,y)->(y,x))

One solution is to say that inside of a headless anonymous function _n refers to the nth argument. So (x, y) -> (y, x) would be written as

-> (_2, _1)

There is precedent for this in Clojure:

The function literal supports multiple arguments via %, %n, and %&.

#(println %1 %2 %3)

and Mathematica:

#n represents the n ^(th) argument.

In[1] := f[#2, #1] &[x, y]
Out[1] = f[y, x]

[rapus95]

Alternatively, we could say that _ always "binds" to the tightest thing it could.

that's exactly what I meant when saying

automatically collects all not-yet-captured underscores into an argument list.

so yes, I'd clearly be in favor of making them bind the tightest. Regarding the case of parametric underscore in headless lambda, I'd propose to already special case that in the parser (or wherever that belongs to 🙈) but make it error for now. That way we're free to add a proper rule once we've found a good solution without being breaking. Though, right now I'm thinking about the following idea: sugar	expanded
->(_, Array{_, 1})	x->(x, Array{<:Any, 1})
->(_, Array{<:_, 1})	(x,y)->(x, Array{<:y, 1})

for types the 2nd approach should work in the most cases since <:LeafType == LeafType for leaf types while one rarely needs actual abstract types as fixed parameters (and even then we have AbstractType <: AbstractType) The drawback is that this approach wouldn't work for bitstypes like integers since <:1 is undefined iirc. BUT! once we have something like Array{String,::Int} to denote the 2nd parameter to be an integer, we might be able to allow ->Array{String, _::T} as (x::T)->Array{String, x}

[rapus95]

regarding _1,_2 to denote the argument order still makes me have mixed feelings. On the one hand it reduces the readability since it's no more simply "put in from left to right" and on the other hand it doesn't save much anymore since now you need 2 characters to denote a single variable. a normal lambda would need 3 characters (definition, comma in argument list and usage)

[mcabbott]

The other advantage of numbering is that it lets you re-use the same argument:

sort(xs, by= -> _.re + _.im/100)  # x -> x.re + x.im/100
sort(xs, lt= -> _1.re < _2.re)    # (x,y) -> x.re < y.re

Edit -- You could argue that the first line only saves one character, and the second 3 not 5. But what it does still save is the need to invent a name for the variable.

Re-using the same symbol with different meanings in an expression seems confusing to me. Is it ever actually ambiguous? (The order of symbols in +(a,b) and a+b differ, with the same Expr, but _ will never be infix. So perhaps that cannot happen?)

Edit' -- Not the most convincing example, but notice that 1,2,3 occur in order in the lowered version of this, as Iterators.filter puts the condition before the argument, but the comprehension puts it afterwards:

dump(:(  [f(x,1) for x in g(z,3) if h(x,2)]  ), maxdepth=10)

[yurivish]

regarding _1,_2 to denote the argument order still makes me have mixed feelings. On the one hand it reduces the readability since it's no more simply "put in from left to right" and on the other hand it doesn't save much anymore since now you need 2 characters to denote a single variable. a normal lambda would need 3 characters (definition, comma in argument list and usage)

You can continue to use _ for that; the numbered underscores are just a syntax for referring to arguments by their position.

In Mathematica's and Clojure _ always refers to the first argument, and numbered underscores are usually used for the second/third/... arguments.

So @mcabbott's first example works as-is and the second example can also be written as

sort(xs, lt= -> _.re < _2.re)    # (x,y) -> x.re < y.re

[rapus95]

to be honest I'm absolutely against making the single underscore refer to the same argument. Because we'd lose the entire convenience for the multi argument cases only to save 1(!!) character in rare situations... ->_.re + _.im/100 vs x->x.re + x.im/100 that's just not worth it. So IMO each underscore should denote its own argument, from left to right. By that approach compared to @mcabbott 's suggestion we'd save 2 characters for the different argument case and only lose a single character for the same argument case. (which btw would currently be handled by the tight binding approach of #24990) I. e. whenever it feels like multiple underscores should denote the same element, just use an ordinary lambda (it'll only cost you a single character extra)

[rapus95]

the numbering approach won't be possible until 2.0 anyway since _2 etc are currently valid identifiers and thus that would be breaking AFAIU

[yurivish]

the numbering approach won't be possible until 2.0 anyway since _2 etc are currently valid identifiers and thus that would be breaking AFAIU

I think it wouldn't be breaking if _2 only means the second argument inside a headless anonymous function.

Good points re: just using an ordinary lambda. I'm curious what fraction of anonymous functions would be made shorter by the "headless" type. It'd be hard to measure accurately, since anonymous functions are used a lot in interactive non-package code.

[rapus95]

Re-using the same symbol with different meanings in an expression seems confusing to me

while certainly true for most characters, the underscore already has the meaning of "fill in whatever you get, I won't refer to it anywhere else" See underscore as left hand side where it's used to tell that we don't need the result; and as a parametric argument to denote "I don't care about the actual type" So I'd try to use that freedom to not to be bound to ordinary variable behavior. If I wanted ordinary behaviour for it, I could just use an ordinary variable 👀 if we were crazy we could just bind any variable which would otherwise result in an undefvarerror 😂 but I'm strongly against that.

[StefanKarpinski]

I'm in agreement with @rapus95 here: let's stick to the simple win with _ for positional arguments. Anything more complicated seems to me better expresses with named arguments.

[StefanKarpinski]

Oh, one more thing to consider: interaction with |>. With just the basic proposal here, one would often need to write something like this:

collection |> -> map(lowercase, _) |> -> filter(in(words), _)

We could either introduce a new pipe syntax as a shorthand for this, or say that |> also acts as a headless lambda delimiter in the absence of -> so if there are unbound _ in the expression following the |> then the -> is implicitly inserted before the |> for you. So you'd write the above like this instead:

collection |> map(lowercase, _) |> filter(in(words), _)

This does allow using another headless lambda for the filter/map operation, like this for example:

collection |> map(-> _ ^ 7, _) |> filter(-> _ % 3 == 0, _)

Might want to do something similar with <| for symmetry. It feels a little weird to single out these two, but I can't see anything more general that makes much sense to me.

[rapus95]

I love that idea tbh because it would give us some part of #24990 for "free". The only thing that holds me back is that, in that case applying the same rule to \circ probably make sense aswell and then it starts to feel like arbitrary special casing again...

Btw, since the headless approach is currently primarily developed around multiple argument cases, it'd be very nice if we had a syntactical solution for splatting. Otherwise cases like (1,2) |> (x,y)->x+y wont work anyway since we'd have to wrap the anonymous function into Base.splat

EDIT: could we use |>> as head that automatically splats Tuple? It already has that extra > which somewhat hints the -> and if it's made specifically for that, we could even include splatting since it would be a special operator the evolves around chaining anonymous functions.

[yurivish]

Could syntax like map(... -> sin(_) - cos(_), xy_tuples) work?

[bramtayl]

I think argument 1 could be 1 underscore (_), argument 2 could be 2 underscores (__), etc. This is how things are currently done in the queryverse. Ref https://www.queryverse.org/Query.jl/stable/experimental/#The-_-and-__-syntax-1 and https://github.com/JuliaLang/julia/pull/24990#issuecomment-431960201 and https://github.com/JuliaLang/julia/pull/24990#issuecomment-449446671

[rapus95]

@bramtayl having every single underscore reference the same single argument doesn't scale well for the headless syntax. Read https://github.com/JuliaLang/julia/issues/38713#issuecomment-739422316 for why. and needing the same argument in multiple places is also a comparably rare case.

[bramtayl]

Hmm, needing to use the same argument in multiple places happens all the time in querying though, I think. Consider processing a row of a table: you might need to reference several fields of the row.

[rapus95]

how many characters more would you need if you switch to using an ordinary lambda with a single letter variable name instead? (remember that you need the -> in any case)

[bramtayl]

The extra _ for the second argument seems to me to be a very small price to pay for the flexibility of using an argument as many times as you want, is all

[rapus95]

a) but it doesn't scale into more different arguments b) it's a huge price! we waste an entire syntax to save a single character compared to an ordinary lambda. You will never be able to save more than a single character if multiple underscores denote the same argument.

[bramtayl]

It seems to me likely that wanting to make a two argument anonymous function will be much less common than wanting to reference an argument more than once

[rapus95]

@bramtayl do the maths yourself. it scales very bad (i.e.negatively) if you intend on using any other than the first argument more than once or if you intend to use more than 3 arguments so, multiple chained underscores just don't benefit us for the general purpose case.

_i for i being a single digit number would still be a better proposal for that case, both for scaling in number of uses per argument and number of arguments. But that can live in its own issue since it is orthogonal to the current proposal

[bramtayl]

_i would definitely work too. I suppose I could go through some code and count the number of times you have different numbers of arguments in anonymous functions and the number of times you reuse an argument. Even though I suspect that reusing an argument will be far more common, it doesn't matter too much, because I think it would be nice to have a syntax flexible enough to do both. Do you know of any use-cases for three-argument anonymous functions?

[rapus95]

just include a lot of code samples which use reduce and similar functions. If you only include code that maps data it would be an unfair comparison. But this already shows what I'm talking about. We don't want a domain specific syntax feature in the general purpose language. And the queryverse definitively is domain specific. And it already has a macro for that exact case. Which doesn't seem like it made it outside of that domain.

[goretkin]

the underscore already has the meaning of "fill in whatever you get, I won't refer to it anywhere else" See underscore as left hand side where it's used to tell that we don't need the result; and as a parametric argument to denote "I don't care about the actual type"

I think more fundamentally, the principle behind _ is something like "I want to avoid giving an arbitrary name to a value". As of now, you can avoid giving an arbitrary name if

• you don't need to access the value that would be named arbitrarily

  (_, r) = divrem(1,2)

foo(_) = 3
foo(nothing)

• (less famously) there is only one field in the struct, which is perfect for "wrapper" types

  
  struct Foo
  _::Int64
  end

Foo(3)._

Note this usage is a counterexample to `_` meaning "I won't refer to it". (xref: https://github.com/JuliaLang/julia/issues/37381)

It seems (at least) roughly consistent with this principle that `Array{_, 3}` mean `Array{<:Any, 3}`, which is `Array{var"#s27", 3} where var"#s27"`, where the need for an arbitrary name is fulfilled by something like `gensym`.

The use of `_` as an arbitrary name for a type parameter, combined with the proposal here, leads to a [possible ambiguity already mentioned](https://github.com/JuliaLang/julia/issues/38713#issuecomment-739341063). To be sure, consider the interaction with `_` as a field name: 

`x->x._` could be written as `->_._`. I do not think there is any chance for ambiguity here, the same way that there is no chance for ambiguity with `x->x.x`.

I agree that it would be more useful if each `_` in the anonymous body referred to a subsequent argument, as opposed to the alternative that `_` always refers to the first argument. I'm not sure if this choice can be justified as a natural consequence of `_` meaning "avoid arbitrary name", and that the alternative cannot be, but it kind of feels that way. Certainly if `_` is used as a type parameter, each `_` would be a unique parameter.

[bramtayl]

Hmm, well, a quick audit of non-single-argument-0-or-1-mention uses of -> in julia/base, excluding splats, is below. A couple of notes:

• 3 argument anonymous functions seem extremely rare; I didn't see any 4 argument ones.
• At least from this sample, reusing an argument in an anonymous function is about as common as having multiple arguments.
• I did this relatively quickly, so I probably missed some

Multiple arguments

sum(map((i, s, o)->s*(i-o), J, strides(x), Tuple(first(CartesianIndices(x)))))*elsize(x)
foldr((v, a) -> prepend!(a, v), iter, init=a)
(r,args) -> (r.x = f(args...))
(i,args) -> (itr.results[i]=itr.f(args...))
((p, q) -> p | ~q))
((p, q) -> ~p | q))
((p, q) -> ~xor(p, q)))
((p, q) -> ~p & q))
((p, q) -> p & ~q)))
map((rng, offset)->rng .+ offset, I.indices, Tuple(j))
dict_with_eltype((K, V) -> Dict{K, V}, kv, eltype(kv))
foldl((x1,x2)->:($x1 || ($expr == $x2)), values[2:end]; init=:($expr == $(values[1])))
retry(http_get, check=(s,e)->e.status == "503")(url)
retry(read, check=(s,e)->isa(e, IOError))(io, 128; all=false)
dict_with_eltype((K, V) -> IdDict{K, V}, kv, eltype(kv))
CartesianIndices(map((i,j) -> i:j, Tuple(I), Tuple(J)))
CartesianIndices(map((i,s,j) -> i:s:j, Tuple(I), Tuple(S), Tuple(J)))
map((isrc, idest)->first(isrc)-first(idest), indssrc, indsdest)
(x,y)->isless(x[2],y[2])
(x, y) -> lt(by(x), by(y))
(io, linestart, idx) -> (print(io, idx > 0 ? lpad(cst[idx], nd+1)
(mod, t) -> (print(rpad(string(mod) * "  ", $maxlen + 3, "─"));
(f, x) -> f(x)
(f, x) -> wait(Threads.@spawn f(x))
afoldl((ys, x) -> f(x) ? (ys..., x) : ys, (), xs...)
Base.dict_with_eltype((K, V) -> WeakKeyDict{K, V}, kv, eltype(kv))
simple_walk(compact, lifted_val, (pi, idx)->true)
(io::IO, indent::String, idx::Int) -> printer(io, indent, idx > 0 ? code.codelocs[idx] : typemin(Int32))

Reuses an argument

dst::typeof(I) = ntuple(i-> _findin(I[i], i < n ? (1:sz[i]) : (1:s)), n)::typeof(I)
src::typeof(I) = ntuple(i-> I[i][_findin(I[i], i < n ? (1:sz[i]) : (1:s))], n)::typeof(I)
CartesianIndices(ntuple(k -> firstindex(A,k):firstindex(A,k)-1+@inbounds(halfsz[k]), Val{N}()))
CartesianIndices(ntuple(k -> k == dims[1] ? (mid:mid) : (firstindex(A,k):lastindex(A,k)), Val{N}()))
all(d->idxs[d]==first(tailinds[d]),1:i-1)
map(x->string("args_tuple: ", x, ", element_val: ", x[1], ", task: ", tskoid()), input)
foreach(x -> (batch_refs[x[1]].x = x[2]), enumerate(results))
map(v -> Symbol(v[1]) => v[2], split.(tag_fields, "+"))
findlast(frame -> !frame.from_c && frame.func === :eval, bt)
ntuple(n -> convert(fieldtype(T, n), x[n]), Val(N))
map(chi -> (chi.filename, chi.mtime), includes)
filter(x -> !(x === empty_sym || '#' in string(x)), slotnames[(kwli.nargs + 1):end])
ntuple(i -> i == dims ? UnitRange(1, last(r[i]) - 1) : UnitRange(r[i]), N)
ntuple(i -> i == dims ? UnitRange(2, last(r[i])) : UnitRange(r[i]), N)
map(n->getfield(sym_in(n, bn) ? b : a, n), names)
filter!(x->!isempty(x) && x!=".", parts)
all(map(d->iperm[perm[d]]==d, 1:N))
ntuple(i -> i == k ? 1 : size(A, i), nd)
ntuple(i -> i == k ? Colon() : idx[i], nd)
map(x->x isa Integer ? UInt64(x) : String(x), pre)
map(x->x isa Integer ? UInt64(x) : String(x), bld))
_any(t -> !isa(t, DataType) || !(t <: Tuple) || !isknownlength(t), utis)
 _all(i->at.val[i] isa fieldtype(t, i), 1:n)
filter(ssa->!isa(ssa, SSAValue) || !(ssa.id in intermediaries), useexpr.args[(6+nccallargs):end])
findfirst(i->last_stack[i] != stack[i], 1:x)
 x -> (x = new_nodes_info[x]; (x.pos, x.attach_after))
filter(p->p != 0 && !(p in bb_defs), cfg.blocks[bb].preds)
filter(ex -> !(isa(ex, LineNumberNode) || isexpr(ex, :line)), ex.args)

[rapus95]

there is only one field in the struct, which is perfect for "wrapper" types

@goretkin that case is perfectly handled by interpreting it as 0-d and using x[] (as Ref does) the "whatever comes I won't refer to it" otoh was the only reason why underscore became reserved. That's the reason why it must not be used as right hand side.

@bramtayl would you be willing to translate these cases into both (or even better all 3 variants) i. e. _ _ _ __ _1 _2 and measure the number of characters saved, compared to the ordinary lambda?

[mcabbott]

Thanks for gathering these, @bramtayl.

In the "multiple arguments" list, it looks like 12/28 don't follow the simple pattern of using every argument, exactly once, in order, and not as a type parameter.

2 of those simply drop trailing arguments, (x, _...) -> stuff, which raises the question (not so-far discussed?) of whether these headless lambdas should in general accept more arguments than they use, or not. Should map(->nothing, xs) work?

Details

Drop last:

simple_walk(compact, lifted_val, (pi, idx)->true)
(mod, t) -> (print(rpad(string(mod) * "  ", $maxlen + 3, "─"));

Drop first or middle:

retry(http_get, check=(s,e)->e.status == "503")(url)
retry(read, check=(s,e)->isa(e, IOError))(io, 128; all=false)
(io, linestart, idx) -> (print(io, idx > 0 ? lpad(cst[idx], nd+1)

Shuffle:

sum(map((i, s, o)->s*(i-o), J, strides(x), Tuple(first(CartesianIndices(x)))))*elsize(x)

Re-use:

afoldl((ys, x) -> f(x) ? (ys..., x) : ys, (), xs...)
foldr((v, a) -> prepend!(a, v), iter, init=a)
(io::IO, indent::String, idx::Int) -> printer(io, indent, idx > 0 ? code.codelocs[idx] : typemin(Int32))

Type parameters:

dict_with_eltype((K, V) -> Dict{K, V}, kv, eltype(kv))
dict_with_eltype((K, V) -> IdDict{K, V}, kv, eltype(kv))
Base.dict_with_eltype((K, V) -> WeakKeyDict{K, V}, kv, eltype(kv))

Simple cases (every parameter used exactly once, in order) [Edit -- now with some brackets removed]:

(r,args) -> (r.x = f(args...))
(i,args) -> (itr.results[i]=itr.f(args...))
((p, q) -> p | ~q )
((p, q) -> ~p | q )
((p, q) -> ~xor(p, q))
((p, q) -> ~p & q)
((p, q) -> p & ~q)
map((rng, offset)->rng .+ offset, I.indices, Tuple(j))
foldl((x1,x2)->:($x1 || ($expr == $x2)), values[2:end]; init=:($expr == $(values[1])))
CartesianIndices(map((i,j) -> i:j, Tuple(I), Tuple(J)))
CartesianIndices(map((i,s,j) -> i:s:j, Tuple(I), Tuple(S), Tuple(J)))
map((isrc, idest)->first(isrc)-first(idest), indssrc, indsdest)
(x,y)->isless(x[2],y[2])
(x, y) -> lt(by(x), by(y))
(f, x) -> f(x)
(f, x) -> wait(Threads.@spawn f(x))

... which could become (with each `_` a new argument)

-> (_.x = f(_...))
-> (itr.results[_]=itr.f(_...))
(-> _ | ~_ )
(-> ~_ | _ )
(-> ~xor(_, _))
(-> ~_ & _ )
(-> _ & ~_ )
map(->_ .+ _, I.indices, Tuple(j))
foldl(->:($_ || ($expr == $_)), values[2:end]; init=:($expr == $(values[1])))
CartesianIndices(map(-> _:_, Tuple(I), Tuple(J)))
CartesianIndices(map(-> _:_:_, Tuple(I), Tuple(S), Tuple(J)))
map(->first(_)-first(_), indssrc, indsdest)
->isless(_[2],_[2])
-> lt(by(_), by(_))
-> _(_)
-> wait(Threads.@spawn _(_))

Interesting how few from either list above would be clearer (IMO) without naming variables -- most are quite long & complicated. So another possibility to consider is that this headless -> syntax could be restricted to zero or one arguments [Edit -- I think I meant to say, "used at most once"], at least initially. To emphasise that it's for writing short things, where clarity may be improved by not having to name the variable.

I'm not sure that counting characters saved is a great measure, as the cases where you could save the most letters also seem like the ones complicated enough that you ought to be explicit. [Nor is counting how many cases in Base, really.] But using |> as a fence seems neat (it's visually -> with the minus rotated, right?) and means that some one-argument cases could become quite a bit shorter & less cluttered. For example you don't have to think about whether it's confusing to re-use the same name for vs & xs here:

[rand(Int8,5) for _ in 1:7] |> vs -> reduce(vcat, vs) |> xs -> filter(x -> x%3 != 0, xs)

collection |> reduce(vcat, _) |> filter(-> _%3 != 0, _)

[rapus95]

tbh I just find some of the spacings hard to read like here: -> ~_ | _ but I guess that has to do with not being used to unary operators in front of an underscore 😄 that'll come with time. Btw, you have more closing than opening parantheses 🙈 Regarding the readability (especially with p and q) I find the underscore variant a lot more readable since I know that I can rely on a) where does the closure manifest and b) in which order the arguments are filled in both in a way where I don't have to scan forward and backward but simply from left to right, once.

I intended the focus to be on >1 arg cases since going from 1 to 2 args costs a lot more characters and thus, focusing on single argument cases just doesn't bring profit at all (except that single saved character). But on 2-arg 1 use each cases the proposal indeed shines IMO. whether more arguments should use explicit naming is a matter of taste and situation I guess.

[goretkin]

there is only one field in the struct, which is perfect for "wrapper" types

@goretkin that case is perfectly handled by interpreting it as 0-d and using x[] (as Ref does) the "whatever comes I won't refer to it" otoh was the only reason why underscore became reserved. That's the reason why it must not be used as right hand side.

@rapus95, I disagree that that is perfect, especially when you would like getindex to have another meaning than "access my one field", which is the case for all of these:

julia> allsubtypes(T) = (s = subtypes(T); union(s, (allsubtypes(K) for K in s)...)) # https://discourse.julialang.org/t/reduce-on-recursive-function/24511/5?u=goretkin
allsubtypes (generic function with 1 method)

julia> allsubtypes(AbstractArray) |> x->filter(!isabstracttype, x) |> x->filter(T->1==fieldcount(T), x) |> x -> map(T->(T, only(fieldnames(T))), x)
25-element Vector{Tuple{UnionAll, Symbol}}:
 (Adjoint, :parent)
 (Base.SCartesianIndices2, :indices2)
 (CartesianIndices, :indices)
 (Core.Compiler.LinearIndices, :indices)
 (Diagonal, :diag)
 (LinearIndices, :indices)
 (PermutedDimsArray, :parent)
 (SuiteSparse.CHOLMOD.FactorComponent, :F)
 (Test.GenericArray, :a)
 (Transpose, :parent)
 (UpperHessenberg, :data)
 (Base.IdentityUnitRange, :indices)
 (Base.OneTo, :stop)
 (Base.Slice, :indices)
 (Core.Compiler.IdentityUnitRange, :indices)
 (Core.Compiler.OneTo, :stop)
 (Core.Compiler.Slice, :indices)
 (Base.CodeUnits, :s)
 (Base.Experimental.Const, :a)
 (SuiteSparse.CHOLMOD.Dense, :ptr)
 (LowerTriangular, :data)
 (UnitLowerTriangular, :data)
 (UnitUpperTriangular, :data)
 (UpperTriangular, :data)
 (SuiteSparse.CHOLMOD.Sparse, :ptr)

I am not saying that all of those field names are arbitrary. stop evokes a useful meaning. data (and to a lesser extent parent) does not. The name ptr does not convey any additional information to the type Ptr{...}.

I am not sure if you were just objecting to my characterization of _, or if you think using _ as a field name is bad. To emphasize a previous point, using it as a field name does not interfere with the proposal here.

[StefanKarpinski]

Just eyeballing it, the lambdas where an argument is referred to multiple times tend to be relatively long and complex and often have fairly long argument names, whereas the lambdas where there are multiple arguments that are each used once in order tend to be relatively short and simple. Since the aim of this feature is to make it easier to write short, simple lambdas, this would seem to favor the originally proposed behavior.

[bramtayl]

Ok, I can see that. How about another argument: having _ mean two different things right next to each other violates the principle of least surprise?

[StefanKarpinski]

Another way of looking at it is this: why is it so annoying to give names for x and y in a lambda like (x, y) -> x[y]? It's not really that much typing. The issue is really that it shifts the focus both when writing and reading the code to these meaningless x and y variables and they are not the point — the key thing is the indexing action. It's annoying when writing it because you have to come up with names that don't matter and it's annoying when reading it because there's so much noise that gets in the way of the action which is what you're trying to express. Using "wildcard names" and writing _[_] instead puts the focus where it belongs — on the indexing as an action. So being able to write _[_] is not only shorter, but also clearer: it's immediately clear that the thing that's being expressed is the action of indexing.

How does that interact with wether _[_] should mean (x, y) -> x[y] or x -> x[x]? Well, the former is the generic act of indexing, naturally a two-argument action. The latter is the action of self-indexing, which, is a pretty unusual and niche action. With the alternate proposal, how would one express "the action of indexing"? You'd have to write _1[_2]... which, is imo firmly back into the territory of letting the arguments dominate; I'd even argue that writing (x, y) -> x[y] is clearer.

Let's take another example: (_ + _) / _. What should it mean? In the original proposal, it means (x, y, z) -> (x + y) / z, and captures the action of adding two things and dividing them by a third thing. In the alternate proposal, it means x -> (x + x) / x and captures the (bizarre and useless) action of adding a thing to itself and then dividing by that same thing. This example may seem silly, but I think the key insight is that the real power and benefit of headless lambdas is that it allows you to write the syntax of an action with "holes" where the arguments go and capture the "essence" of that action in a way that syntactically focuses on the action instead of the names.

[bramtayl]

Hmm, well, still not convinced. One final argument: there was an argument above that querying tables is a niche application, but I'd argue it's far and away the most common use of numerical programming. If you combine all the users of SQL, LINQ, dplyr, Query.jl, and DataFramesMeta.jl, you'd get half of stackoverflow IMHO. And not being able to refer to a row more than once is a deal-breaker for querying. A prototypical example would be c = _.a + _.b.

In either case, I think conservatively making a headless -> with zero or one _ after work and integrate with |> would be a great improvement.

[StefanKarpinski]

The thing is that people who want to write row queries don't even want to write the _. — they just want the row to object to be fully implicit and just write a + b. The syntax :a + :b is already a de facto standard among Julia packages for row operations and allows not mentioning the row object. It seems like a step backwards for them to write _.a + _.b instead.

[MasonProtter]

I just can't help but feel that requiring a glyph at the beginning of the expression defeats the purpose.

-> foo(_, 1)

is literally the same number of characters in the same positions as

@_ foo(_, 1)

but requires new parser support, rather than a simple macro definition. What's the point?

The reason people complain about @_ is that they don't want to backtrack after they've already written foo. I really think that a glyph at the end of the expression would be a better solution to this problem. e.g.

foo(_, 1) _@

I've proposed foo bar@ as postfix macrocall syntax equivalent to @bar foo before, but that offended many people's sensibilities. Perhaps we can instead agree on a single postfix glyph specifically for this problem, rather than allowing all macros to be used in postfix? I still maintain though that postfix macros would be a more general solution.

[bramtayl]

I think the :a syntax for getting a field from row is confusing because it is the same syntax for Symbol("a"). I think _.a + _.b is clearer and just as convenient (and it's what Query uses). I don't think dot overloading and named tuples even existed when DataFramesMeta was written. To a certain extent, a proposal like this could even make the macro processing side of Query obsolete, or to put it another way, it would allow the entire ecosystem to benefit from the syntax.

[bramtayl]

I suppose I should also mention that the proposal here is very similar to the @_ and @> macros in LightQuery, which is part of why I'm so excited about it. I had to use some hacks to get the macros to resolve inside out (like functions) rather than outside in (like macros). It would be a lot nicer if they were officially supported as function syntax.

[rapus95]

I think the :a syntax for getting a field from row is confusing because it is the same syntax for Symbol("a"). I think _.a + _.b is clearer and just as convenient (and it's what Query uses).

:a being the same as Symbol("a") is the whole intent here. Just as a slight reminder: _.a == getproperty(_, :a) so yes, that's a Symbol. :name is a widespread way to index by column name. And since these are ordinary objects in Julia they take part in everything else, like grouping/broadcasting: getindex.(obj, [:a, :b, :c]) so having Symbols as selectors is a good thing anyway. And macros deliver the domain specific convenience.

To a certain extent, a proposal like this could even make the macro processing side of Query obsolete, or to put it another way, it would allow the entire ecosystem to benefit from the syntax.

Of course, if we embed domain specific features into the Base language, then that makes the domain specific features obsolete. Only drawback: We arrive at a domain specific language which excels in its domain but feels bulkloaded for everything else (i.e. a lot of features that aren't useful in generic cases). And tbh I don't want Julia to be a domain specific language.

[clarkevans]

I'm not convinced with the general usefulness of this construct vs making code less readable (and maintainable) -- this is an additional syntax burden on "accidental programmers" where using Julia is not their primary job. I think the wide-open interpretation of _[_] is a perfect example of why this is not a particularly great idea? How about an off-the-wall suggestion -- why not use bare subscripts? -> ₁[₁] at least this one is a bit unambiguous, (x₁,x₂,...) -> x₁[x₁], then -> ₂[₁] could mean (x₁,x₂,...) -> x₂[x₁]. I'd worry less on how easy it is to type than on how easy it is to read ~18 months later when you're fixing a bug. If you wish to keep _ for the 1 argument case, then we could use subscripts to access the nth item in the input... -> _₂[_₁] could mean (x₁,x₂,...) -> x₂[x₁].

[xitology]

Have you considered using broadcasting instead? That is, lift f.(_) to _ -> f(_). Add special sugar for getproperty and getkey: _.x = _ -> _.x, and then _.x .> _.y will be translated to _ -> ((_ -> _.x)(_) > (_ -> _.y)(_)), which can be used as a predicate function.

Perhaps it could even support multiple arguments, by defining _(xs...) = _(xs).

Pipelining with |> could be supported by introducing one-argument curried forms of map() and filter():

1:10 |> filter(isodd.(_)) |> map(_ .* 2) |> foldl(_[1] .+ _[2])

[rapus95]

making code less readable (and maintainable)

In my opinion neither of that is true since the use case for that new syntax is mostly compact multiple argument anonymous functions which are needed to partially apply arguments (and, since it's syntactical, in the future, it might even resolve into something more elaborate like a generalization of the Fix1/Fix2 types including actual partial pre-optimization). In ->g(_, a, h(_), _, f) as opposed to (x,y,z)->g(x,a,h(y),z,f) I, to be honest, find the first case way more readable. Regarding maintainability, I don't see where that would need any maintenance.

Regarding your proposal of using subscript numbers for indexing, I really like the idea for more complex cases, but it's an orthogonal proposal since they don't conflict and are indeed more cumbersome to type for very straight forward cases like my example above.

Have you considered using broadcasting instead?

That's not an option since we want ->f.(_) to mean x->f.(x)

1:10 |> filter(isodd.(_))

with your proposal that would error since it would resolve to 1:10 |> x->filter(isodd(x)) which in turn would be the same as filter(isodd(1:10))

[bramtayl]

@clarkevans I'm not sure bare subscripts could work until post 1.0 because they don't currently error I think, but I do like the idea of _₁ _₂ etc especially if it would make it clearer what's going on. EDIT nevermind

[rapus95]

@bramtayl they do error:

julia> ₂=2
ERROR: syntax: invalid character "₂" near column 1
Stacktrace:
 [1] top-level scope at none:1

I had to check that myself though, since I wanted to write the same as you 😂

[xitology]

1:10 |> filter(isodd.(_))

with your proposal that would error since it would resolve to 1:10 |> x->filter(isodd(x)) which in turn would be the same as filter(isodd(1:10))

It works, here's the proof of concept (using it instead of _): https://gist.github.com/xitology/347e01ed012e9e2a5530c719b7163500

[mcabbott]

@xitology If you haven't seen it, https://github.com/tkf/UnderscoreOh.jl uses broadcast like this, as a marker of where the function stops.

@rapus95 I think filter(isodd.(_), 1:10) resolves to filter(x -> isodd(x), 1:10), not to x -> filter(isodd(x), 1:10), in this proposal. The end of the broadcasting tells it where to stop.

That said, it does seem pretty confusing to re-use broadcasting for something unrelated, and will mean you can't use actual broadcasting at the same time.

[rapus95]

@xitology a) still, we don't want to lose the broadcasting ability to allow for a new syntax b) in your example you didn't have the same scoping mechanics as we are discussing here (i. e. 1:10 |> filter(isodd.(_)) would resolve to 1:10 |> ->filter(isodd.(_)) which in turn would resolve to 1:10 |> x->filter(isodd(x)) according to you (that in turn would error). for yours to work you would need an inner left barrier. i. e. something like that: 1:10 |> filter(->isodd.(_)) then that would work, but I don't yet get where the benefit would be to use broadcasting for it? especially since we'd lose actual broadcasting.

and please consider using some other example since your example is written the shortest and most concise by not using any special syntax at all 1:10 |> filter(isodd) 🙈

[xitology]

@mcabbott Thanks, I wasn't aware of it, but it's not surprising that someone did it before. Extending functions to combinators is a natural application of broadcasting.

@rapus95 The example works, see line 35 in the proof of concept https://gist.github.com/xitology/347e01ed012e9e2a5530c719b7163500#file-it-jl-L35.

Broadcasting is designed to work with arbitrary containers and fits very naturally to this particular case. We just need consider a function as a "container" where the argument of the function plays the role of the key. Then broadcasting a function over a functional argument is reduced to a function composition. Taking identity as the starting point, we can convert any broadcasted expression of a function.

[rapus95]

@mcabbott oh thanks, now I understand the proposal. The idea is to use the broadcasting dot as the barrier instead of ->. Well then, only my first point still holds. Mixing broadcasting and anonymous function definition feels like it has only drawbacks compared to using -> as the barrier. (regarding feature orthogonality and feature compatibility)

so, f.(_.(_)) would resolve to x->f(y->x(y)) right? and f.(_(_)) would resolve to (x, y)->f(x(y)). So far that looks good, but it isn't compatible with broadcasting which is a showstopper IMO map(->_.+_, A, B) wouldn't be possible with that proposal. If we use broadcasting we'd have to actually give underscores a value wouldn't we?