Closed ljharb closed 2 months ago
See the README:
do we support iterators and iterables like
Iterator.from
andflatMap
?
Arrays are iterable, so they would "just work" if we answer "yes" to this question. Also, even if we answer "no", it's pretty easy to turn arrays into iterators with .values()
or Iterator.from()
.
so they would "just work" if we answer "yes" to this question
Well, you'd get an iterator out rather than an array, presumably? So you'd need to .toArray
at the end. Iterator.zip(x, y).toArray()
is a bit more awkward than Array.zip(x, y)
.
What I mean is, why not have these methods also on Array/Array.prototype?
We could do that, but I think the Array
helper is a lot less motivated if we have a good Iterator
helper. It's hard and complicated to write your own zip
. It's not hard or complicated to write Iterator.zip(a, b).toArray()
. And I imagine a good portion of invocations on arrays won't even need to realise an array as the output anyway. new Map(Iterator.zip(keys, values))
or for (let [k, v] of Iterator.zip(keys, values)) ...
will work fine for array inputs. For the remainder of cases, calling .toArray()
doesn't seem like much of a burden to me.
Most of my need for zip starts with two arrays and ends with one array, and it seems useful to me to avoid the iterator protocol entirely when it’s not needed.
Having it on Array prototype would probably allow various optimizations in the JS engine, which might be impossible with the generic iterator version.
I wouldn't believe such claims unless they came directly from the implementors. The engines already keep taint bits on things like Symbol.iterator on built-ins and then short-circuit these kinds of operations to optimise when passed arrays or other built-ins. I don't see why they wouldn't do the same here.
I don't see why they wouldn't do the same here.
I thought it's impossible to know beforehand that the developer wants to produce an array in the end so the engine would have to speculate, which is wasteful.
Closing this as we've taken the Iterator.from
approach of accepting iterators/iterables, which includes arrays.
@michaelficarra i don't think that addresses the ergonomics issue nor precludes also adding the method on Arrays.
@ljharb You're right, this proposal does not preclude another proposal that would add a similar static method on Array
. I don't think such a method should be included in this proposal, though.
Why not? This issue shouldn’t be closed until that original question is answered.
I agree that it should be added to Array
too. Extra .toArray()
is too ugly for simple cases.
Mozilla has already said they'll reject anything adding to Array.prototype
, so a method isn't happening. And Iterator.zip(array1, array2)
already works. What's the end goal of this discussion?
@jridgewell that was not my interpretation of what they said; it was that new methods on Array.prototype would require a stronger motivation.
Array.zip
would produce an array, which is the whole point - may programs don't want to work with the iterator protocol at all.
Can you give an example of a time when you'd want an array as the result? In my experience, zip
is almost exclusively used as a step in a chain: for (let [a, b] of zip(x, y))
, or Object.fromEntries(zip(keys, values))
, or whatever. Wanting an array seems like an unusual case which would need quite strong motivation, given that Iterator.zip(a, b).toArray()
is not exactly hard to write.
I don't have a concrete example off hand - i'll certainly try to find one - but i definitely use "zip" at times as not a step in a chain (where "a chain" means "within the same function/scope") - and i want to pass around arrays, not iterators, basically 100% of the time. I only use iterators as part of an inline transformation.
I don't know how much there is to lose by adding Array.zip
(or such) alongside Iterator.zip
, except that perhaps it would be nicer to have syntax like Array.prototype.zip
, which is obviously a no-go. I agree with @ljharb about the case for transferring between functions or contexts.
IMO Iterator.zip
expresses something that is more complicated than Array.zip
: "I would like to start an altogether new context of iterating over multiple things by zipping together these iterables." If you're creating that context (i.e. an iterator) out of an array, only to turn it right back into an array, then Iterator.zip
is behavioral noise and Array.zip
is much more to the point.
Of course, Iterator.zip
is appropriate to use in for..of
or alongside any others of the iterator helpers, where you might break early and not need all the results, or otherwise want to express a complex operation without immediately computing it. That's an obvious and widely applicable use for iterators in general, which Iterator.zip
lends perfectly to, but it's not the only use for zipping stuff together.
If you're creating that context (i.e. an iterator) out of an array, only to turn it right back into an array, then
Iterator.zip
is behavioral noise andArray.zip
is much more to the point.
This is a big "if". I agree with @bakkot that the zip result is almost always used as an intermediate result. If that's true, we should not make it as easy to use the version that realises an array. It's easy enough to do Iterator.zip(...).toArray()
and better (in my opinion) that it calls out the eagerness. And I'm not concerned about the efficiency of iterating over an array as input, as engines can certainly optimise this case.
we should not make it as easy to use the version that realises an array
Looks like a very heavy-handed way of helping the programmers. According to this logic there should be no Array.prototype.map, filter, and other established methods because they are often chained and produce an unnecessary array. And although I sometimes avoid using these methods for this exact reason, it doesn't mean I agree that this functionality shouldn't be present in the language as it's useful in case where performance/memory/GC is not a concern.
I don't know if that's what @michaelficarra is getting at. It sort of feels like the opposite, i.e. optimization isn't a concern here and we are talking much more about expression rather than performance (I interpret "engines can certainly optimise this case" as saying "Iterator.zip(...).toArray()
can be trivially detected and flattened into a single operation that doesn't actually perform a second pass of iteration in toArray
" — but I could be misinterpreting of course).
I don't understand what "better that it calls out the eagerness" means. Are you indicating that being forced to use toArray()
is a way of explicitly saying, "OK, I really do need all the stuff from this zip operation as an array now, I know that's a bit weird so I'd better have some darn good justification"?
If that's the case, then I kind of agree with @tophf that it's heavy-handed. I don't think it's a good idea to consider iterator operations as the absolute replacement for array operations - I don't want to accuse you of that, mind! but if it's in line with your thinking, then it feels like the wrong angle to me. We have array methods and yes, those are a legacy inclusion because they've been around forever, and yes, there are a ton of cases where iterators are more appropriate and better.
But array methods still exist, and I feel as though introducing a behavior for iterators without also bringing along the direct analogue in arrays would be a missed opportunity. I feel the onus should be on the developer to learn and identify whether arrays or iterators are more appropriate for their use case, including the skill to effectively identify not just "if iterators are better than arrays", but in what specific instances their code would benefit from using each.
I don't think the spec should be arbitrarily lending preference to one choice or the other if there is not an extremely clear reason that zipping is fundamentally an operation that does not make sense for arrays.
The criteria for adding something is not "there is no identifiable reason that this operation does not make sense here". It's "this operation makes sense and is a common enough need to warrant adding something to the language, rather than expecting programmers to use the next best option".
And I'm just not convinced that it is actually particularly common to need zip
on arrays specifically, rather than on iterators. I went to check and every place I could find that I've used zip
in the last couple of years has been one-shot. I'm not strongly opposed, and I'm open to being convinced, but it would help to have concrete examples of cases where you need this.
There's historical ~parity~integration1 between Array and Iterator methods, so a much stronger reason is necessary to break it and not to introduce the new method on both.
[1]: iteration protocol is already seamlessly integrated in array syntax like [...arr1, ...arr2]
, we don't have to explicitly convert arrays to iterators in for-of, and there's even a Stage-3 proposal to add explicit array-like methods on iterators.
(edit: Just to clarify, all my comments are made in the context of the Stage 3 Iterator Helpers proposal, which I was taking as common knowledge since this is also an iterator helper, just a new kind of operation rather than an equivalent to an existing array helper.)
I agree with @tophf's point above but I think it would help to have a lot more examples shared anyway. We should challenge and try to justify the historical parity going forward, rather than take it as a (functionally weaker) given, too!
Real-world use cases:
zip
would often be the first statement in a chain of such operations, eventually resolving into an array that I operate on with array-specific behavior (sorting, length, etc).Iterator.zip(whatever).toArray()
becomes boilerplate and noise.Array.zip
.Use-cases that I don't think are that strong:
Array.from
on its own input. It's on the API to take responsibility for this, i.e. the library / function should be updated, I shouldn't be dealing with tacking on a .toArray()
or using specifically Array.zip
. If I have to then this is "bad code". It's a real-world use, sure, but it's jank and expecting a .toArray()
here is fine..toArray()
until the infra adapts to apply Array.from
where appropriate or else respect iterators is fine. (Updating infrastructure can be a tall order, but there has to be some push for it.)I admit that in our own code the vast majority of the uses of our stitchArrays
function (equivalent to Array.zipToObjects
) is in an intermediary position, e.g. for-of, or in a finalizing position where the outside consumer has no justification to assume arrays rather than iterables (usually "here is a list of stuff I want embedded in this HTML context please"). The infrastructure needs updating to check for [Symbol.iterator]
instead of Array.isArray
(and so on), but that's not a complicated fix to bring about.
I'd love to have these use cases expanded upon. I strongly disagree with the premise that we need to justify retaining parity with concrete, real cases... (Wasn't much the point of iterator helpers to improve parity? Is there not an inherent value in making two API surfaces overlap with plainly analogous behavior in both? Don't we want to make room for programmer expression, rather than say, no, this is the way this works, iterators are arrays 2.0 and we won't help you use arrays anymore?)
...But I still think we should try to make that case. Losing sight of why parity is a good thing only hurts the case for parity overall, and keeps the conversation on new iterator features in general from having nearly as much perspective as is deserved.
Feedback from committee was to not include Array-specific handling in this proposal. Closing.
@michaelficarra um, that's not my recollection of the outcome? There's nothing in the notes that states that as a foregone conclusion, and I consider this something to resolve within stage 2, that will block stage 3.
Array-specific handling
This phrasing appears incorrect because the issue was about extending
the idea to Array, not to introduce Array-specific stuff. FWIW, I might be just splitting hairs though.
@ljharb Nobody spoke up for it when prompted, and one person explicitly spoke against it. But I'll re-open for now.
@tophf No, we're talking about having Array-specific accommodations. The proposal already works fine for Arrays because they are iterable.
@devsnek can you elaborate on your comment? did you mean, you think it should be a separate proposal, or that it shouldn't happen at all?
@ljharb To be clear, I don't think this was like a "1 vs 0" thing. Since the default state was to not do anything, I took the complete lack of expressed support as a strong negative signal.
I was pretty clear that I'd be doing a followup if it wasn't part of this proposal, and that the cross-cutting concerns would mean that their fates were linked anyways, would would effectively mean that this proposal couldn't advance to stage 2.7 until that one hit stage 2. At that point I'm not sure why a separate proposal adds value.
@ljharb Ideally I'd prefer that we aren't duplicating iterator methods on arrays just for convenience. If there's an issue expressiveness or performance, I think that's worth discussing, but I'm not sure either point has been raised yet.
Gotcha - why not? Most all of the iteration helpers are for convenience. (it's certainly also about expressiveness and performance, and a number of other things, but i think "convenience" is pretty compelling on its own)
I agree with @ljharb's argument above and especially wtih @tophf in https://github.com/tc39/proposal-joint-iteration/issues/1#issuecomment-1891139447.
Parity is a big deal for iterator helpers. In my mind, these proposals bring the same functionality to iterators as you're used to working with in arrays. If iterators are not intended to canonically serve as the replacement for arrays, then we shouldn't give preference to adding a useful feature for iterators but not for arrays — namely because that goes against the parity that we're establishing through the rest of the iterator helpers.
Here are the helpers in Iterator Helpers:
Method | Parity | Notes |
---|---|---|
.map |
yes | |
.filter |
yes | |
.take |
no | This is iterator-specific functionality, loosely analogous to .slice(0, n) |
.drop |
no | This is iterator-specific functionality, loosely analogous to .slice(n) |
.flatMap |
yes | |
.reduce |
yes | |
.toArray |
N/A | However, Iterator.from is analogous |
.forEach |
yes | |
.some |
yes | |
.every |
yes | |
.find |
yes |
Here are the other methods on Array.prototype that aren't paired in Iterator Helpers:
Method | Index? | From end? | Mutating? | Note |
---|---|---|---|---|
.at |
yes | sometimes | N/A | |
.concat |
no | no? | no | |
.copyWithin |
yes | sometimes | yes | |
.entries |
yes | no | N/A | |
.fill |
no | yes | sure | |
.findIndex |
yes | no | N/A | |
.findLast |
no | yes | N/A | |
.findLastIndex |
yes | yes | N/A | |
.flat |
no | no | no | |
.includes |
no | no | N/A | |
.indexOf |
yes | no | N/A | |
.join |
no | no | no | |
.keys |
yes | no | N/A | |
.lastIndexOf |
yes | yes | N/A | |
.pop |
no | yes | yes | |
.push |
no | yes? | yes | |
.reduceRight |
no | yes | no | |
.reverse |
no | yes | no | |
.shift |
no | no | yes | somewhat analogous to .drop |
.slice |
yes | sometimes | no | somewhat analogous to .drop or .limit |
.sort |
no | no | yes | result wouldn't have meaning as an iterator |
.splice |
yes | sometimes | yes | |
.toReversed |
no | yes | no | |
.toSorted |
no | no | no | result wouldn't have meanign as an iterator |
.toSpliced |
yes | sometimes | no | |
.unshift |
no | no | yes | |
.values |
no | no | no | iterators are already iterators |
.with |
yes | sometimes | no |
I put this table together to substantiate or counter my argument — to actually assess the overall parity. And yeah, basically everything here fairly obviously doesn't work as an iterator method, so isn't part of Iterator Helpers. I can only really make cases for .flat(Infinity)
, .includes()
, and .concat()
maybe having analogies in iterators — but my point isn't that we should split hairs, only that as far as I can tell, parity and completeness matters in Iterator Helpers.
Going counter to that parity because we don't have effective circumstantial justification / precedent for Array.zip
(etc) feels like a mistake to me — I don't think it needs "real-world use" precedent, and that parity is good enough in its own right. (Although I'd still argue there are real-world use cases, anyway!)
@towerofnix The intention of the iterator helpers proposal was not to "bring parity with array methods". It was to take a common interface shared among a great deal of the ecosystem and make it useful by default for js developers. Whether or not array has a certain method, while certainly not irrelevant to the discussion, should not be unto itself a reason to include or deny an iterator method. And this argument applies the other way too, just because iterators have a certain method doesn't mean that arrays (or any other collection type) must include it.
@ljharb I just don't think its particularly burdensome to jump between arrays and iterators. Maybe rust has desensitized me from typing array.iter().map(f).collect()
a lot but I like this pattern because it gives you a consistent interface to work with across all the different collection types in the language. This is just like, my opinion, though. I'm not going to block joint array iteration if people want to spend time on it, but I will continue to hold my opinion that I think spending time on it is unnecessary.
The danger of not maintaining the historical parity is that it will confuse developers as there's no logical substantiation for why some methods are missing in Array/Iterator. After developers needlessly suffer for a while a new proposal would appear to bring these new methods to arrays, similarly to the ongoing Iterator Helpers proposal.
TBF there isn't really "historical" parity, because Iterator Helpers and this proposal are both new additions to the language. I think parity is a good idea in general, and I feel that even if it wasn't part of the designed intent of Iterator Helpers there does appear to be a lot of parity where it makes sense. But those are also just my opinion.
If someone sees Iterator.zip
exists and Array.zip
does not, if they wonder anything, it's going to be along the lines of "huh, I think this has an obvious analogy in arrays, wonder why it's missing here even though most of the array functions with obvious analogies in iterators are there?" — not "huh, I thought we wanted parity from the start, very weird that it isn't pivotal now?"
I do think developers are going to have personal experiences that lead them to expect parity between obviously-analogous array and iterator functions, and IMO Iterator.zip
/ Array.zip
are pretty obviously analogous. But I don't know if others feel that parity as important or if we're just a vocal minority LOL.
I'd like to say it should always matter as something that's fundamental to how people learn programming languages (and interact with them in general), but that's a somewhat high-and-mighty perspective if I haven't done research or at least got anecdotal experience beyond the way I've learned programming languages.
@devsnek i think it's totally fine if users want to only use the iterator interface. I definitely don't think that preference should have any bearing on the separate existence of an array interface, for those that prefer that.
In general, "it's not personally interesting to me" isn't a persuasive reason not to progress on any functionality.
As a general principle, I also prefer to work with arrays where cache locality matters. But that argues that the use cases ought to be evaluated case-by-case. I am not sure zipping in particular is an operation I care about from a cache locality perspective. Should I?
I'm not sure what "cache locality" means here, and I don't think I have that use case - i definitely zip arrays together on a small number of projects, frequently.
Reminder to @ljharb and @syg to please discuss this and advise me on which direction to go.
I'm happy with either direction and am leaning towards omitting given the ease of calling toArray()
, just want to hear use cases.
By "cache locality" I mean if I care about the performance of accessing adjacent elements (both adjacent in their index and in time of access), I'll pay the up-front memory of the array. So @michaelficarra I was really asking you if I should care about zipping in that performance context. I still see mostly abstract discussion of use cases. @bakkot has a comment up-thread about zipping mostly being an intermediate step, which would make me think the per-item processing takes enough time that the performance of accessing adjacent, zipped elements isn't usually important.
@syg In that kind of performance context, you're probably going to try to avoid creating any kind of intermediate structure. Zip is useful when describing your program as a series of data transforms. In pure FP languages, those transforms get fused and the intermediate structures are never actually realised. But this is JavaScript, so if you want to achieve performance characteristics similar to what you'd get if you were manually jointly iterating (assuming you are doing further processing after your zip), you'd have to actually do just that.
which would make me think the per-item processing takes enough time that the performance of accessing adjacent, zipped elements isn't usually important.
This is probably usually true, which is another reason that Array zipping is not motivated by performance IMO. However, while you seem to want to use performance to justify Array zipping to yourself, I don't believe @ljharb was making a (solely) performance-based argument for it. From above,
it's certainly also about expressiveness and performance, and a number of other things, but i think "convenience" is pretty compelling on its own
If you find that unconvincing, we can omit it for now and pursue it as a follow-up. My goal here is to not put the Iterator-based zipping portion of this proposal at risk of not advancing.
If you find that unconvincing, we can omit it for now and pursue it as a follow-up. My goal here is to not put the Iterator-based zipping portion of this proposal at risk of not advancing.
I don't find the convenience argument convincing by itself because I think toArray
is convenient enough.
It's definitely not solely or even primarily performance-based; it's about the mental model.
Iterator.zip(a, b).toArray()
Array.zip(a, b)
Both are more or less identically convenient, but why should i have to reach for Iterator when i only want to work with arrays?
Yeah, and providing a standard language for that mental model is exactly why Array.zip
is a good idea iMO.
Like, if you only work with arrays and are not interested in interacting with iterators, then there's no reason your own project can't export a utility for that in some convenient-to-access place:
// util.js
export function arrayZip(...args) {
return Iterator.zip(...args).toArray()
}
But then it's up to each program to decide what name it wants to use for "array zip", and there aren't any good choices:
arrayZip
is the "nicest" one analogous to Array.zip
or Iterator.zip
...zipArray
("zip for arrays!"), and that's liable for confusion with zipToArrays
, for example...zip
and other code will use Iterator.zip
, and it's all a mess!Of course, each program that decides to make up a shorthand name will likely use a different one, and some just won't and will keep using Iterator.zip(a, b).toArray()
, and it'll be a no-good mishmash—of lots of programs representing the exact same, extremely simple and rather common logic with a wide variety of different forms/names.
It's impossible to get ahead of the curve on every possible "simple function", but because Array.zip
is such a direct and obvious analogy for Iterator.zip
, it would be reasonable to give it a standard (and predictable) name.
You can more or less address supporting your particular mental model using a shorthand utility function, but you certainly won't be using the same language as everyone else. Or you can hope to use the same language as people "should" use (longhand Iterator.zip().toArray()
), but sacrifice the legibility of your code and the benefit of sticking to a clear mental model.
Hey, just caught up on the TC39 meeting minutes from April 8 and April 11—didn't see those til now, despite our more recent reply! (I guess these weren't prepared and online til a week ago, so no surprise there in the end. https://github.com/tc39/notes/commit/52827613f4fd277959719a1ca59db78b5d4e0d82)
I'm not too worried about Array.zip
not existing initially, i.e. to be added in a separate proposal, though the concerns over that being a waste of committee time make sense. Still in favor of Array.zip
being a thing from the start because it aligns with differing mental models. I think the only comment I take an issue with is (April 11):
SYG: Like I think it behooves people to think about the difference. Like if you want to zip giant things you probably don’t want an array, right
I agree that it's of course a good idea for developers to be attentive about this. I just don't think it makes sense for the language to be, like, forcing a developer to make that choice, just because of a difference in API surface. That doesn't seem like a very useful precedent and I think it would be an un-fun time arguing against that when an Array.zip
proposal comes around, i.e. "OK but wouldn't that negate one of the reasons we didn't include Iterator.zip
? Like there's no way to add Array.zip
without losing out of the teaching benefit of not having Array.zip
". Like yes, of course this is true, but maybe it shouldn't be considered a meaningful teaching benefit in the first place.
Other than that I feel totally OK with this going either way for now. I hope if Array.zip
is not included that a proposal is able to progress without too much trouble or "wasting" committee time, but it's also cool if more nuances are figured out in the space of a dedicated proposal. If we end up in a world where we only have Iterator.zip().toArray()
then that's hardly the worst world to be coding in, too.
@towerofnix I don't understand your reply to my comment. I wasn't talking about teachability, but the performance cost of zipping giant things. That is, zipping giant arrays means holding the entire zipped result in an array, which uses a lot of memory. If most of the time, the point of zipping giant lists of things is to process the pairs one at a time, you do not want an array.
Yeah sorry, I can see that my point wasn't clear. What I meant is: That's a meaningful observation about performance, but it's something developers have to learn for themselves, so they can decide whether an iterator-based operation or an array-based operation is more suitable. Like it was discussed, for some kinds of data/operations zipping as an iterator is better, other times zipping into an array is. I could be misreading the minute here but that's what I took from your comment:
Like the performance characteristics are just different. I think we’ll be doing a disservice with performance if we have a catch all. We should have a version of zip on array because it’s more performance. That’s not true. It’s more performance in different cases, you’re making certain trade-offs and I want to be explicit that whatever proposal comes out of this, it’s called out, it’s not a general thing.
edit, also from @ljharb's reply to the above, emphasis mine:
JHD: Yea, I mean there are some use cases where you might want one over the other, of course, but that doesn’t change the conceptual operations you might want to perform.
Those trade-offs are things a dev should be aware of so they have to learn what they are, but they don't need to learn by being "forced" to consider the trade-off by the language, if that makes sense.
Like yeah, totally agreed with "If most of the time, the point of zipping giant lists of things is to process the pairs one at a time, you do not want an array." But zip isn't going to be used just for giant lists of things, it will also be used for comparatively very small lists of things, and the memory overhead of having all those things exist at once is not an issue. (Could be preferable compared to CPU overhead of initializing and processing an iterator, esp. if you're just toArray()
ing it immediately, depending on how the JS engine is optimized.)
So 1) there may be a case for Array.zip
being better in certain scenarios (in terms of perf), but my point is moreso 2) it's up to the JS dev to understand and learn how to differentiate, and the language doesn't need to move them along with that (by only having Iterator.zip
). IMO if you have a basic understanding of iterators then it's pretty clear that zipping several super-long lists all at once may be a cause for memory concern, that's something people hopefully acknowledge when they use array methods instead of iterators (prior to a refactor or bc arrays are just more suitable for their context).
IMO the symmetry (for like operations, not in general) between arrays and iterators already does a good job of encouraging you to consider if iterators are better for you, i.e. if your process could be represented similarly with iterators, then JavaScript is going to make it easy for you (by giving map
, filter
, etc). Iterator.zip
w/o Array.zip
could cause you to just scratch your head and be annoyed and then .toArray()
. Which TBF could cause you to rethink your decision (using arrays in the first place), but I think it's mostly a bad developer experience and they'd learn the trade-off better if they were presented with the choice, not forced / strongly pushed one way or the other.
Thanks for the explanation. I see where you're coming from. The general principle here is "performance footgun", where browsers especially (I represent V8/Chrome) don't want the slow thing to be too easy to reach for. The thinking goes as you've laid out: that the lack of availability would be met with some surprise and the developer would give extra thought to why there isn't symmetry.
It is reasonable to disagree with the above principle. At V8 we've drawn the opposite conclusion because performance, AFAICT, is by and large deprioritized by most web developers. The MO is to ship and to ship fast. Performance matters for the tail of very mature products, certainly, but what we've observed is that performance engineering is mostly a luxury -- features, correctness, etc, usually take priority. If there's a choice in APIs, the more convenient one usually wins out. As a browser, we have a goal in making the web browsing experience, and therefore websites, performant. Taken together, that's why we prefer to not spec potential performance footguns, especially if it's a matter of convenience and not expressibility.
I disagree with the "forced/strongly pushed" framing. toArray()
is not that inconvenient.
I have this need on Arrays just as often as on non-array iterators; any reason not to add it to both?