scrapinghub / shublang

Pluggable DSL that uses pipes to perform a series of linear transformations to extract data
BSD 3-Clause "New" or "Revised" License
15 stars 8 forks source link

Fix join pipe function #23

Open akshayphilar opened 4 years ago

renancunha commented 4 years ago

I don't know exactly the purpose of this issue, but I'll describe a situation that I found using the join pipe and that deserves some discussion about it.

Generally, all shublang functions are executed through the iterable that they receive. But the join pipe is executed joining the iterable by a string. I think that instead of joining the iterable, it should apply the join operation on every item of the iterable.

In other words, instead of having the join function like this:

def join(iterable, separator=", "):
    return separator.join(builtins.map(str, iterable))

we should have:

def join(iterable, separator=", "):
    return (separator.join(builtins.map(str, x)) for x in iterable)

I came to this analyzing the following situation:

What do you think about it? cc @BurnzZ @akshayphilar

akshayphilar commented 4 years ago

@renancunha you have captured the intent of this issue. Agree, join needs to be brought in line with other pipe functions as you have described above