search

pipes-digital / pipes

Repository for Pipes
https://pipes.digital
GNU Affero General Public License v3.0
264 stars 21 forks source link

Extract content from feed link? #56

Closed lixxbox closed 4 years ago

lixxbox commented 5 years ago

Hi, is it possible to extract content from a feed link? (//item/link) I'd like to add some data to the feed.

According to my understanding, I would have to pass the extracted url to the downloader, which only accepts user input.

Am I missing something?

onli commented 5 years ago

Hi, so probably instead of showing the content transported in the link you'd like to download //item/link (but those are several items, right?) and download the page linked there?

lixxbox commented 5 years ago

It's hard to describe for me.

https://www.pipes.digital/editor/K9Pg72qD In this example i have two parts. The upper part extracts the link of every feed item. In the bottom part i pasted >one< link in the download item, and extract content for my output.

It would be great if I could connect the upper part, with the lower part and do this for every feed item.

onli commented 5 years ago

I see! Yes, that's exactly the scenario I meant as well. Pipes has a missing link there, some missing functionality to properly support it.

To change that I will have to change existing blocks or add new ones. So far, I see three options. I'm not sure which one is the best, so I will describe them here:

  1. Add a regular input to the download block, meaning "download all the links and put the result each into a new feed item", meant to be connected to the //item/link extractor block. But I think that would be confusing with the regular text input field the download block already has.
  2. Introduce a forEach-block, that applies the input it gets to the or to some inputs of the block added to it. That's the way Yahoo Pipes solved this scenario. if I remember correctly. It's powerful, but I always thought it a bit too complex.
  3. The download block can be connected to a text input block. I could add a text list input. And maybe a second block to convert from the extractor block result to that text list, if it's not up to the text list input to handle this with a regular input. Like with option 1, this would then change the behaviour of the download block to not just download the page once, but once for each given link and putting it all into a feed.

The third option is the newest idea and the one I like so far the most. Do you (or someone else) have a preference? Am I missing a better solution?

lixxbox commented 5 years ago

I don‘t have any preference. I am still thinking about, if option 2 or 3 has any advantages in conjunction with other blocks, but I can't come up with anything.

onli commented 5 years ago

I went with a foreach block so far, gave it some restrictions like only working with feed, download and tweet blocks (blocks without input and only one text input). Its prototype is working fine, but I still have to finish its functionality and the UI.

onli commented 5 years ago

Working example online under https://www.pipes.digital/pipe/GqKg2Lqx - does this work for you as expected?

The idea is that you can create a foreach block and then drop a download feed onto it to make it download all the links connected to it.

lixxbox commented 5 years ago

At first glance, I'd say yes. I will take a closer look at it again this weekend. I thank you very much for your efforts!