Crafting a custom Download URL with Text Inputs?

[gingerbeardman]

Existing issues

Having looked through the issues, briefly, the closest I can find is https://github.com/pipes-digital/pipes/issues/37 "Concatenate or just link text input to download block"

which talks of creating a Download URL from one static base part and one Text Input user-definable part.

I'd like to do exactly that but the example seems to no longer use this feature, as there is no text input block in it?

My problem

Ideally, I'd like to replace specific keywords in a URL with user-defined terms.

Is there a way to go about this?

Example

https://auctions.yahoo.co.jp/search/search?p={q}&auccat={cat}&va={q}&exflg=1&b=1&n=50

• replace {q} with 花札
• replace {cat} with 23336

Workaround

The workaround is to pass the whole URL each time, I guess? But I worry about encoding Japanese characters, editing the encoded URL, etc.

Thoughts

Can blocks like replace work on text strings?

[onli]

Wanted to say this can't be done yet, but it can be done.

But only indirectly. We would need a simple text holder block to start from the URL directly, which we don't have. But we can download the URL:

You download the URL from somewhere public, here from a gist. Create a temporary feed (I called it fake above) so the following blocks can work with it. It will have one item for each line. The replace block will replace the content in each line.

The download block only has text inputs. Which is a problem, because we don't have text anymore, but a whole feed (with one item). Thankfully the foreach block transforms feed content to text input. So drop a download block onto a foreach block and you are done.

Feed preview without the download block: https://www.pipes.digital/feedpreview/MOQ7gg9e

That's without the download block because the download with this specific url, with just one parameter removed, can't work, can it? and it's also possible that yahoo.jp can't be parsed by Pipes, they might detect us as a crawler and block the access. Pipes tries to behave properly, it accepts timeouts, but still.

It is also possible the download block won't work at all, this approach is a bit crazy and I doubt anyone has created a pipe like that. But it works until the download block, so it might work completely. Give it a shot, if there are problems with the pipe evaluation in the last step that would be fixable.

[gingerbeardman]

Pipe DNJyxk9a has been working so they don't seem to block. At least not yet :)

This is really cool lateral thinking! Thanks.

Feel free to clone the above pipe.

I'm also testing out Huginn and some other RSS generators.

You can read about my adventures so far: https://blog.gingerbeardman.com/2022/01/30/using-rss-to-create-saved-searches-for-any-website/

[gingerbeardman]

Corrected pipe name: DNJyxk9a

[onli]

Happy this works for you. If you write about Pipes I'd be happy to get a ping ;)