Search requests (templates approach)

Gallaecio commented 7 months ago

Search request support implemented by returning items that work as templates to be rendered into actual requests with some arguments.

Uses jinja2 for template handling, no strong opinion about that.

This approach is meant to support the following syntax:

# web-poet example implementation
search_request_template = get_item(
    "https://books.toscrape.com/",
    SearchRequestTemplate,
)
search_request = search_request_template.request(keyword="foo")
search_result = get_item(
    search_request.url,
    BookList,
)

# scrapy-poet
def parse_homepage(self, response: DummyResponse, search_request_template: SearchRequestTemplate):
    for keyword in self.keywords:
        yield search_request_template.request(keyword=keyword).to_scrapy(self.parse_search_results)

~This implementation is limited to the url field in line with the current implementation of scrapy-zyte-api. Before adding new fields, we might want to support them in scrapy-zyte-api.~

~For an alternative, see https://github.com/zytedata/zyte-common-items/pull/81.~

codecov-commenter commented 7 months ago

Codecov Report

Merging #80 (4f2e744) into main (56e9709) will not change coverage. Report is 1 commits behind head on main. The diff coverage is 0.00%.

:exclamation: Current head 4f2e744 differs from pull request most recent head fd6b471. Consider uploading reports for the commit fd6b471 to get more accurate results

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #80 +/- ## ===================================== Coverage 0.00% 0.00% ===================================== Files 11 11 Lines 1766 1791 +25 ===================================== - Misses 1766 1791 +25 ``` | [Files](https://app.codecov.io/gh/zytedata/zyte-common-items/pull/80?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) | Coverage Δ | | |---|---|---| | [zyte\_common\_items/\_\_init\_\_.py](https://app.codecov.io/gh/zytedata/zyte-common-items/pull/80?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-enl0ZV9jb21tb25faXRlbXMvX19pbml0X18ucHk=) | `0.00% <ø> (ø)` | | | [zyte\_common\_items/pipelines.py](https://app.codecov.io/gh/zytedata/zyte-common-items/pull/80?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-enl0ZV9jb21tb25faXRlbXMvcGlwZWxpbmVzLnB5) | `0.00% <ø> (ø)` | | | [zyte\_common\_items/pages.py](https://app.codecov.io/gh/zytedata/zyte-common-items/pull/80?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-enl0ZV9jb21tb25faXRlbXMvcGFnZXMucHk=) | `0.00% <0.00%> (ø)` | | | [zyte\_common\_items/items.py](https://app.codecov.io/gh/zytedata/zyte-common-items/pull/80?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-enl0ZV9jb21tb25faXRlbXMvaXRlbXMucHk=) | `0.00% <0.00%> (ø)` | |

kmike commented 7 months ago

Looks good overall!

zytedata / zyte-common-items

Search requests (templates approach) #80

Codecov Report