search

deepset-ai / haystack-integrations

🚀 A list of Haystack Integrations, maintained by the community or deepset.
57 stars 67 forks source link

Adding Duckduckgoapi for Websearch #263

Closed GivAlz closed 3 weeks ago

GivAlz commented 1 month ago

Hi everyone!

Since all SearchAPIs available as default with Haystack are commercial (or with a free trial) I thought it would be a nice idea to add a free API.

Introducing DuckduckgoApiWebSearch which uses Duckduckgo_search under the hood to make the searches.

The class is essentially a modification of the haystack classes.

Any feedback is welcome. Thank you!

GivAlz commented 3 weeks ago

Thank you for the review @anakin87 ...that was goofy of me! I changed the package's name at the beginning, I guess this caused the issue.

I just released version 0.1.11, which contains a typo in the Readme but I double checked and the command 

  from duckduckgo_api_haystack import DuckduckgoApiWebSearch

runs properly...plus I added a frequency limiter to avoid limiting rates.

Hopefully now everything will run correctly! Thank you!!

anakin87 commented 3 weeks ago

I have tried but I still get a ModuleNotFoundError. See this notebook.

There might be some problem with packaging. Here's a link to an old project of mine, which I hope it can help: https://github.com/anakin87/haystack-entailment-checker.

GivAlz commented 3 weeks ago

I have tried on another machine and now I can reproduce the error, it likely is caused by the use of "-" in the package name (I can see that the package.egg-info didn't track the files correctly).

I'll fix and test on different machines, then let you know...sorry, still new to packaging. And thank you, @anakin87 !

GivAlz commented 3 weeks ago

There was an issue in the pyproject.toml, the following part was loading from a non-existent directory

[tool.setuptools.packages.find]
include = ["duckduckgo_api_haystack*"] # <--- wrong line was here

I have packaged a new version (v 0.1.12) and tested it locally and from a google colab..sorry again @anakin87 !

GivAlz commented 3 weeks ago

Now it works like a charm ✨

I will merge this PR (I felt free to do some cosmetic changes)...

Just a heads-up... Since you are using this syntax: str | None, your library requires python>=3.10. It would be better to change the pyproject.toml file to reflect this (I got errors with older python versions -> https://stackoverflow.com/questions/76712720/typeerror-unsupported-operand-types-for-type-and-nonetype).

Thank you for the changes and suggestions: I'll fix them in the next package version!!