Real-time data imports question

[Outstep]

Hello,

I am starting to use YaCy more and more now and think that is could be the solution that I have been trying to locate for some time now.

For the project that I am doing, I need to extend YaCy in some way so that when a user query is made on the main page, then I can get that query and send it to and external meta search engine that returns JSON or XML results such as URL, Title, Description, image link, date, etc...

I then want to import that data in real-time into my YaCy node for storage and usage later as well as to return those results to back to the user in the frontend.

The idea is that the node will get some external results from the meta-search engines data source, store them for later use in YaCy, as well as to aggregate the external results with any unique internal results in the YaCy storage and return them to the user. This approach allows YaCy to also act as a "cache" for trending and current user queries as well.

I found a little information on the main YaCy pages (https://yacy.net/en/API.html) under "Data import with XML" but that may not be a solution and I am trying to find out what might be needed to augment YaCy or write some type of Plugin that could make this work.

Any ideas or suggestions would be greatly appreciated. Thanks in advance :)

[tangdou1]

It is a cool idea !

[smokingwheels]

cool

[luccioman]

@lonnieOST your entry point in YaCy code base is likely the FederateSearchManager class. You will see that it currently support parsing and integrating results from :

• OpenSearch systems in RSS/Atom feed format or HTML with custom mapping, through the OpenSearchConnector
• external Solr systems with custom mapping through the SolrFederateSearchConnector.