search

yacy / yacy_grid_mcp

The YaCy Grid Master Connect Program
GNU Lesser General Public License v2.1
654 stars 28 forks source link

add a query interface which provides search results like the yjson API from legacy YaCy #11

Open Orbiter opened 7 years ago

Orbiter commented 7 years ago

This shall implement the query attributes as described in http://www.yacy-websuche.de/wiki/index.php/Dev:APIyacysearch

It shall additionally understand Solr facet query attributes.

The result format is not documented well. It shall return the same result that susper.com understands. Here is an example: http://yacy.searchlab.eu/solr/select?hl=false&wt=yjson&facet=true&facet.mincount=1&facet.field=url_file_ext_s&start=0&rows=10&query=berlin

The query interface uses elasticsearch as search endpoint and translates the result into the yjson format.

harshit98 commented 7 years ago

I would like to work on this. :+1:

harshit98 commented 7 years ago

@nikhilrayaprolu @Orbiter How to proceed with this issue?

nikhilrayaprolu commented 7 years ago

@harshit98 according to this issue we might have to create a YjsonResponse similar to this https://github.com/yacy/yacy_search_server/blob/6fe735945da97abcbb91ac545fb11cff9d48effc/source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java but replacing the solr interface with the elastic search.

nikhilrayaprolu commented 7 years ago

additional files related to yacysearch.json End point could be found here. https://github.com/yacy/yacy_search_server/blob/master/htroot/yacysearch.json https://github.com/yacy/yacy_search_server/blob/master/htroot/yacysearch.java

nikhilrayaprolu commented 7 years ago

@harshit98 what is the status of this issue. @Orbiter any suggestions you would like to give regarding this?

harshit98 commented 7 years ago

@nikhilrayaprolu I have not started it.

Orbiter commented 7 years ago

I already implemented the search service in most parts here: https://github.com/yacy/yacy_grid_mcp/blob/master/src/main/java/net/yacy/grid/mcp/api/index/YaCySearchService.java

Because the mcp is the part which has the elasticsearch connection, the search api is also inside the mcp.

What this code has not yet implemented are the facets - no facets at all. To implement them, the search endpoint must be changed in such a way that it uses https://github.com/yacy/yacy_grid_mcp/blob/master/src/main/java/net/yacy/grid/io/index/ElasticsearchClient.java#L726

To do that, a QueryBuilder object must be constructed from the query. Code for this can be copied from https://github.com/yacy/yacy_grid_mcp/blob/master/src/main/java/net/yacy/grid/io/index/ElasticsearchClient.java#L670

Once the search endpoint is changed and is working, the facets can be taken from the aggregations object https://github.com/yacy/yacy_grid_mcp/blob/master/src/main/java/net/yacy/grid/io/index/ElasticsearchClient.java#L714

nikhilrayaprolu commented 7 years ago

@Orbiter I am trying to build the QueryBuilder object and make it use the Query class I have code in this form now:

public List<Map<String, Object>> query(final String indexName, final String q, final Operator operator, final int offset, final int count, String... fieldNames) {
        QueryBuilder queryBuilder = QueryBuilders.multiMatchQuery(q, fieldNames).operator(operator).zeroTermsQuery(ZeroTermsQuery.ALL);
        return new Query(indexName,  queryBuilder, order_field, 0, 10, 0,aggregations);

right now if we check we are not passing any order_field and aggregations. if so what should be the default values to be passed for both of these parameters?