Search Engine Parameters

[ParthS007]

Current behavior: Presently, Query server is extracting data from sources which are not search-engine.

Expected behavior: Query server needs to extract data from only search engines.

This issue needs discussion as search engine parameters needs to be clear because there are many Pending PRs and open Issues related to this.

Please give your views @mariobehling @vaibhavsingh97 @cclauss @bhaveshAn

[vaibhavsingh97]

@ParthS007 I think the same, as quora is not a search engine. I don't know why we are adding? Also, As @mariobehling said this project is only for the support of susper to provide quality search. Can anybody help with the understanding the need for adding searches from different source like quora, stackoverflow, etc

[cclauss]

My sense is that this project has captured the imagination of a lot of developers. Those developers each think a bit differently about the purpose of this project and they are delivering new capabilities that increase the value of the query-server beyond its original purpose.

The project maintainers have two choices:

• Reject all pull requests for scrapers that are not directly aligned with the original purpose --or--
• Create one directory of supported scrapers and another directory of unsupported scrapers

The second option allows contributors to continue to add value to the project but it imposes a burden on the project maintainers that might not be aligned with their interests. No matter which choice the maintainers make, README.md should be updated to make that choice clear.

[bhaveshAn]

Well at the time i added Quora support, I agreed to add Quora due to the fact that there was no statement which said Quora was invalid in this project in the PR https://github.com/fossasia/query-server/pull/143

[raju249]

Can I make a comment, if allowed ?

Its just that any project should be a product in itself. For example, if we talk about AWS, what Amazon is doing is renting the server space that they themselves used to use for their hosting and computing purposes. But that didn't stop them from adding those services which may or may not be useful for them, but the users found it useful.

So I am thinking that we keep the same mind set while developing for our FOSSASIA projects. Thanks

[cclauss]

Good comments!! A big difference between Amazon and this project is that AWS makes money while this project does not. That means that while Amazon can hire employees to add new capabilities and services, the maintainers of this project can not. They need to deal with the complexities of this project along with all of their other responsibilities and activities. Therefore we must be patient and understanding if the maintainers declare that they only want to support a limited number of scrapers. After all, we are not paying them for the use of query-server.

[raju249]

Yeah, That's right @cclauss :smile: Just wanted to share my thoughts.

[niranjan94]

Presently, Query server is extracting data from sources which are not search-engine.

@ParthS007 a search engine need not a website that indexes and searches websites. It could mean any form of data.

Google, Yahoo, Bing, Ask, DuckDuckGo, Yandex, Baidu, Exalead, Parsijoo - Websites Quora, Stackoverflow - Q & A YouTube - Video

So, i don't see how this issue is relevant right now.

[ParthS007]

@niranjan94 I have raised this issue because I have made PR which is adding support of Medium but then Mario said that It is not a search engine and need not be added, This Issue was just to clarify what exactly we need to add. Is Medium fine in your opinion?

[ParthS007]

@niranjan94 Link to Issue - Add Medium Support .