search

stackmuncher / stm_server

The server-side code behind stackmuncher.com, a global directory of software developers.
https://stackmuncher.com
GNU Affero General Public License v3.0
2 stars 0 forks source link

Reduce amount of data returned by ES #32

Open rimutaka opened 2 years ago

rimutaka commented 2 years ago

Dev and Org profiles can get really big. Returning the entire _source as JSON makes no sense for lists when only a few properties are being used.

Exclude some of the fields with 

GET /org/_search?_source_excludes=report.file_types,report.reports_included,report.projects_included,report.commit_time_histo, report.contributor_git_ids

Refs and packages may be needed if there was a search for them.

rimutaka commented 2 years ago

This was a dumb idea. The performance of this feature is even worse than dealing with the full document. See https://discuss.elastic.co/t/source-excludes-includes-makes-query-2-times-slower/222227/2 for details.

The best way to solve this is to have a separate index with minimal amount of info + the current index for more detailed queries.