elastic / connectors

Source code for all Elastic connectors, developed by the Search team at Elastic, and home of our Python connector development framework
https://www.elastic.co/guide/en/enterprise-search/master/index.html
Other
58 stars 116 forks source link

[Github] Add support for `path` filtering in Advanced Rules #2637

Open spong opened 2 weeks ago

spong commented 2 weeks ago

Problem Description

As I was working to ingest our kibana documentation, I wasn't able to get the sync to work initially (and so had no data) so ended up attempting to do a Github path filter via Advanced Filters instead of the Basic Rules (since I wasn't sure what the path field was or the format to make the regex for).

I added the following advanced filter with the path:/dev_docs/**/*.md* filter (which works in Github search) and the sync failed with the following error:

Proposed Solution

Github path filter works in Advanced Rules just as it does in the Github search interface.

Alternatives

This can be done via Basic Rules as mentioned, but in that instance all docs are retrieved and then filtered, so this would be a performance enhancement by only retrieving the desired documents (which would be nice for a repo like kibana which has 75k+ files).