algolia / docsearch-scraper

DocSearch - Scraper
https://docsearch.algolia.com/
Other
305 stars 106 forks source link

Ignore sidebar headings in algolia indexing #572

Open cybersaksham opened 1 year ago

cybersaksham commented 1 year ago

I am having a problem with algolia. I want to exclude sidebar headings from getting listed.

When I search gallery I am getting 6 results with no difference. Because I have a sidebar named Website Gallery and it has 6 items inside it. That's why the search is being done for all 6.

gallery

Can you please tell me how can I ignore sidebar headings in config.json?

My config.json is

{
  "index_name": "portfolio-generator",
  "start_urls": ["https://portfolio-generator.cybersaksham.co.in/"],
  "sitemap_urls": [
    "https://portfolio-generator.cybersaksham.co.in/sitemap.xml"
  ],
  "sitemap_alternate_links": true,
  "stop_urls": ["/tests"],
  "js_render": true,
  "js_wait": true,
  "selectors": {
    "lvl0": {
      "selector": "(//ul[contains(@class,'menu__list')]//a[contains(@class, 'menu__link menu__link--sublist menu__link--active')]/text() | //nav[contains(@class, 'navbar')]//a[contains(@class, 'navbar__link--active')]/text())[last()]",
      "type": "xpath",
      "global": true,
      "default_value": "Documentation"
    },
    "lvl1": "header h1",
    "lvl2": "article h2",
    "lvl3": "article h3",
    "lvl4": "article h4",
    "lvl5": "article h5, article td:first-child",
    "lvl6": "article h6",
    "text": "article p, article li, article td:last-child"
  },
  "strip_chars": " .,;:#",
  "custom_settings": {
    "separatorsToIndex": "_",
    "attributesForFaceting": ["language", "version", "type", "docusaurus_tag"],
    "attributesToRetrieve": [
      "hierarchy",
      "content",
      "anchor",
      "url",
      "url_without_anchor",
      "type"
    ]
  },
  "conversation_id": ["833762294"],
  "nb_hits": 9000
}

Originally posted by @cybersaksham in https://github.com/algolia/docsearch-scraper/issues/571#issuecomment-1368670652