algolia / docsearch-scraper

DocSearch - Scraper
https://docsearch.algolia.com/
Other
305 stars 106 forks source link

The prompt record is 0 after running the container #577

Open pptfz opened 1 year ago

pptfz commented 1 year ago

config.json

{
  "index_name": "pptfz",
  "start_urls": [
    "https://blog.pptfz.cn/"
  ],
  "sitemap_urls": [
    "https://blog.pptfz.cn/sitemap.xml"
  ],
  "sitemap_alternate_links": true,
  "stop_urls": [],
  "selectors": {
    "lvl0": {
      "selector": "(//ul[contains(@class,'menu__list')]//a[contains(@class, 'menu__link menu__link--sublist menu__link--active')]/text() | //nav[contains(@class, 'navbar')]//a[contains(@class, 'navbar__link--active')]/text())[last()]",
      "type": "xpath",
      "global": true,
      "default_value": "Documentation"
    },
    "lvl1": "header h1",
    "lvl2": "article h2",
    "lvl3": "article h3",
    "lvl4": "article h4",
    "lvl5": "article h5, article td:first-child",
    "lvl6": "article h6",
    "text": "article p, article li, article td:last-child"
  },
  "strip_chars": " .,;:#",
  "custom_settings": {
    "separatorsToIndex": "_",
    "attributesForFaceting": [
      "language",
      "version",
      "type",
      "docusaurus_tag"
    ],
    "attributesToRetrieve": [
      "hierarchy",
      "content",
      "anchor",
      "url",
      "url_without_anchor",
      "type"
    ]
  },
  "conversation_id": [
    "833762294"
  ],
  "nb_hits": 9510
}

After running, the following message is displayed

docker run -it --env-file=/docusaurus/algolia/.env -e "CONFIG=$(cat /docusaurus/algolia/config.json | jq -r tostring)" algolia/docsearch-scraper
> DocSearch: https://blog.pptfz.cn/ 0 records)

Crawling issue: nbHits 0 for pptfz

How should this problem be solved