Hey, I have set up an OSS (via .deb package) with a database crawler and it used to work perfectly until I did another crawl. With every db crawl we do everything is created again, so if I do two db crawls I have every document twice. So with every new db crawl one sees one more search result for the same thing. Is there a way to implement some sort of deduplication in OSS or have I set up the db crawler wrong? For now I deleted all documents and did one re-crawl.
Hey, I have set up an OSS (via .deb package) with a database crawler and it used to work perfectly until I did another crawl. With every db crawl we do everything is created again, so if I do two db crawls I have every document twice. So with every new db crawl one sees one more search result for the same thing. Is there a way to implement some sort of deduplication in OSS or have I set up the db crawler wrong? For now I deleted all documents and did one re-crawl.
Thanks in advance and have a nice day!