Open JochenHWezel opened 8 years ago
requires plugin concept
Pipeline
apps might provide crawler Setup for standard search index of Server Group, they might additionally or alternatively specify additional search index names for purpose of in-app-search-index
pages being crawled by crawler provide the data from the crawler Setup as follows:
crawler tracks page Status - on repetitive page error of an URL, it stops crawling for a configured time
search page can be set up with a Parser pre-set - requiring that user search text is logically valid/closed nexted logic Levels automatically on demand
crawler should be able to crawl
CrawlerRequestSetup must configure
crawler,should consider meta robots/index no-follow/follow, /robots.txt commands, /robots.txt-sitemap
might make sense to make indexing use a 2nd database for big Environments/heavy load and use the cwm db only for small Environments/light load
might make sense to re-use other technics/modules like luscene or other engines
Search using CMM crawler but with index for each Server Group and with security Information descriptor on each record
Cwm.Page implements Standard Interface
CrawlerRecord class, as List
IsCrawlerRequest checks for user Agent with individual token per crawler-user
cwm Needs an additional user for doing the crawling Jobs
crawler crawls