New search concept

[JochenHWezel]

Search using CMM crawler but with index for each Server Group and with security Information descriptor on each record

Cwm.Page implements Standard Interface

• IsCrawlerRequest
• CrawlerRequest
  • SiteMapByVisitor
  • Anonymous
  • Public
  • IndividualUser -> causing crawler to scan and save Content for all users separately -> warning: expensive
  • AuthorizationForSecurityObject(secObj as string)
  • SecurityObjectsInvolved as String()
  • RecordByVisitor
  • depends on market/language/None (Default market or must-override)
  • all methods are functions executed on crawlerrequest - no additionalmcode should be executed if a Standard user opens a page
  • app's initial page usually provides the crawler Setup, addional pages only Need it if additional crawler Setup has to be provided (i expect this Situation to be very seldom)

CrawlerRecord class, as List

• PlainTextContent
• TargetUrl
• TargetReference (app specific value, e.g. an object no. - required for in-app-searches)
• RelatedToMarket
• AvailableForSecurityObject
• MetaFieldName (nothing for Default Content, a string for a meta field Name)
• MetaFieldType (see additional flag types) Without CrawlingSetup Within an app, crawler crawls for anonymous only stops on web forms (no submit)

IsCrawlerRequest checks for user Agent with individual token per crawler-user

cwm Needs an additional user for doing the crawling Jobs

crawler crawls

• on configured Server Groups (with auto-replace rules for Server names in URLs in case crawler Needs to scan at a virtual host which has got another official Name, typically when crawler runs at Intranet but for Extranet Server Group)
• entry pages are
  • servergroupfficialurl
  • auto-completed navurls for all available nav items
• every activated market
• as webslave Service recommended, as core webservice Task by cwm trigger for shared Hostings without webslave machine (webcron or jit-page-access-cron)

[JochenHWezel]

requires plugin concept

[JochenHWezel]

Pipeline

1. craler,sitemap scan list
2. crawler record data
3. split up into words
4. usage by search form with Logical parser

[JochenHWezel]

apps might provide crawler Setup for standard search index of Server Group, they might additionally or alternatively specify additional search index names for purpose of in-app-search-index

[JochenHWezel]

pages being crawled by crawler provide the data from the crawler Setup as follows:

• sitemap as link list in non-visible,div
• record data as ...???
  • maybe additional sitemap URL to Special URL for providing record data?

[JochenHWezel]

crawler tracks page Status - on repetitive page error of an URL, it stops crawling for a configured time

[JochenHWezel]

search page can be set up with a Parser pre-set - requiring that user search text is logically valid/closed nexted logic Levels automatically on demand

[JochenHWezel]

crawler should be able to crawl

• pdf
• docx
• xlsx,cells as formattedtext
• plain text
• html
• jpg meta data
• etc.

[JochenHWezel]

CrawlerRequestSetup must configure

• follow links for X Levels (e.g. as xenu)
• follow links
  • within nav URL Folder (default)
  • within custum list of URLs
  • everywhere (up,to Limit of X Levels)

[JochenHWezel]

crawler,should consider meta robots/index no-follow/follow, /robots.txt commands, /robots.txt-sitemap

[JochenHWezel]

might make sense to make indexing use a 2nd database for big Environments/heavy load and use the cwm db only for small Environments/light load

might make sense to re-use other technics/modules like luscene or other engines