essiembre
commented
10 years ago Here is a minimalist configuration sample for Solr that should address your requirements (except for Oracle, explained further down).
<!-- You can define variables with the "set" directive to simplify your
config maintenance (using Apache Velocity syntax) -->
#set($urlFilter = "com.norconex.collector.http.filter.impl.RegexURLFilter")
<httpcollector id="Minimal Config HTTP Collector">
<crawlers>
<crawler id="Minimal Solr Config Example">
<!-- Requires at least one start URL. -->
<startURLs>
<url>http://lucene.apache.org/solr/</url>
</startURLs>
<!-- Put a maximum depth to avoid infinite crawling (e.g. calendars). -->
<maxDepth>2</maxDepth>
<!-- Be as nice as you can to sites you crawl. -->
<delay default="1500" />
<!-- At a minimum make sure you stay on your domain. -->
<httpURLFilters>
<filter class="$urlFilter"
onMatch="include">http://lucene.apache.org/solr/.*</filter>
<filter class="$urlFilter"
onMatch="exclude">.+\.(png|jpg|jpeg|gif|ico|css|js)$</filter>
<filter class="$urlFilter"
onMatch="exclude">.+\?.*</filter>
</httpURLFilters>
<importer>
<postParseHandlers>
<!-- Unless you configured Solr to accept ANY fields, it will fail
when you try to add documents. This "KeepOnlyTagger" ensures
to drop every field crawled except those you want. -->
<tagger class="com.norconex.importer.tagger.impl.KeepOnlyTagger"
fields="document.reference,title" />
<!-- Strip content you do not want -->
<transformer class="com.norconex.importer.transformer.impl.StripBetweenTransformer"
inclusive="true" caseSensitive="false" >
<stripBetween>
<start><!-- whatever start text, like a comment --></start>
<end><!-- whatever end text, like a comment --></end>
</stripBetween>
<!-- multiple strignBetween tags allowed -->
</transformer>
<!-- The importer has a lot of config options where you can define
constants, rename fields, manipulate your content, etc. -->
</postParseHandlers>
</importer>
<!-- A "committer" dictates where the crawled content goes. -->
<committer class="com.norconex.committer.solr.SolrCommitter">
<solrURL>http://localhost:8983/solr/collection1</solrURL>
</committer>
<!-- When developing or troubleshooting, you can use the filesystem
committer so you can have a precise look at the content
that would be sent to Solr. -->
<!--
<committer class="com.norconex.committer.impl.FileSystemCommitter">
<directory>./examples-output/minimum/crawledFiles</directory>
</committer>
-->
</crawler>
</crawlers>
</httpcollector>A lot more tuning options are available to you and a fast way to discover them all is to look at these summarized configuration documentation pages:
HTTP-specific options: http://www.norconex.com/product/collector-http/configuration.html Importing options: http://www.norconex.com/product/importer/configuration.html Solr-specific options: http://www.norconex.com/product/committer-solr/
For Oracle, we have not yet released a database collector (the day will come). So you have a few options, like having a web app to crawl on top of your database, or, extract the data from the database to files, and then use the Filesystem Collector to index that data. The Filesystem Collector supports the same importing options and the same Solr "committer". It's location: http://www.norconex.com/product/collector-filesystem/
Let us know if you need clarification on anything.
raviks007
liar666
shreya-singh-tech
Please provide a sample setup to crawl a website and store the content in Solr repo. Also we have other requirements like, indexing Metadata, skip certain URLs, parsing only part of a content page and also parsing the data from oracle database.
Is it possible to give a best example to help me to implement the above requirements. We are actually looking to finalise to go with Apache Nutch or Norconex. I have no experience in Norconex as I just read about it since yesterday. It will be helpful if you can provide the inputs so that I can showcase and decide on the crawler.
Thanks, Ravi