3.0.0-M2 release: ready for testing

[essiembre]

A second milestone release of the HTTP Collector was just made. One of the most significant features comes from the Importer module and allows you to better control the Importer handler flow via XML configuration.

It aims to replace the <restrictTo ...> options applied to each handler and the Importer "filter" handlers. It is better explained with a before/after example from the <importer> section of a configuration file...

Before:

    <tagger class="com.norconex.importer.handler.tagger.impl.ConstantTagger">
      <restrictTo field="document.reference">.*/hr/.*</restrictTo>
      <constant name="department">Human Resources</constant>
    </tagger>
    <tagger class="com.norconex.importer.handler.tagger.impl.DOMTagger">
      <restrictTo field="document.reference">.*/hr/.*</restrictTo>
      <dom selector=".contact .name" toField="fullName" />
    </tagger>

After:

    <if>
      <condition class="ReferenceCondition">
        <valueMatcher>.*/hr/.*</valueMatcher>
      </condition>
      <then>
        <handler class="ConstantTagger">
          <constant name="department">Human Resources</constant>
        </handler>
        <handler class="DOMTagger">
          <dom selector=".contact .name" toField="fullName" />
        </handler>
      </then>
    </if>

For very simple use cases it can be a bit more verbose, but you can see it allows grouping multiple handlers under the same condition. There are several other limitations to "restrictTo" that are being addressed by this new approach. In addition to the above example...

• Multiple conditions can be grouped under a <conditions operator="[AND|OR]"> tag (where they should all be matched, or any should be matched -- AND vs OR).
• While being the default condition (TextMatcher), conditions are no longer limited to text matching. They can be anything, starting with these.
• Negations are much easier to perform thanks to <if> vs </ifNot> as well as <then> vs <else>.
• Conditions can be nested.
• The <then> or <else> clauses can contain a mix of nested conditions and handlers.
• Filter handlers are no longer necessary thanks to using conditions along with the special <reject/> tag.

Here is a more complex example to better illustrate the above options:

      <if>
        <conditions operator="AND">
          <condition class="ScriptCondition">
            metadata.getString('document.contentFamily') === 'html';
          </condition>
          <condition class="NumericCondition">
            <fieldMatcher>Content-Length</fieldMatcher>
            <valueMatcher operator="GE" number="10000"/>
            <valueMatcher operator="LT" number="20000"/>
          </condition>
        </conditions>
        <then>
          <handler class="ConstantTagger">
            <constant name="htmlDocSize">10K-20K</constant>
          </handler>
        </then>
        <else>
          <ifNot>
            <condition>
              <fieldMatcher>document.contentType</fieldMatcher>
              <valueMatcher method="wildcard">text/*</valueMatcher>
            </condition>
          </ifNot>
          <then>
            <reject/>
          </then>
        </else>
      </if>

Open questions:

• Given the above being a lot more flexible, is there value in keeping the <restrictTo> around for Importer handlers? We are considering deprecating them.
• For the same reason, we are also considering deprecating the filter handlers in favor of conditions + <reject/>. Any value in keeping them around in the long run?

What's next?

This should be the last milestone release before the release candidate. To help us release it sooner, here is how you can help:

• Try out this new milestone, and more specifically the new Importer XML-flow configuration options. Give your feedback and report issues (make sure to mention this version)
• Documentation is still being put together. Share areas where you struggled the most, or you feel deserve more documentation and examples.

Happy crawling!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.