No CachedCrawlData loaded for nested importer responses

[mariuspruski]

My issue: I'm using a DOMSplitter to create several documents out of one reference, and these child documents will always appear as modified because their cached crawl data is never loaded from the store.

My goal: Child documents should not be added if they weren't modified since the last crawler run.

Additional info: I have invested in the Norconex core and noticed that

• the checksum of the child documents is correctly computed and put into the crawl store
• the checksum comparison is called also for the child documents, but it always returns the result that the document has been modified
• the checksum comparison is based on the cachedCrawlData which is only loaded one time for the parent reference: See AbstractCrawler.java::517. This cachedCrawlData is passed on unchanged also for all child documents. However, it contains no checksums for the child documents, there is no oldChecksum available for their references, that's why the checksum comparison always returns that the document has been modified and needs to be added

Solution suggestion: I manage to fix this issue (and produce the desired result) when I change code in AbstractCrawler.java::609 to the following:

        for (ImporterResponse child : children) {
            BaseCrawlData embeddedCrawlData = createEmbeddedCrawlData(
                    child.getReference(), crawlData);
+++         cachedCrawlData = (BaseCrawlData)crawlDataStore.getCached(child.getReference())
            processImportResponse(
                    child, crawlDataStore, embeddedCrawlData, cachedCrawlData);
        }

Meaning that for every processed child document before processing its import response I load the cachedCrawlData for this child's reference (under which it was saved in the last run).

[essiembre]

The latest release has this fix in. Can you try it and confirm it works for you?

[mariuspruski]

Thanks for addressing the issue; unfortunately I can't test it right now, as I'm off the project.

[essiembre]

No problem. I will close then. Thanks for your fix.