search

Norconex / crawlers

Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
https://opensource.norconex.com/crawlers
Apache License 2.0
183 stars 68 forks source link

Documents in <queueDir> not processed #105

Closed csaezl closed 9 years ago

csaezl commented 9 years ago

Related to issue #69, this is a small sample of the problem I experience concerning <queueDir>. That is, when a crawler is runnning, sometimes I see documents in <queueDir> registered some time ago, that still are there and don't exist in Solr.. The sample contains 2 documents in <queueDir> registered 5 hours ago. 

<commitBatchSize>10</commitBatchSize>
<queueSize>100</queueSize>

To no extend too much this text, I only copy the REF file for each document and the excerpt from the log file where the documents are commited and sent to Solr.

REF file-1
http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=5&amp;page=4
REF file-2
http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=4&amp;page=3

Log:

MC(crawler): 2015-05-07 12:59:46 INFO -            URLS_EXTRACTED: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=5&amp;page=16 (Subject: [http://www.government.nl/documents-and-publications/leaflets/2015/04/28/iob-iob-evaluation-newsletter-14-08-renewable-energy-in-rwanda-benefits-for-households-yet-limited-impact-on-co2-reduction.html, http://www.rijksoverheid.nl/papiamento/, http://www.government.nl/documents-and-publications/leaflets/2014/11/07/iob-iob-evaluation-newsletter-14-07-a-strategic-approach-dutch-coalitionbuilding-and-the-multi-bi-approach-to-influencing-eu-decision-making-2008-2012.html, http://www.government.nl/privacy, http://roimg.nl/presentation/images/logotype-government-of-the-netherlands-en.png, http://www.government.nl/ministries, http://www.government.nl/documents-and-publications/annual-reports/2010/08/01/annual-social-report-2010.html, http://www.government.nl/documents-and-publications/reports/2014/09/1/iob-a-strategic-approach-dutch-coalition-building-and-the-multi-bi-approach-in-the-context-of-eu-decision-making-2008-2012-summary-main-findings-and-issues-for-consideration.html, http://www.government.nl/documents-and-publications/parliamentary-documents/2015/02/23/state-of-the-european-union-2015.html, http://www.government.nl/cookies, http://www.government.nl/sitemap, http://www.government.nl/contact, http://www.government.nl/issues, http://www.government.nl/documents-and-publications, http://www.government.nl/news, http://www.rijksoverheid.nl/, http://www.rijksoverheid.nl/papiamentu/, http://www.government.nl/documents-and-publications/reports/2015/04/28/iob-riding-the-wave-of-sustainable-commodity-sourcing-review-of-the-sustainable-trade-initiative-idh-2008-2013.html, http://www.government.nl/, http://www.government.nl/documents-and-publications/reports/2015/04/28/iob-access-to-energy-in-rwanda-impact-evaluation-of-activities-supported-by-the-dutch-promoting-renewable-energy-programme.html, http://www.government.nl/rss, http://www.government.nl/documents-and-publications/reports/2015/04/14/iob-evaluation-of-dutch-development-cooperation-in-the-palestinian-territories-2008-2014.html, http://www.government.nl/documents-and-publications/reports/2015/01/01/iob-impact-of-ending-dutch-development-support.html, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=5&amp;amp;page=16&amp;page=5, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=5&amp;amp;page=16&amp;page=6, http://www.government.nl/accessibility, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=5&amp;amp;page=16&amp;page=2, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=5&amp;amp;page=16&amp;page=3, http://www.government.nl/help, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=5&amp;amp;page=16&amp;page=4, http://www.government.nl/international, http://subscribe.government.nl, http://www.government.nl/government, http://www.government.nl/documents-and-publications/forms/2015/04/07/administrative-rules-and-model-application-form-srhr-partnership-fund.html, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=5&amp;amp;page=16&amp;page=16, http://www.government.nl/copyright])
MC(crawler): 2015-05-07 12:59:46 INFO -         DOCUMENT_IMPORTED: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=5&amp;page=16 (Subject: com.norconex.importer.response.ImporterResponse@7673d589)
MC(crawler): 2015-05-07 12:59:46 INFO -    DOCUMENT_COMMITTED_ADD: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=5&amp;page=16 (Subject: SolrCommitter [solrURL=http://localhost:8983/solr/MyRepository, updateUrlParams={update.chain=langid}, solrServerFactory=DefaultSolrServerFactory [server=com.norconex.committer.solr.SolrCommitter$CommitterSolrClient@2422ddad], com.norconex.committer.solr.SolrCommitter@38bbca65[queueSize=100,docCount=206797,queue=com.norconex.committer.core.impl.FileSystemCommitter@113f6dc4[directory=D:/CRAWLER-MyRepository/collectors/CollectorA/work//queue],commitBatchSize=10,maxRetries=2,maxRetryWait=5000,operations=[],docCount=0,targetReferenceField=id,sourceReferenceField=document.reference,keepSourceReferenceField=false,targetContentField=content,sourceContentField=<null>,keepSourceContentField=false]])
MC(crawler): 2015-05-07 12:59:47 INFO -          DOCUMENT_FETCHED: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=5&amp;page=4 (Subject: com.norconex.collector.http.fetch.impl.GenericDocumentFetcher@2811e442)
MC(crawler): 2015-05-07 12:59:47 INFO -       CREATED_ROBOTS_META: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=5&amp;page=4 (Subject: none)
MC(crawler): 2015-05-07 12:59:47 INFO -           REJECTED_FILTER: http://www.rijksoverheid.nl/papiamento/ (Subject: none)
MC(crawler): 2015-05-07 12:59:47 INFO -           REJECTED_FILTER: http://subscribe.government.nl (Subject: none)
MC(crawler): 2015-05-07 12:59:47 INFO -           REJECTED_FILTER: http://www.rijksoverheid.nl/ (Subject: none)
MC(crawler): 2015-05-07 12:59:47 INFO -           REJECTED_FILTER: http://www.rijksoverheid.nl/papiamentu/ (Subject: none)
MC(crawler): 2015-05-07 12:59:47 INFO -           REJECTED_FILTER: http://roimg.nl/presentation/images/logotype-government-of-the-netherlands-en.png (Subject: none)
MC(crawler): 2015-05-07 12:59:47 INFO -            URLS_EXTRACTED: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=5&amp;page=4 (Subject: [http://www.government.nl/documents-and-publications/leaflets/2015/04/28/iob-iob-evaluation-newsletter-14-08-renewable-energy-in-rwanda-benefits-for-households-yet-limited-impact-on-co2-reduction.html, http://www.rijksoverheid.nl/papiamento/, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=5&amp;amp;page=4&amp;page=6, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=5&amp;amp;page=4&amp;page=5, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=5&amp;amp;page=4&amp;page=4, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=5&amp;amp;page=4&amp;page=3, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=5&amp;amp;page=4&amp;page=2, http://www.government.nl/documents-and-publications/leaflets/2014/11/07/iob-iob-evaluation-newsletter-14-07-a-strategic-approach-dutch-coalitionbuilding-and-the-multi-bi-approach-to-influencing-eu-decision-making-2008-2012.html, http://www.government.nl/privacy, http://roimg.nl/presentation/images/logotype-government-of-the-netherlands-en.png, http://www.government.nl/ministries, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=5&amp;amp;page=4&amp;page=16, http://www.government.nl/documents-and-publications/annual-reports/2010/08/01/annual-social-report-2010.html, http://www.government.nl/documents-and-publications/reports/2014/09/1/iob-a-strategic-approach-dutch-coalition-building-and-the-multi-bi-approach-in-the-context-of-eu-decision-making-2008-2012-summary-main-findings-and-issues-for-consideration.html, http://www.government.nl/documents-and-publications/parliamentary-documents/2015/02/23/state-of-the-european-union-2015.html, http://www.government.nl/cookies, http://www.government.nl/sitemap, http://www.government.nl/contact, http://www.government.nl/issues, http://www.government.nl/documents-and-publications, http://www.government.nl/news, http://www.rijksoverheid.nl/, http://www.rijksoverheid.nl/papiamentu/, http://www.government.nl/documents-and-publications/reports/2015/04/28/iob-riding-the-wave-of-sustainable-commodity-sourcing-review-of-the-sustainable-trade-initiative-idh-2008-2013.html, http://www.government.nl/, http://www.government.nl/documents-and-publications/reports/2015/04/28/iob-access-to-energy-in-rwanda-impact-evaluation-of-activities-supported-by-the-dutch-promoting-renewable-energy-programme.html, http://www.government.nl/rss, http://www.government.nl/documents-and-publications/reports/2015/04/14/iob-evaluation-of-dutch-development-cooperation-in-the-palestinian-territories-2008-2014.html, http://www.government.nl/documents-and-publications/reports/2015/01/01/iob-impact-of-ending-dutch-development-support.html, http://www.government.nl/accessibility, http://www.government.nl/help, http://www.government.nl/international, http://subscribe.government.nl, http://www.government.nl/government, http://www.government.nl/documents-and-publications/forms/2015/04/07/administrative-rules-and-model-application-form-srhr-partnership-fund.html, http://www.government.nl/copyright])
MC(crawler): 2015-05-07 12:59:47 INFO -         DOCUMENT_IMPORTED: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=5&amp;page=4 (Subject: com.norconex.importer.response.ImporterResponse@333c8b1f)
MC(crawler): 2015-05-07 12:59:47 INFO -    DOCUMENT_COMMITTED_ADD: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=5&amp;page=4 (Subject: SolrCommitter [solrURL=http://localhost:8983/solr/MyRepository, updateUrlParams={update.chain=langid}, solrServerFactory=DefaultSolrServerFactory [server=com.norconex.committer.solr.SolrCommitter$CommitterSolrClient@2422ddad], com.norconex.committer.solr.SolrCommitter@38bbca65[queueSize=100,docCount=206798,queue=com.norconex.committer.core.impl.FileSystemCommitter@113f6dc4[directory=D:/CRAWLER-MyRepository/collectors/CollectorA/work//queue],commitBatchSize=10,maxRetries=2,maxRetryWait=5000,operations=[],docCount=0,targetReferenceField=id,sourceReferenceField=document.reference,keepSourceReferenceField=false,targetContentField=content,sourceContentField=<null>,keepSourceContentField=false]])
MC(crawler): 2015-05-07 12:59:48 INFO -          DOCUMENT_FETCHED: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=4&amp;page=3 (Subject: com.norconex.collector.http.fetch.impl.GenericDocumentFetcher@2811e442)
MC(crawler): 2015-05-07 12:59:48 INFO -       CREATED_ROBOTS_META: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=4&amp;page=3 (Subject: none)
MC(crawler): 2015-05-07 12:59:48 INFO -           REJECTED_FILTER: http://www.rijksoverheid.nl/papiamento/ (Subject: none)
MC(crawler): 2015-05-07 12:59:48 INFO -           REJECTED_FILTER: http://subscribe.government.nl (Subject: none)
MC(crawler): 2015-05-07 12:59:48 INFO -           REJECTED_FILTER: http://www.rijksoverheid.nl/ (Subject: none)
MC(crawler): 2015-05-07 12:59:48 INFO -           REJECTED_FILTER: http://www.rijksoverheid.nl/papiamentu/ (Subject: none)
MC(crawler): 2015-05-07 12:59:48 INFO -           REJECTED_FILTER: http://roimg.nl/presentation/images/logotype-government-of-the-netherlands-en.png (Subject: none)
MC(crawler): 2015-05-07 12:59:48 INFO -            URLS_EXTRACTED: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=4&amp;page=3 (Subject: [http://www.government.nl/documents-and-publications/leaflets/2015/04/28/iob-iob-evaluation-newsletter-14-08-renewable-energy-in-rwanda-benefits-for-households-yet-limited-impact-on-co2-reduction.html, http://www.rijksoverheid.nl/papiamento/, http://www.government.nl/documents-and-publications/leaflets/2014/11/07/iob-iob-evaluation-newsletter-14-07-a-strategic-approach-dutch-coalitionbuilding-and-the-multi-bi-approach-to-influencing-eu-decision-making-2008-2012.html, http://www.government.nl/privacy, http://roimg.nl/presentation/images/logotype-government-of-the-netherlands-en.png, http://www.government.nl/ministries, http://www.government.nl/documents-and-publications/annual-reports/2010/08/01/annual-social-report-2010.html, http://www.government.nl/documents-and-publications/reports/2014/09/1/iob-a-strategic-approach-dutch-coalition-building-and-the-multi-bi-approach-in-the-context-of-eu-decision-making-2008-2012-summary-main-findings-and-issues-for-consideration.html, http://www.government.nl/documents-and-publications/parliamentary-documents/2015/02/23/state-of-the-european-union-2015.html, http://www.government.nl/cookies, http://www.government.nl/sitemap, http://www.government.nl/contact, http://www.government.nl/issues, http://www.government.nl/documents-and-publications, http://www.government.nl/news, http://www.rijksoverheid.nl/, http://www.rijksoverheid.nl/papiamentu/, http://www.government.nl/documents-and-publications/reports/2015/04/28/iob-riding-the-wave-of-sustainable-commodity-sourcing-review-of-the-sustainable-trade-initiative-idh-2008-2013.html, http://www.government.nl/, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=4&amp;amp;page=3&amp;page=16, http://www.government.nl/documents-and-publications/reports/2015/04/28/iob-access-to-energy-in-rwanda-impact-evaluation-of-activities-supported-by-the-dutch-promoting-renewable-energy-programme.html, http://www.government.nl/rss, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=4&amp;amp;page=3&amp;page=6, http://www.government.nl/documents-and-publications/reports/2015/04/14/iob-evaluation-of-dutch-development-cooperation-in-the-palestinian-territories-2008-2014.html, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=4&amp;amp;page=3&amp;page=2, http://www.government.nl/documents-and-publications/reports/2015/01/01/iob-impact-of-ending-dutch-development-support.html, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=4&amp;amp;page=3&amp;page=3, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=4&amp;amp;page=3&amp;page=4, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=4&amp;amp;page=3&amp;page=5, http://www.government.nl/accessibility, http://www.government.nl/help, http://www.government.nl/international, http://subscribe.government.nl, http://www.government.nl/government, http://www.government.nl/documents-and-publications/forms/2015/04/07/administrative-rules-and-model-application-form-srhr-partnership-fund.html, http://www.government.nl/copyright])
MC(crawler): 2015-05-07 12:59:48 INFO -         DOCUMENT_IMPORTED: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=4&amp;page=3 (Subject: com.norconex.importer.response.ImporterResponse@3017868f)
MC(crawler): 2015-05-07 12:59:48 INFO -    DOCUMENT_COMMITTED_ADD: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=4&amp;page=3 (Subject: SolrCommitter [solrURL=http://localhost:8983/solr/MyRepository, updateUrlParams={update.chain=langid}, solrServerFactory=DefaultSolrServerFactory [server=com.norconex.committer.solr.SolrCommitter$CommitterSolrClient@2422ddad], com.norconex.committer.solr.SolrCommitter@38bbca65[queueSize=100,docCount=206799,queue=com.norconex.committer.core.impl.FileSystemCommitter@113f6dc4[directory=D:/CRAWLER-MyRepository/collectors/CollectorA/work//queue],commitBatchSize=10,maxRetries=2,maxRetryWait=5000,operations=[],docCount=0,targetReferenceField=id,sourceReferenceField=document.reference,keepSourceReferenceField=false,targetContentField=content,sourceContentField=<null>,keepSourceContentField=false]])
MC(crawler): 2015-05-07 12:59:49 INFO -          DOCUMENT_FETCHED: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=4&amp;page=16 (Subject: com.norconex.collector.http.fetch.impl.GenericDocumentFetcher@2811e442)
MC(crawler): 2015-05-07 12:59:49 INFO -       CREATED_ROBOTS_META: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=4&amp;page=16 (Subject: none)
MC(crawler): 2015-05-07 12:59:49 INFO -           REJECTED_FILTER: http://www.rijksoverheid.nl/papiamento/ (Subject: none)
MC(crawler): 2015-05-07 12:59:49 INFO -           REJECTED_FILTER: http://subscribe.government.nl (Subject: none)
MC(crawler): 2015-05-07 12:59:49 INFO -           REJECTED_FILTER: http://www.rijksoverheid.nl/ (Subject: none)
MC(crawler): 2015-05-07 12:59:49 INFO -           REJECTED_FILTER: http://www.rijksoverheid.nl/papiamentu/ (Subject: none)
MC(crawler): 2015-05-07 12:59:49 INFO -           REJECTED_FILTER: http://roimg.nl/presentation/images/logotype-government-of-the-netherlands-en.png (Subject: none)
MC(crawler): 2015-05-07 12:59:49 INFO -            URLS_EXTRACTED: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=4&amp;page=16 (Subject: [http://www.government.nl/documents-and-publications/leaflets/2015/04/28/iob-iob-evaluation-newsletter-14-08-renewable-energy-in-rwanda-benefits-for-households-yet-limited-impact-on-co2-reduction.html, http://www.rijksoverheid.nl/papiamento/, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=4&amp;amp;page=16&amp;page=16, http://www.government.nl/documents-and-publications/leaflets/2014/11/07/iob-iob-evaluation-newsletter-14-07-a-strategic-approach-dutch-coalitionbuilding-and-the-multi-bi-approach-to-influencing-eu-decision-making-2008-2012.html, http://www.government.nl/privacy, http://roimg.nl/presentation/images/logotype-government-of-the-netherlands-en.png, http://www.government.nl/ministries, http://www.government.nl/documents-and-publications/annual-reports/2010/08/01/annual-social-report-2010.html, http://www.government.nl/documents-and-publications/reports/2014/09/1/iob-a-strategic-approach-dutch-coalition-building-and-the-multi-bi-approach-in-the-context-of-eu-decision-making-2008-2012-summary-main-findings-and-issues-for-consideration.html, http://www.government.nl/documents-and-publications/parliamentary-documents/2015/02/23/state-of-the-european-union-2015.html, http://www.government.nl/cookies, http://www.government.nl/sitemap, http://www.government.nl/contact, http://www.government.nl/issues, http://www.government.nl/documents-and-publications, http://www.government.nl/news, http://www.rijksoverheid.nl/, http://www.rijksoverheid.nl/papiamentu/, http://www.government.nl/documents-and-publications/reports/2015/04/28/iob-riding-the-wave-of-sustainable-commodity-sourcing-review-of-the-sustainable-trade-initiative-idh-2008-2013.html, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=4&amp;amp;page=16&amp;page=3, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=4&amp;amp;page=16&amp;page=2, http://www.government.nl/, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=4&amp;amp;page=16&amp;page=6, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=4&amp;amp;page=16&amp;page=5, http://www.government.nl/documents-and-publications/reports/2015/04/28/iob-access-to-energy-in-rwanda-impact-evaluation-of-activities-supported-by-the-dutch-promoting-renewable-energy-programme.html, http://www.government.nl/rss, http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;page=3&amp;amp;amp;page=4&amp;amp;page=16&amp;page=4, http://www.government.nl/documents-and-publications/reports/2015/04/14/iob-evaluation-of-dutch-development-cooperation-in-the-palestinian-territories-2008-2014.html, http://www.government.nl/documents-and-publications/reports/2015/01/01/iob-impact-of-ending-dutch-development-support.html, http://www.government.nl/accessibility, http://www.government.nl/help, http://www.government.nl/international, http://subscribe.government.nl, http://www.government.nl/government, http://www.government.nl/documents-and-publications/forms/2015/04/07/administrative-rules-and-model-application-form-srhr-partnership-fund.html, http://www.government.nl/copyright])
MC(crawler): 2015-05-07 12:59:49 INFO -         DOCUMENT_IMPORTED: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=4&amp;page=16 (Subject: com.norconex.importer.response.ImporterResponse@5e2bf9c9)
MC(crawler): 2015-05-07 12:59:49 INFO - Max queue size reached (100). Committing
MC(crawler): 2015-05-07 12:59:49 INFO - Committing 100 files
MC(crawler): 2015-05-07 12:59:49 INFO - Sending 10 documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:49 INFO - Done sending documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:49 INFO - Sending 10 documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:50 INFO - Done sending documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:50 INFO - Sending 10 documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:51 INFO - Done sending documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:51 INFO - Sending 10 documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:51 INFO - Done sending documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:51 INFO - Sending 10 documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:52 INFO - Done sending documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:52 INFO - Sending 10 documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:53 INFO - Done sending documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:53 INFO - Sending 10 documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:53 INFO - Done sending documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:53 INFO - Sending 10 documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:54 INFO - Done sending documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:54 INFO - Sending 10 documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:54 INFO - Done sending documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:54 INFO - Sending 10 documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:55 INFO - Done sending documents to Solr for update/deletion.
MC(crawler): 2015-05-07 12:59:55 INFO -    DOCUMENT_COMMITTED_ADD: http://www.government.nl/documents-and-publications?keyword=IOB&amp;amp;amp;amp;amp;amp;amp;amp;form-department=BZ&amp;amp;amp;amp;amp;amp;amp;amp;form-information-type=reports&amp;amp;amp;amp;amp;amp;amp;page=5&amp;amp;amp;amp;amp;amp;page=6&amp;amp;amp;amp;amp;page=2&amp;amp;amp;amp;page=5&amp;amp;amp;page=3&amp;amp;page=4&amp;page=16 (Subject: SolrCommitter [solrURL=http://localhost:8983/solr/MyRepository, updateUrlParams={update.chain=langid}, solrServerFactory=DefaultSolrServerFactory [server=com.norconex.committer.solr.SolrCommitter$CommitterSolrClient@2422ddad], com.norconex.committer.solr.SolrCommitter@38bbca65[queueSize=100,docCount=206800,queue=com.norconex.committer.core.impl.FileSystemCommitter@113f6dc4[directory=D:/CRAWLER-MyRepository/collectors/CollectorA/work//queue],commitBatchSize=10,maxRetries=2,maxRetryWait=5000,operations=[],docCount=0,targetReferenceField=id,sourceReferenceField=document.reference,keepSourceReferenceField=false,targetContentField=content,sourceContentField=<null>,keepSourceContentField=false]])
essiembre commented 9 years ago

2.2.0 was just released. Can you confirm you still have this issue?

csaezl commented 9 years ago

It seems to work better than before