Norconex / collector-core

Collector-related code shared between different collector implementations
http://www.norconex.com/collectors/collector-core/
Apache License 2.0
7 stars 15 forks source link

How to iterate the Crawl log file? #27

Closed nancygoyal1 closed 4 years ago

nancygoyal1 commented 4 years ago

Is there any way the log files mvstore or db.mv.db can be analysed in more readable format.? Can these file be iterated through code so that the information required can be pulled out to store in a different file? I am able to get the examples for creating these files online but not how to iterate, if there is a way can you please share the reference or details of how to do it. Thanks in advance.

essiembre commented 4 years ago

I am assuming if you closed you found a way to do so. For others wondering about this, you need to use the MVStore API. You can find code samples from H2 website here: http://www.h2database.com/html/mvstore.html#example_code

You can also look how it is used by Collector Core: https://github.com/Norconex/collector-core/blob/master/norconex-collector-core/src/main/java/com/norconex/collector/core/data/store/impl/mvstore/MVStoreCrawlDataStore.java