search

amir-jakoby / crawler-commons

Automatically exported from code.google.com/p/crawler-commons
0 stars 0 forks source link

Add Fetch Report to FetchedResult #50

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
We have loads of fine grained method available to us via FetchedResult.
I think it would be really cool however if we were able to print a report of 
the FetchedResult including some timing statistics as well as an account of all 
page metadata, it't content, etc. 
Just as the Nutch WebTableReader does

https://svn.apache.org/repos/asf/nutch/branches/2.x/src/java/org/apache/nutch/cr
awl/WebTableReader.java

Patch coming up.

Original issue reported on code.google.com by lewis.mc...@gmail.com on 28 Aug 2014 at 4:35

GoogleCodeExporter commented 8 years ago

Original comment by lewis.mc...@gmail.com on 28 Aug 2014 at 4:36

GoogleCodeExporter commented 8 years ago 
Should it be overriding a toString method ?

Original comment by avrah...@gmail.com on 28 Aug 2014 at 5:32

GoogleCodeExporter commented 8 years ago 
Hi Avi no it should not but I will make this better and reload a patch. Thanks

Original comment by lewis.mc...@gmail.com on 10 Oct 2014 at 4:00

GoogleCodeExporter commented 8 years ago 
Initial patch before I loose it.

Original comment by lewis.mc...@gmail.com on 10 Oct 2014 at 4:02

Attachments:

GoogleCodeExporter commented 8 years ago 
Example of how to fully populate the object then print the report.

Original comment by lewis.mc...@gmail.com on 10 Oct 2014 at 4:05

Attachments:

GoogleCodeExporter commented 8 years ago 
I have checked it out and it is very nice!

I think that your Test file should also be included in the commit. (currently 
it is missing from the patch you committed).

I have some minor comments:
* The line: report.append("    PayLoad       : PAYLOAD_OBJECT) -- Shouldn't 
print the payload as it doesn't have a toString()

* (very minor) I think that the line with "End of report" should be after the 
ending stars line ("*********")

* In the test file the last line should print the report method: 
System.out.println(result.report());

I have done those changes and added your test file into one patch which is 
attached

Original comment by avrah...@gmail.com on 19 Oct 2014 at 3:05

Attachments:

GoogleCodeExporter commented 8 years ago 
Fantastic Avi, 

lmcgibbn@LMC-032857 /usr/local/crawler-commons/trunk(master) $ svn ci -m "Issue 
50:  Add Fetch Report to FetchedResult "
Sending        CHANGES.txt
Sending        src/main/java/crawlercommons/fetcher/FetchedResult.java
Transmitting file data ..
Committed revision 150.

Original comment by lewis.mc...@gmail.com on 19 Oct 2014 at 7:00

GoogleCodeExporter commented 8 years ago 
lmcgibbn@LMC-032857 /usr/local/crawler-commons/trunk(master) $ svn ci -m "Issue 
50:  Add Fetch Report to FetchedResult"
Adding         src/test/java/crawlercommons/fetcher/FetchedResultTest.java
Transmitting file data .
Committed revision 151.

Original comment by lewis.mc...@gmail.com on 20 Oct 2014 at 2:07