[Sitemaps] SiteMapParser Tika detection doesn't work well on some cases

amir-jakoby / crawler-commons

Automatically exported from code.google.com/p/crawler-commons

0 stars 0 forks source link

[Sitemaps] SiteMapParser Tika detection doesn't work well on some cases #47

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago

When using the parse method which gets only a sitemap URL, we use Tika to 
detect the Mime type.

On some cases, the detection is bad.

We need to use a better Tika detection.

Use:
new Tika().detect(URL)

Instead of the current:
new Tika().detect(bytes)

Original issue reported on code.google.com by avrah...@gmail.com on 12 Jul 2014 at 8:29

Blocked on: #40

GoogleCodeExporter commented 8 years ago

Will submit the patch after submission of issue40. (same file touched)

Original comment by avrah...@gmail.com on 12 Jul 2014 at 8:30

GoogleCodeExporter commented 8 years ago

Scenario where the bug can be reproduced:
Run the SitemapParser Tool on the following URL:
http://www.amazon.com/sitemap_video.xml

Original comment by avrah...@gmail.com on 12 Jul 2014 at 8:32

Changed title: [Sitemaps] SiteMapParser Tika detection doesn't work well on some cases

GoogleCodeExporter commented 8 years ago

[deleted comment]

GoogleCodeExporter commented 8 years ago

New parse(Url url) method introduced on issue39

Using the above method in SitemapTool: issue43 (not yet committed to svn)

Original comment by avrah...@gmail.com on 14 Jul 2014 at 1:21

GoogleCodeExporter commented 8 years ago

new Tika().detect(URL) -- Will solve the mentioned problem.

BUT it will cause out library to fetch the sitemap twice.

A better solution should be sought.
Maybe use new Tika().detect(bytes, filename);

Original comment by avrah...@gmail.com on 16 Jul 2014 at 5:24

GoogleCodeExporter commented 8 years ago

Original comment by avrah...@gmail.com on 18 Jul 2014 at 8:05

Now blocked on: #40

GoogleCodeExporter commented 8 years ago

Let's instanciate the Tika instance only once and reuse it - otherwise we have 
to reload the Tika config everytime which is definitely not needed. (Julien)

Original comment by avrah...@gmail.com on 1 Aug 2014 at 3:51

GoogleCodeExporter commented 8 years ago

I will begin working on this one

Original comment by avrah...@gmail.com on 6 Aug 2014 at 7:10

GoogleCodeExporter commented 8 years ago

Attached is a patch with the required optimization.

Now the Tika detection is being called with the byte array + filename

The Tika object is being instantiated only once

Original comment by avrah...@gmail.com on 18 Aug 2014 at 8:10

Added labels: Type-Enhancement
Removed labels: Type-Defect

Attachments:

Issue_47.patch

GoogleCodeExporter commented 8 years ago

+ 1 ship it Thanks

Original comment by lewis.mc...@gmail.com on 18 Aug 2014 at 9:22

GoogleCodeExporter commented 8 years ago

Shipped in revision: r134

Original comment by avrah...@gmail.com on 19 Aug 2014 at 7:10

Changed state: Fixed