search

opendata / Open-Data-Needs

An ongoing effort to catalog the holes in the open data ecosystem. [RETIRED]
15 stars 0 forks source link

PDF content extraction #3

Open waldoj opened 10 years ago

waldoj commented 10 years ago

This remains a horrible slog. We have a lot of tools that are various shades of not good. The one bright light is @jsfenfen's What World Where. That presents the first novel path forward in a long time: using hOCR page contour data and using GIS tools to navigate and address individual units of data within.

Basically, this is not a solved problem, but it should be for the simplest ~90% of documents. We need the tool that does that.

knowtheory commented 10 years ago

We've got a plan and have roughly specced out what needs to happen for this problem, and how to roll it into https://github.com/documentcloud/docsplit but don't have the capacity to implement on it right now. Would be happy to chat with folks about what our intended plan looks like.

waldoj commented 10 years ago

Whoa. Awesome. Let's talk about to make this happen! I'm basically wide open for the rest of today, tomorrow, and the first half of Friday.

waldoj commented 10 years ago

For the record, @knowtheory and I just spent half an hour talking about Docsplit. There is, indeed, some work to be done that could make it a lot easier to use, potentially both a much nicer method of extracting text from PDFs and a platform to continue to advance the state of the art, insofar as it uses hOCR page contour data and thus slots into that tiny, growing data ecosystem. We'll keep talking, to see how US ODI can help to improve that.

waldoj commented 10 years ago

FYI, @knowtheory, I spent a while trying to get Docsplit running, but I haven't had any luck. Installing poppler doesn't work—I don't know if it's a Homebrew problem or what.

FYI:

$ brew install poppler
==> Downloading http://poppler.freedesktop.org/poppler-0.24.5.tar.xz
Already downloaded: /Library/Caches/Homebrew/poppler-0.24.5.tar.xz
==> ./configure --prefix=/usr/local/Cellar/poppler/0.24.5 --enable-xpdf-headers 
==> make install
clang: error: unknown argument: '-fno-check-new' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
make[1]: *** [GooList.lo] Error 1
make[1]: *** [GooHash.lo] Error 1
make: *** [install-recursive] Error 1

I've tried brew update, but that eventually errors out with this:

Aborting
Error: Failure while executing: git pull -q origin refs/heads/master:refs/remotes/origin/master

All of which is to say that I am now thoroughly convinced that more people would be able to extract text from PDFs if the install process was far, far easier. :)