search

cmrivers / ebola

Data for the 2014 ebola outbeak in West Africa
Other
264 stars 144 forks source link

Investigate automated table pdf scraping with pdftables #29

Open samccone opened 9 years ago

samccone commented 9 years ago

https://blog.scraperwiki.com/2013/07/pdftables-a-python-library-for-getting-tables-out-of-pdf-files/

chendaniely commented 9 years ago

wow this is pretty cool.

the last time I was did table scraping from pdf's I redirected less output to a file and went line by line using regex. It was not pretty... nor fun..

donpdonp commented 9 years ago

what is happening currently for getting new data from PDFs into _data/ csv files? is there any automation and are those scripts in the repo?

donpdonp commented 9 years ago

I just tried Tabula after a tip-off from someone at CodeForPortland last night. http://tabula.nerdpower.org/

The tables are manually selected using the mouse, but the data in the tables are read into csv automatically. It seems to work fairly well. I'm not sure if thats any better than whats being done already.

srinivvenkat commented 9 years ago

I think these tools work fine, as long as the text in the PDF is readable. I don't think it would help if they are scanned pages (which, at least in the guinea dataset, is a majority). I suspect most such are manually done. (@Caitlin, please clarify)

Tried a crude crowdsourced initiative to do it --> bit.ly/ebola_guinea (A few were digitized but, due to lack of incentives/spreading the 'crowd' didn't come). The key reason for using google doc is to lower the tech barrier (no need of github account), and partial contribution (fill a few columns until you're bored).

If anybody has ideas on extending this idea, shoot!

cmrivers commented 9 years ago

I use a combination of Tabula, a few little scripts to reformat the data, and some manual work. The broad overview is here: http://www.caitlinrivers.com/blog/the-setup-tools-i-use-to-track-emerging-infectious-diseases. I keep meaning to make a screencast of my workflow, but...no time.

Each Liberia file takes about 3-5 minutes from download to upload, and Sierra Leone takes a bit longer.

On Wed, Oct 15, 2014 at 11:28 PM, srinivvenkat notifications@github.com wrote:

I suspect most are manually done. (@Caitlin https://github.com/Caitlin, please clarify)

I think these tools work fine, as long as the text in the PDF is readable. I don't think it would help if they are scanned pages (which, at least in the guinea dataset, is a majority).

Tried a crude crowdsourced initiative to do it --> bit.ly/ebola_guinea (A few were digitized but, due to lack of incentives/spreading the 'crowd' didn't come). The key reason for using google doc is to lower the tech barrier (no need of github account), and partial contribution (fill a few columns until you're bored).

If anybody has ideas on extending this idea, shoot!

— Reply to this email directly or view it on GitHub https://github.com/cmrivers/ebola/issues/29#issuecomment-59309406.[image: Web Bug from https://github.com/notifications/beacon/1302262__eyJzY29wZSI6Ik5ld3NpZXM6QmVhY29uIiwiZXhwaXJlcyI6MTcyOTA0OTI4OCwiZGF0YSI6eyJpZCI6NDQxNzY4NTl9fQ==--d66722f2a22b3ea35926f18e416190cd46d220be.gif]