Page Scraping Enhancer - Githubissues

CINERGI / DDH-Work

0 stars 0 forks source link

Page Scraping Enhancer #26

Open valentinedwv opened 6 years ago

valentinedwv commented 6 years ago

Grab URL from form page, scrape page for text for submitting to pipeline

valentinedwv commented 6 years ago

Creating a service using sumy https://github.com/miso-belica/sumy and spyne

Thoughts:

two endpoints, get_summary_url, get_summary

get_summary_url (url=,method='LexRank',sentences_count=10,keywords=false)

[ ] uses sumy to get a summary from the url
[ ] calls scigraph on the aggreated text to get keywords

get_summary (text=,method='LexRank',sentences_count=10,keywords=false,isHtml=false)

[ ] uses sumy to get a summary from the string
[ ] calls scigraph on the aggreated text to get keywords

returns:

{ 
summary:[string,string]
keywords:[{string,reference},{string,reference}]
}

Deep dive: Attempts to identify urls on page. retrieves if appropriate

known service (OGC, Swagger description)
known format (CSV, TSV, Excel)