Triples returned are not correct

sritejakv commented 5 years ago

https://github.com/dice-group/Squirrel/blob/f5e798c50af231fd47f73907bb5b6c64251897b3/squirrel.worker/src/main/java/org/dice_research/squirrel/analyzer/impl/html/scraper/HtmlScraper.java#L138

Triples returned by HtmlScrapper.java have the same subject and predicate.

gsjunior86 commented 5 years ago

@sritejakv , please post the yaml file that you are using and the URL that you want to crawl.

sritejakv commented 5 years ago

For the url - http://prototype-gismontgomery.opendata.arcgis.com/datasets here is the yaml file - https://github.com/abhihc/Squirrel/blob/enhanced_data_portal_crawling/squirrel.worker/src/test/resources/html_scraper_analyzer/yaml/prototype_gismontgomery.yaml

here are the list of triples returned by the code in - https://github.com/abhihc/Squirrel/blob/enhanced_data_portal_crawling/squirrel.worker/src/main/java/org/dice_research/squirrel/analyzer/impl/html/scraper/HtmlScraper.java

gsjunior86 commented 5 years ago

@sritejakv

i found some issues in your file:

1 - The indentation was wrong.

2 - You forgot to specify the subject in the yaml file. Check it:

file_descriptor:
 check:
  domain: prototype-gismontgomery.opendata.arcgis.com
  ignore-request: true

 search-result-page:
  regex: datasets?q=
  resources:
   "$uri":
    "http://projekt-opal.de/dataset#link": .card-title a

 detail-page:
  regex: datasets
  resources:
   "$uri":
    "http://purl.org/dc/terms/title": div#main-message
    "http://purl.org/dc/terms/description": span#dataset-description
    "http://purl.org/dc/terms/publisher": span[iatemprop="author"]
    "http://purl.org/dc/terms/issued": span[itemprop="datePublished"]
    "http://purl.org/dsnotify/vocab/eventset/sourceDataset": ul#dataset-meta-list li:eq(0) a
    "http://www.w3.org/ns/dcat#accessURL": ul#dataset-meta-list li:eq(1) a
    "http://purl.org/dc/terms/license": ul#dataset-meta-list li:eq(2)
    "http://purl.org/dc/terms/modified": li[itemprop="dateModified"]
    "http://www.w3.org/ns/dcat#downloadURL": .dl-links a

The $Uri variable stands for the current URI.

3 - This website loads additional html data through ajax calls. This is a feature not yet available in the scraper. Because of that, it will not be able to capture the pagination links, because it is loaded after. Btw, the pagination is not working. But the crawling for the first page works fine for me.

4 - i removed redundant data from your file

sritejakv commented 5 years ago

@gsjunior86 Did you run the above yaml file using the HtmlScrapper.java file which is linked in the previous comment?

sritejakv commented 5 years ago

@gsjunior86 Thank you for the comments.

dice-group / Squirrel

Triples returned are not correct #108