Entities per review 0.0 and Entities per claim 0.0

Slashdacoda commented 3 years ago

(base) C:\Users\schue\Desktop\Repos\claimskg_generator> C: && cd C:\Users\schue\Desktop\Repos\claimskg_generator && cmd /C "C:\Users\schue\anaconda3\python.exe c:\Users\schue\.vscode\extensions\ms-python.pythpython-2021.3.680753044\pythonFiles\lib\python\debugpy\launcher 62001 -- c:\Users\schue\Desktop\Repos\claimskg_generator\export.py --input=output_got.csv --output=test.nt --format=nt "
2021-04-06 15:38:23,428 - root - INFO - Loading data...
2021-04-06 15:38:23,437 - root - INFO - Loading thesaurus into ClaimsKG graph... [claimskg/data/thesoz-komplett.xml]
Now loading the dictionary...
Now indexing the dictionary...
100%|████████████████████████████████████████| 8222/8222 [00:25<00:00, 319.37it/s] /s]
2021-04-06 15:41:42,048 - root - INFO - Loading thesaurus into ClaimsKG graph... [claimskg/data/unesco-thesaurus.xml]
Now loading the dictionary...
Now indexing the dictionary...                                                                                                                                                                                ] 
100%|████████████████████████████████████████| 4420/4420 [00:12<00:00, 358.22it/s/s]
2021-04-06 16:53:53,723 - root - INFO - Generating model from CSV data...
100%|████████████████████████████████████████| 34/34 [00:00<00:00, 134.92it/s]
2021-04-06 16:53:53,988 - root - INFO - 
Serializing graph...

Global dataset statistics
ClaimReview,34
CreativeWork,34
Total Number of Entities,0
Keywords,25
Keywords reconciled with TheSoz,0
Keywords reconciled with Unesco,0
Keywords reconciled with DBPedia,0
Keywords reconciled with both TheSoz and DBPedia,0
Keywords reconciled with both UNESCO and DBPedia,0
Entities per review,0.0
Entities per claim,0.0
Keywords per review,0.7352941176470589
Citations per CreativeWork,9.794117647058824
Claims reviews without author,0
CreativeWork Mappings,0
Claims with text,100.0
Claims with author,100.0
Claims with date published,100.0
Claims with at least one citations,97.05882352941177
Claims with at least one entity mention,0.0
Reviews with a headline,100.0
Reviews with at least one keyword,55.88235294117647
Reviews with at least one entity mention,0.0
Claims rated as FALSE under our normalized scale,7
Claims rated as MIXTURE under our normalized scale,0
Claims rated as TRUE under our normalized scale,9
Claims rated as OTHER under our normalized scale,18

Per source site statistics

snopes statistics...
ClaimReview,34
CreativeWork,34
Total Number of Entities,0
Keywords,25
Keywords reconciled with TheSoz,0
Keywords reconciled with Unesco,0
Keywords reconciled with DBPedia,0
Keywords reconciled with both TheSoz and DBPedia,0
Keywords reconciled with both UNESCO and DBPedia,0
Entities per review,0.0
Entities per claim,0.0
Keywords per review,0.7352941176470589
Citations per CreativeWork,9.794117647058824
Claims reviews without author,0
CreativeWork Mappings,0
Claims with text,100.0
Claims with author,100.0
Claims with date published,100.0
Claims with at least one citations,97.05882352941177
Claims with at least one entity mention,0.0
Reviews with a headline,100.0
Reviews with at least one keyword,55.88235294117647
Reviews with at least one entity mention,0.0
Claims rated as FALSE under our normalized scale,7
Claims rated as MIXTURE under our normalized scale,0
Claims rated as TRUE under our normalized scale,9
Claims rated as OTHER under our normalized scale,18
C:\Users\schue\anaconda3\lib\site-packages\rdflib\plugins\serializers\nt.py:29: UserWarning: NTSerializer does not use custom encoding.
  warnings.warn("NTSerializer does not use custom encoding.")
2021-04-06 17:17:41,964 - root - INFO - Writing to test.nt ...

Slashdacoda commented 3 years ago

to track the issue related maintance:

https://github.com/claimskg/claimskg-extractor/blob/13bf2f3b18a81bf6e3f60ebda4a0039b8630f5e1/claim_extractor/__init__.py#L71 https://github.com/claimskg/claimskg-extractor/tree/master/claim_extractor/tagme https://github.com/claimskg/claimskg_generator/blob/02f87d23586732b5798762fae5cc06948d5bf2fd/claimskg/generator/__init__.py

https://github.com/issa-project/entity-fishing-client-python https://nerd.readthedocs.io/en/latest/build.html

Slashdacoda commented 3 years ago

[ ] Install and setup Entity Fishing service API >>> Entity Fishing Documentation
[x] Resolve checkout problem: https://github.com/kermitt2/entity-fishing/issues/120
[ ] Install GROBID and grboid-ner for Entity Fishing Service: https://github.com/kermitt2/grobid > https://grobid.readthedocs.io/en/latest/Install-Grobid/
[ ] Small adaptation in the generator: go from the tagme json format to the entity fishing json format
[ ] Adapt/move from https://github.com/claimskg/claimskg-extractor/tree/master/claim_extractor/annotation to generator
[ ] Test1: entity-fishing-client-python > hearing on http://localhost:8090/service/
[ ] Test 2: with claimskg-extractor snopes CSV keep in mind https://nerd.readthedocs.io/en/latest/build.html

Slashdacoda commented 3 years ago

Related fields for adaption: old (extractor) CSV fieldname > new source from CSV fieldname (generator)

[ ] to clarify: source for JSON (entity-fishing-client-python)
extra_entities_claimReview_claimReviewed > claim_entities > claim? claimReview_claimReviewed
extra_entities_body > body_entities > body? extra_body
extra_entities_keywords > keyword_entities > tags? extra_tags
extra_entities_author > author_entities > author? creativeWork_author_name

Slashdacoda / claimskg_generator

Entities per review 0.0 and Entities per claim 0.0 #2