WolfgangFahl / ProceedingsTitleParser

Shallow Semantic Parser to extract metadata from scientific proceedings titles
Apache License 2.0
3 stars 1 forks source link

regression in caching wikidata cities #54

Closed WolfgangFahl closed 3 years ago

WolfgangFahl commented 3 years ago
ERROR: testWikiDataCities (tests.test_Locations.TestLocations)
test getting cities(human settlements to be precise)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/wf/Documents/pyworkspace/ProceedingsTitleParser/tests/test_Locations.py", line 131, in testWikiDataCities
    cm.fromCache()
  File "/Users/wf/Documents/pyworkspace/ProceedingsTitleParser/ptp/location.py", line 41, in fromCache
    self.cityList=super().fromCache()
  File "/Users/wf/Documents/pyworkspace/ProceedingsTitleParser/storage/entity.py", line 179, in fromCache
    listOfDicts=self.fromStore()
  File "/Users/wf/Documents/pyworkspace/ProceedingsTitleParser/storage/entity.py", line 221, in fromStore
    listOfDicts=sqlDB.query(sqlQuery)
  File "/Users/wf/Library/Python/3.8/lib/python/site-packages/lodstorage/sql.py", line 198, in query
    for row in query:
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/sqlite3/dbapi2.py", line 67, in convert_timestamp
    datepart, timepart = val.split(b" ")
ValueError: not enough values to unpack (expected 2, got 1)
WolfgangFahl commented 3 years ago

There are invalid date entries:

select date from city_wikidata where 
date>"2200-12-31"
or date<"1000-01-01"
order by date desc
date
b9
b8
b7
b6
b5
b4
b3
b2
b19
b17
b16
b15
b14
b13
b12
b11
b10
b1
b0
0700-01-01 00:00:00
0622-01-01 00:00:00
0216-01-01 00:00:00
0213-01-01 00:00:00
0200-01-01 00:00:00
0200-01-01 00:00:00
0200-01-01 00:00:00
0198-01-01 00:00:00
0017-10-29 00:00:00
0017-10-29 00:00:00
0017-10-29 00:00:00
0017-10-29 00:00:00
0017-10-29 00:00:00
0017-10-29 00:00:00
0017-10-29 00:00:00
0017-10-29 00:00:00
0017-10-29 00:00:00
0017-10-29 00:00:00
0017-10-29 00:00:00
0017-10-29 00:00:00
0002-01-01 00:00:00
0002-01-01 00:00:00
WolfgangFahl commented 3 years ago

try work-around by fixing as:

update city_wikidata set date=null where 
date>"2200-12-31"
or date<"1000-01-01"