search

WRI-Cities / static-GTFS-manager

GUI interface for creating, editing, exporting of static GTFS data for a public transit authority
GNU General Public License v3.0
147 stars 46 forks source link

delete by ID : when everything gets deleted.. #106

Closed answerquest closed 6 years ago

answerquest commented 6 years ago

console output: 

deleteByKey GET call
Deleted 1 rows with fare_id="Reg" in table: fare_attributes<br>
Deleted 5 rows with fare_id="Reg" in table: fare_rules<br>
deleteByKey GET call took 0.46 seconds.

listAll GET call
readTableDB: Loaded stops, 111 records
readTableDB: Loaded stops, 111 records
readTableDB: Loaded routes, 5 records
readTableDB: Loaded trips, 1318 records
ERROR:tornado.application:Uncaught exception GET /API/listAll (::1)
HTTPServerRequest(protocol='http', host='localhost:5000', method='GET', uri='/API/listAll', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
  File "/mnt/STUFF/py36/lib/python3.6/site-packages/tornado/web.py", line 1590, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "GTFSManager.py", line 1083, in get
    zoneCollector.update( readColumnDB('fare_rules','origin_id') )
  File "<string>", line 1111, in readColumnDB
  File "<string>", line 405, in readTableDB
  File "/mnt/STUFF/py36/lib/python3.6/site-packages/pandas/io/pytables.py", line 394, in read_hdf
    return store.select(key, auto_close=auto_close, **kwargs)
  File "/mnt/STUFF/py36/lib/python3.6/site-packages/pandas/io/pytables.py", line 723, in select
    raise KeyError('No object named %s in the file' % key)
KeyError: 'No object named df in the file'
ERROR:tornado.access:500 GET /API/listAll (::1) 478.55ms

fareAttributes GET call
ERROR:tornado.application:Uncaught exception GET /API/fareAttributes (::1)
HTTPServerRequest(protocol='http', host='localhost:5000', method='GET', uri='/API/fareAttributes', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
  File "/mnt/STUFF/py36/lib/python3.6/site-packages/tornado/web.py", line 1590, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "GTFSManager.py", line 174, in get
    fareAttributesJson = readTableDB('fare_attributes').to_json(orient='records', force_ascii=False)
  File "<string>", line 405, in readTableDB
  File "/mnt/STUFF/py36/lib/python3.6/site-packages/pandas/io/pytables.py", line 394, in read_hdf
    return store.select(key, auto_close=auto_close, **kwargs)
  File "/mnt/STUFF/py36/lib/python3.6/site-packages/pandas/io/pytables.py", line 723, in select
    raise KeyError('No object named %s in the file' % key)
KeyError: 'No object named df in the file'
ERROR:tornado.access:500 GET /API/fareAttributes (::1) 19.48ms

fareRules GET call
ERROR:tornado.application:Uncaught exception GET /API/fareRules (::1)
HTTPServerRequest(protocol='http', host='localhost:5000', method='GET', uri='/API/fareRules', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
  File "/mnt/STUFF/py36/lib/python3.6/site-packages/tornado/web.py", line 1590, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "GTFSManager.py", line 207, in get
    fareRulesSimpleJson = readTableDB('fare_rules').to_json(orient='records', force_ascii=False)
  File "<string>", line 405, in readTableDB
  File "/mnt/STUFF/py36/lib/python3.6/site-packages/pandas/io/pytables.py", line 394, in read_hdf
    return store.select(key, auto_close=auto_close, **kwargs)
  File "/mnt/STUFF/py36/lib/python3.6/site-packages/pandas/io/pytables.py", line 723, in select
    raise KeyError('No object named %s in the file' % key)
KeyError: 'No object named df in the file'
ERROR:tornado.access:500 GET /API/fareRules (::1) 41.74ms

fareRulesPivoted GET call
ERROR:tornado.application:Uncaught exception GET /API/fareRulesPivoted (::1)
HTTPServerRequest(protocol='http', host='localhost:5000', method='GET', uri='/API/fareRulesPivoted', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
  File "/mnt/STUFF/py36/lib/python3.6/site-packages/tornado/web.py", line 1590, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "GTFSManager.py", line 240, in get
    fareRulesDf = readTableDB('fare_rules')
  File "<string>", line 405, in readTableDB
  File "/mnt/STUFF/py36/lib/python3.6/site-packages/pandas/io/pytables.py", line 394, in read_hdf
    return store.select(key, auto_close=auto_close, **kwargs)
  File "/mnt/STUFF/py36/lib/python3.6/site-packages/pandas/io/pytables.py", line 723, in select
    raise KeyError('No object named %s in the file' % key)
KeyError: 'No object named df in the file'
ERROR:tornado.access:500 GET /API/fareRulesPivoted (::1) 40.34ms
answerquest commented 6 years ago

There's multiple approaches to tackle this:

  1. At the deleting function, if the remaining table after removal has no rows left, then delete the file entirely. But this can get problematic with chunks. Crikey, we need to find ways to create new chunks when lot of data addition happens.
  2. If there are multiple places where deleting of entries can occur, then we'd have to code this in at all places.
  3. Alternatively, when reading the file there could be a pre-check or try-except block that returns an empty dataframe for KeyError exceptions.
answerquest commented 6 years ago

Fixed by putting all db read commands in try-except blocks.

pd.HDFStore:

hdf = pd.HDFStore(dbFolder + tablename + '.h5')
try:
    count = hdf.get_storer('df').nrows
except KeyError as e:
    count = 0
    logmessage('Note: {} does not have any data.'.format(tablename + '.h5'))
hdf.close()

pd.read_hdf:

try:
    df = pd.read_hdf(dbFolder + h5File,'df').fillna('').astype(str)
except KeyError as e:
    df = pd.DataFrame()
    logmessage('Note: {} does not have any data.'.format(h5File))

Will update in code with v2.1.0