Trying to save Trip, getting 500: Internal Server Error

[ArtemVN]

Operating system

Windows 10 64-bit

Python version

Python 2.7.16 Running downloaded .exe

Problem description

Trying to add a trip, then save in to DB and getting this

500: Internal Server Error

Console / terminal output if any

static GTFS Manager
Fork it on Github: https://github.com/WRI-Cities/static-GTFS-manager/
Starting up the program, loading dependencies, please wait...

Loaded dependencies, starting static GTFS Manager program.

Open http://localhost:5000 in your Web Browser if you don't see it opening automatically in 5 seconds.

Note: If this is through docker, then it's not going to auto-open in browser, don't wait.
C:\Users\Art\AppData\Local\Temp\_MEI140642\urllib3\connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

pastCommits GET call
pastCommits GET call took 0.01 seconds.

stats GET call
readTableDB: Loaded agency, 1 records
agency              : 1 entries
calendar            : 4 entries
stops               : 40 entries
routes              : 11 entries
Note: trips.h5 does not have any data.
trips               : 0 entries
stop_times          : 0 entries
calendar_dates      : 0 entries
fare_attributes     : 0 entries
fare_rules          : 0 entries
Note: shapes_1.h5 does not have any data.
shapes              : 0 entries
frequencies         : 0 entries
transfers           : 0 entries
feed_info           : 1 entries
stats GET call took 0.31 seconds.
C:\Users\Art\AppData\Local\Temp\_MEI140642\urllib3\connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

tableReadSave GET call for table=routes
readTableDB: Loaded routes, 11 records
C:\Users\Art\AppData\Local\Temp\_MEI140642\urllib3\connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
tableReadSave GET call for table=routes took 0.06 seconds.

calendar GET call
readTableDB: Loaded calendar, 4 records
20190805.0
calendar GET call took 0.06 seconds.

allShapesList GET call
allShapesList GET call took 0.04 seconds.

allStopsKeyed GET call
readTableDB: Loaded stops, 40 records
allStopsKeyed GET call took 0.03 seconds.

tripIdList GET call
Note: trips.h5 does not have any data.
readTableDB: Loaded trips, 0 records
tripIdList GET call took 0.04 seconds.

trips GET call
Note: trips.h5 does not have any data.
readTableDB: table:trips, column:route_id, value:"01001"
readTableDB: Error: column route_id not found in trips.h5. Skipping it.
readTableDB: Loaded trips, 0 records
Got the sequence from sequence db file.
trips GET call took 0.06 seconds.

tableReadSave POST call for table=trips
Note: trips.h5 does not have any data.
ERROR:tornado.application:Uncaught exception POST /API/tableReadSave?pw=program&table=trips&key=route_id&value=01001 (::1)
HTTPServerRequest(protocol='http', host='localhost:5000', method='POST', uri='/API/tableReadSave?pw=program&table=trips&key=route_id&value=01001', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
  File "site-packages\pandas\core\indexes\base.py", line 3078, in get_loc
  File "pandas\_libs\index.pyx", line 140, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 162, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'route_id'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "site-packages\tornado\web.py", line 1590, in _execute
  File "GTFSManager.py", line 1317, in post
    if not decrypt(pw):
  File "<string>", line 531, in replaceTableDB
  File "site-packages\pandas\core\frame.py", line 2688, in __getitem__
  File "site-packages\pandas\core\frame.py", line 2695, in _getitem_column
  File "site-packages\pandas\core\generic.py", line 2489, in _get_item_cache
  File "site-packages\pandas\core\internals.py", line 4115, in get
  File "site-packages\pandas\core\indexes\base.py", line 3080, in get_loc
  File "pandas\_libs\index.pyx", line 140, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 162, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'route_id'
ERROR:tornado.access:500 POST /API/tableReadSave?pw=program&table=trips&key=route_id&value=01001 (::1) 345.79ms

Expected Action

I expected saving it and then go to Stop_times or Frequencies tabs to add them

Details of data you are working with, if relevant

Uploaded shapes in .geojson for both trips of 1 route, no extra columns, filled only the needed columns

What is the size of the dataset : how many lines in stops.txt, routes.txt and stop_times.txt?

40 stops, 11 routes, 0 stop_times, haven't created them yet because of Error

Are you starting a new feed from scratch and haven't filled in any of the other files like calendar.txt?

Yes, starting new feed from scratch, calendar filled up

[answerquest]

Hi @ArtemVN , thanks for posting this. First off, can you confirm if your particular route_id is "01001" ? Apparently the program isn't able to find any route_id "01001".

I suggest using an alphabet letter somewhere in the id to avoid risking number-parsing (though it shouldn't happen: I've tried my best to ensure things are always read as strings not as numbers)

Also, I'm guessing you would have set up a default route pattern already if the frontend allowed you to try creating a new trip.

The .exe is lagging behind the rest of the code - I still have to compile a new version. Can you try getting python3 at your end (anaconda / miniconda) and running through python? See https://github.com/WRI-Cities/static-GTFS-manager/wiki/Running-on-Windows-OS-with-Python-3

[ArtemVN]

Hello @answerquest Yes, my first route_id is "01001", it's auto-generated.

These commands I couldn't run:

`conda create -n gtfs python=3.7
'conda' is not recognized as an internal or external command,
operable program or batch file.
activate gtfs
'activate' is not recognized as an internal or external command,
operable program or batch file.

But I've installed python 3.7.4 and uninstalled the previous version.

Installed dependencies from requirements.txt

Installing collected packages: xmltodict, numpy, tinydb, six, python-dateutil, pytz, pandas, tornado, pycryptodomex, numexpr, tables, chardet, idna, certifi, urllib3, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.7 numexpr-2.6.9 numpy-1.15.2 pandas-0.23.4 pycryptodomex-3.6.6 python-dateutil-2.8.0 pytz-2019.2 requests-2.20.0 six-1.12.0 tables-3.4.4 tinydb-3.11.1 tornado-5.1.1 urllib3-1.24.3 xmltodict-0.11.0
You are using pip version 19.0.3, however version 19.2.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

C:\Users\Art\Documents\Google-TRANSIT\GTFS Manager\static-GTFS-manager-3.4.3>python -m pip install --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/62/ca/94d32a6516ed197a491d17d46595ce58a83cbb2fca280414e57cd86b84dc/pip-19.2.1-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 1.4MB/s
Installing collected packages: pip
  Found existing installation: pip 19.0.3
    Uninstalling pip-19.0.3:
      Successfully uninstalled pip-19.0.3
Successfully installed pip-19.2.1

After this I executed GTFSManager.py and still getting the same error:

C:\Users\Art\Documents\Google-TRANSIT\GTFS Manager\static-GTFS-manager-3.4.3>python GTFSManager.py

static GTFS Manager
Fork it on Github: https://github.com/WRI-Cities/static-GTFS-manager/
Starting up the program, loading dependencies, please wait...

Loaded dependencies, starting static GTFS Manager program.

Open http://localhost:5000 in your Web Browser if you don't see it opening automatically in 5 seconds.

Note: If this is through docker, then it's not going to auto-open in browser, don't wait.

stats GET call
readTableDB: Loaded agency, 1 records
agency              : 1 entries
calendar            : 4 entries
stops               : 95 entries
routes              : 11 entries
Note: trips.h5 does not have any data.
trips               : 0 entries
stop_times          : 0 entries
calendar_dates      : 0 entries
fare_attributes     : 0 entries
fare_rules          : 0 entries
shapes              : 88 entries
frequencies         : 0 entries
transfers           : 0 entries
feed_info           : 1 entries
stats GET call took 0.51 seconds.

pastCommits GET call
pastCommits GET call took 0.01 seconds.

tableReadSave GET call for table=routes
readTableDB: Loaded routes, 11 records
tableReadSave GET call for table=routes took 0.06 seconds.

calendar GET call
readTableDB: Loaded calendar, 4 records
20190805.0
calendar GET call took 0.07 seconds.

allShapesList GET call
allShapesList GET call took 0.03 seconds.

allStopsKeyed GET call
readTableDB: Loaded stops, 95 records
allStopsKeyed GET call took 0.04 seconds.

tripIdList GET call
Note: trips.h5 does not have any data.
readTableDB: Loaded trips, 0 records
tripIdList GET call took 0.04 seconds.

trips GET call
Note: trips.h5 does not have any data.
readTableDB: table:trips, column:route_id, value:"01001"
readTableDB: Error: column route_id not found in trips.h5. Skipping it.
readTableDB: Loaded trips, 0 records
Got the sequence from sequence db file.
trips GET call took 0.05 seconds.

tableReadSave POST call for table=trips
Note: trips.h5 does not have any data.
ERROR:tornado.application:Uncaught exception POST /API/tableReadSave?pw=program&table=trips&key=route_id&value=01001 (::1)
HTTPServerRequest(protocol='http', host='localhost:5000', method='POST', uri='/API/tableReadSave?pw=program&table=trips&key=route_id&value=01001', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
  File "C:\Users\Art\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\indexes\base.py", line 3078, in get_loc
    return self._engine.get_loc(key)
  File "pandas\_libs\index.pyx", line 140, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 162, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'route_id'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Art\AppData\Local\Programs\Python\Python37\lib\site-packages\tornado\web.py", line 1590, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "GTFSManager.py", line 1336, in post
    status = replaceTableDB(table, data, key, value)
  File "<string>", line 531, in replaceTableDB
  File "C:\Users\Art\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\frame.py", line 2688, in __getitem__
    return self._getitem_column(key)
  File "C:\Users\Art\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\frame.py", line 2695, in _getitem_column
    return self._get_item_cache(key)
  File "C:\Users\Art\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\generic.py", line 2489, in _get_item_cache
    values = self._data.get(item)
  File "C:\Users\Art\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\internals.py", line 4115, in get
    loc = self.items.get_loc(item)
  File "C:\Users\Art\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\indexes\base.py", line 3080, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas\_libs\index.pyx", line 140, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 162, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'route_id'
ERROR:tornado.access:500 POST /API/tableReadSave?pw=program&table=trips&key=route_id&value=01001 (::1) 203.87ms

[answerquest]

Hi @ArtemVN , thanks a lot for the detailed feedback. I think I know where the error is happening : In an event of there being no pre-existing trips data, the program is getting stuck for the sake of a route_id query instead of directly going in and saving the data at hand. A try-except block ought to resolve this. I'll try to post a fix on this as soon as possible (may take a day). Thanks a ton for pointing this out!

Regarding conda / pip : I guess you're not using Anaconda / Miniconda to do your python work; rather having original python itself. That's fine, and I'm glad to see you figured your way through.

[ArtemVN]

Hello @answerquest, glad to be helpful, can't wait for the updates.

I just remembered, seems like the error happened after I added these trips for 1 route (onward/backward) some days before (July 20th),then removed them because I didn't understood how to add the trips with frequencies, not with timings. Searched for other software, which turned up not easy to install and there was not enough features. So I came back to here.

At this point I want to request for an ability to choose other route types from Extended GTFS Route Types like Trolleybus Service (800) and Share Taxi Service (717) which are very popular in the cities of Russia including mine.

Yes, it was easier to run original python. Thank you for your support. By the way I'm still able to add and safe stops.

[answerquest]

Hi @ArtemVN ,
Quickly answering for Extended route types: Please see https://github.com/WRI-Cities/static-GTFS-manager/issues/125#issuecomment-432276784
You can edit config/settings.js and put in all the route types you want at the const route_type_options = line.

after I added these trips for 1 route (onward/backward) some days before (July 20th),then removed them

Ok this helps. (Thinking aloud..) db/trips.h5 exists, but is empty. So it passes the "if file exists" check. But upon loading, it's an empty table, hence the query by route_id fails and causes an exception. This can very well happen with all the tables; it'll be best to build in a generic empty-table check.

[answerquest]

Hi @ArtemVN I've just updated the code. Key change is here: https://github.com/WRI-Cities/static-GTFS-manager/blob/master/GTFSserverfunctions.py#L528

        try:
            df = pd.read_hdf(dbFolder+h5File).fillna('').astype(str)
            oldLen = len( df[ df[key] == str(value)])
            df.query(key + ' != "' + str(value) + '"', inplace=True)
        except (KeyError, ValueError) as e:
            df = pd.DataFrame()
            logmessage('Note: {} does not have any data.'.format(h5File))
            oldLen = 0

Moved the length and query lines into the already existing try-except block. See diff here

Please sync in the new code, give it a shot again and let me know if you are still facing the same error. Note: I've kept ".gitignore" file to ignore the data-carrying folders like db/, but still it's better to clone the repo elsewhere and copy over the changed files; avoid mistakenly wiping out your data. Only GTFSserverfunctions.py file has been changed. You can also just download that individually and replace.

[answerquest]

At this point I want to request for an ability to choose other route types from Extended GTFS Route Types like Trolleybus Service (800) and Share Taxi Service (717) which are very popular in the cities of Russia including mine.

Added these in as well. See diff for config/settings.js

[ArtemVN]

Thank you so much @answerquest, it's worked. Really appreciate your help! Trips are saves now.

[answerquest]

Great to know. I'll leave this issue open until I make new binaries to reflect the changes.