search

skinkie / reference

Personal repository where I collect working examples to understand inner workings while building PyNeTExConv
GNU Affero General Public License v3.0
0 stars 1 forks source link

Blablacar conversion not working yet: error in #42

Closed ue71603 closed 2 months ago

ue71603 commented 2 months ago

sequence of commands:

REM Blablacar conversion GTFS->NeTEx EPIP
REM to correct before run: feed_info.txt misses feed_end_date
del C:/Users/ue71603/MG_Daten/conversion/blablacar/*.duckdb
python gtfs_import_to_db.py C:/Users/ue71603/MG_Daten/conversion/blablacar/gtfs.zip  C:/Users/ue71603/MG_Daten/conversion/blablacar/gtfs-import.duckdb
python gtfs_convert_to_db.py C:/Users/ue71603/MG_Daten/conversion/blablacar/gtfs-import.duckdb C:/Users/ue71603/MG_Daten/conversion/blablacar/netex-import.duckdb
python epip_db_to_db.py C:/Users/ue71603/MG_Daten/conversion/blablacar/netex-import.duckdb C:/Users/ue71603/MG_Daten/conversion/blablacar/netex-import-epip.duckdb
python epip_db_to_xml.py C:/Users/ue71603/MG_Daten/conversion/blablacar/netex-import.duckdb C:/Users/ue71603/MG_Daten/conversion/blablacar/netex-import-epip.duckdb C:/Users/ue71603/MG_Daten/conversion/blablacar/netex.xml.gz
python netex_stats.py C:/Users/ue71603/MG_Daten/conversion/blablacar/netex-import-epip.duckdb C:/Users/ue71603/MG_Daten/conversion/blablacar/netex.xml
del C:/Users/ue71603/MG_Daten/conversion/blablacar/*.duckdb

error in: 

(venv) PS C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test> python epip_db_to_xml.py C:/Users/ue71603/MG_Daten/conversion/blablacar/netex-import.duckdb C:/Users/ue71603/MG_Daten/conversion/blablacar/netex-import-epip.duckd
b C:/Users/ue71603/MG_Daten/conversion/blablacar/netex.xml.gz
Traceback (most recent call last):
  File "C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test\epip_db_to_xml.py", line 243, in <module>
    export_epip_network_offer(args.original, args.target, args.output)
  File "C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test\epip_db_to_xml.py", line 119, in export_epip_network_offer
    organisation_or_transport_organisation = load_local(con_orig, Authority) + load_local(con_orig, Operator)
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test\dbaccess.py", line 32, in load_local
    cur.execute(f"SELECT object FROM {type};")
duckdb.duckdb.InvalidInputException: Invalid Input Error: Python Object "Authority" of type "type" found on line "C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test\epip_db_to_xml.py:119" not suitable for replacement scans. 
Make sure that "Authority" is either a pandas.DataFrame, duckdb.DuckDBPyRelation, pyarrow Table, Dataset, RecordBatchReader, Scanner, or NumPy ndarrays with supported format
(venv) PS C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test>

I use exactly the code that I checked out of your repository.

ue71603 commented 2 months ago

gtfs.zip

skinkie commented 2 months ago

I think I understand the error. It happens when creating the SQL statement, I wonder why I am not seeing it. Can your replace {type} with {type.__name__}?

ue71603 commented 2 months ago

Doesn't work for flixbus either. Same problem. I guess it is rather small. 

(venv) PS C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test> python epip_db_to_xml.py C:/Users/ue71603/MG_Daten/conversion/flixbus/netex-import.duckdb C:/Users/ue71603/MG_Daten/conversion/flixbus/netex-import-epip.duckdb C:
/Users/ue71603/MG_Daten/conversion/flixbus/netex.xml.gz
Traceback (most recent call last):
  File "C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test\epip_db_to_xml.py", line 243, in <module>
    export_epip_network_offer(args.original, args.target, args.output)
  File "C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test\epip_db_to_xml.py", line 119, in export_epip_network_offer
    organisation_or_transport_organisation = load_local(con_orig, Authority) + load_local(con_orig, Operator)
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test\dbaccess.py", line 32, in load_local
    cur.execute(f"SELECT object FROM {type};")
duckdb.duckdb.InvalidInputException: Invalid Input Error: Python Object "Authority" of type "type" found on line "C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test\epip_db_to_xml.py:119" not suitable for replacement scans. 
Make sure that "Authority" is either a pandas.DataFrame, duckdb.DuckDBPyRelation, pyarrow Table, Dataset, RecordBatchReader, Scanner, or NumPy ndarrays with supported format
(venv) PS C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test> del C:/Users/ue71603/MG_Daten/conversion/flixbus/*.duckdb
ue71603 commented 2 months ago

When I did your change the error was: (venv) PS C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test> python epip_db_to_xml.py C:/Users/ue71603/MG_Daten/conversion/flixbus/netex-import.duckdb C:/Users/ue71603/MG_Daten/conversion/flixbus/netex-import-epip.duckdb C: /Users/ue71603/MG_Daten/conversion/flixbus/netex.xml.gz Traceback (most recent call last): File "C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test\epip_db_to_xml.py", line 243, in export_epip_network_offer(args.original, args.target, args.output) File "C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test\epip_db_to_xml.py", line 119, in export_epip_network_offer organisation_or_transport_organisation = load_local(con_orig, Authority) + load_local(con_orig, Operator) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\ue71603\MG_Daten\github\reference\gtfs-netex-test\dbaccess.py", line 32, in load_local cur.execute(f"SELECT object FROM {type.name};") ^^^^^^^^^^^^^ AttributeError: 'str' object has no attribute 'name'. Did you mean: 'ne'?

ue71603 commented 2 months ago

It seems to be a string...

skinkie commented 2 months ago

I am testing it on my laptop now. Upgrading DuckDB already solved one issue. pip install duckdb --upgrade --pre (unrelated to your report)

I can reproduce your issue.

skinkie commented 2 months ago

My error message is much more clear:

Traceback (most recent call last):
  File "/mnt/storage/home/skinkie/Sources/references/gtfs-netex-test/epip_db_to_xml.py", line 243, in <module>
    export_epip_network_offer(args.original, args.target, args.output)
  File "/mnt/storage/home/skinkie/Sources/references/gtfs-netex-test/epip_db_to_xml.py", line 119, in export_epip_network_offer
    organisation_or_transport_organisation = load_local(con_orig, Authority) + load_local(con_orig, Operator)
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/storage/home/skinkie/Sources/references/gtfs-netex-test/dbaccess.py", line 32, in load_local
    cur.execute(f"SELECT object FROM {type};")
duckdb.duckdb.CatalogException: Catalog Error: Table with name Authority does not exist!
Did you mean "Operator"?
skinkie commented 2 months ago

@ue71603 please pull, and try again.

ue71603 commented 2 months ago

worked.... validated in XMLSpy.

doing Flixbus now.