Attempting to use the config-sample.json

coding-to-music commented 1 year ago

Hello, I am getting an error, please see below

create config.json using the config-sample.json

cp config-sample.json config.json

Install gtfs-to-chart globally

npm install gtfs-to-chart -g

contents of config-sample.json

{
  "agencies": [
    {
      "agency_key": "bart",
      "url": "http://www.bart.gov/dev/schedules/google_transit.zip"
    }
  ],
  "sqlitePath": "/tmp/gtfs"
}

Running

gtfs-to-chart --configPath config.json

Output

Starting GTFS import for 1 file using SQLite database at /tmp/gtfs
Downloading GTFS from http://www.bart.gov/dev/schedules/google_transit.zip
Download successful
Importing GTFS from /tmp/tmp-141173-eank3fXmavG0/gtfs.zip
Importing - agency.txt - 1 lines imported
Importing - areas.txt - No file found
Importing - attributions.txt - No file found
Importing - calendar_dates.txt - 32 lines imported
Importing - calendar.txt - 13 lines imported
Importing - fare_attributes.txt - 2500 lines imported
Importing - fare_leg_rules.txt - No file found
Importing - fare_products.txt - No file found
Importing - fare_rules.txt - 2500 lines imported
Importing - fare_transfer_rules.txt - No file found
Importing - feed_info.txt - 1 lines imported
Importing - frequencies.txt - No file found
Importing - levels.txt - No file found
Importing - pathways.txt - No file found
Importing - routes.txt - 14 lines imported
Importing - shapes.txt - 38044 lines imported
Importing - stop_areas.txt - No file found
Importing - stop_times.txt - 61853 lines imported
Importing - stops.txt - 186 lines imported
Importing - transfers.txt - 41 lines imported
Importing - translations.txt - No file found
Importing - trips.txt - 4225 lines imported
Importing - calendar_attributes.txt - 11 lines imported
Importing - directions.txt - 14 lines imported
Importing - route_attributes.txt
Warning: Check route_attributes.txt for invalid data between lines 0 and 14.

file:///home/tmc/.nvm/versions/node/v16.19.1/lib/node_modules/gtfs-to-chart/node_modules/gtfs/lib/import.js:473
    throw error;
    ^
[  SqliteError: route_attributes.route_id

  - import.js:466 importLines
    [lib]/[gtfs-to-chart]/[gtfs]/lib/import.js:466:7

  - import.js:547 Parser.<anonymous>
    [lib]/[gtfs-to-chart]/[gtfs]/lib/import.js:547:11

  - node:events:513 Parser.emit
    node:events:513:28

  - readable:1358 endReadableNT
    node:internal/streams/readable:1358:12

  - task_queues:83 processTicksAndRejections
    node:internal/process/task_queues:83:21

] {
  code: 'SQLITE_CONSTRAINT_PRIMARYKEY'
}

Contents of route_attributes.txt

route_id,category,subcategory,running_way
1,2,201,1
2,2,201,1
3,2,201,1
4,2,201,1
5,2,201,1
6,2,201,1
7,2,201,1
8,2,201,1
11,2,201,1
12,2,201,1
19,3,301,1
20,3,301,1
2,2,201,1
1,2,201,1

Also, how would you recommend I get the data for Boston's MBTA? I'm a little confused about sourcing that data

I see these data files https://transitfeeds.com/search?q=mbta

Thanks very much

coding-to-music commented 1 year ago

Perhaps this is for Boston https://transitfeeds.com/p/mbta/64/latest

coding-to-music commented 1 year ago

Not sure how to get MBTA data, this is my attempt:

Boston MBTA data

https://transitfeeds.com/p/mbta/64/latest

https://transitfeeds.com/p/mbta/64/latest/download

create config-mbta.json

{
  "agencies": [
    {
      "agency_key": "mbta",
      "url": "https://transitfeeds.com/p/mbta/64/latest/download/gtfs.zip"
    }
  ],
  "sqlitePath": "/tmp/gtfs"
}

Running

gtfs-to-chart --configPath config-mbta.json

Output

gtfs-to-chart --configPath config-mbta.json

Starting GTFS import for 1 file using SQLite database at /tmp/gtfs
Downloading GTFS from https://transitfeeds.com/p/mbta/64/latest/download/gtfs.zip
Error: Couldn’t download files
file:///home/tmc/.nvm/versions/node/v16.19.1/lib/node_modules/gtfs-to-chart/node_modules/gtfs/lib/import.js:41
    throw new Error('Couldn’t download files');
          ^

[  Error: Couldn’t download files

  - import.js:41 downloadFiles
    [lib]/[gtfs-to-chart]/[gtfs]/lib/import.js:41:11

  - task_queues:96 processTicksAndRejections
    node:internal/process/task_queues:96:5

  - import.js:597 
    [lib]/[gtfs-to-chart]/[gtfs]/lib/import.js:597:9

  - index.js:7 async promiseMapSeries
    [lib]/[gtfs-to-chart]/[promise-map-series]/index.js:7:18

  - import.js:577 async importGtfs
    [lib]/[gtfs-to-chart]/[gtfs]/lib/import.js:577:5

  - gtfs-to-chart.js:43 
    [lib]/[gtfs-to-chart]/lib/gtfs-to-chart.js:43:7

  - async Promise.all

  - gtfs-to-chart.js:66 
    [lib]/[gtfs-to-chart]/bin/gtfs-to-chart.js:66:5

]

brendannee commented 1 year ago

It looks like you can get MBTA's GTFS here: https://cdn.mbta.com/MBTA_GTFS.zip - try that URL

brendannee commented 1 year ago

Also, try the gtfs-to-chart version 2.0.6 which was just released - this solves some issues with unzipping files in node.js v18 https://github.com/BlinkTagInc/gtfs-to-chart/releases/tag/2.0.6

coding-to-music commented 1 year ago

Hmmm, still the same error, although a different line number is referenced

was

file:///home/tmc/.nvm/versions/node/v16.19.1/lib/node_modules/gtfs-to-chart/node_modules/gtfs/lib/import.js:473

now is

file:///home/tmc/.nvm/versions/node/v16.19.1/lib/node_modules/gtfs-to-chart/node_modules/gtfs/lib/import.js:482

Install latest version

npm install gtfs-to-chart -g

Output

added 1 package, removed 22 packages, and changed 318 packages in 15s

49 packages are looking for funding
  run `npm fund` for details

Check version

gtfs-to-chart --version

Output

2.0.6

Run

gtfs-to-chart --configPath config.json

Output

Starting GTFS import for 1 file using SQLite database at /tmp/gtfs
Downloading GTFS from http://www.bart.gov/dev/schedules/google_transit.zip
Download successful
Importing GTFS from /tmp/tmp-178870-serzBY5d4W2p/gtfs.zip
Importing - agency.txt - 1 lines imported
Importing - areas.txt - No file found
Importing - attributions.txt - No file found
Importing - calendar_dates.txt - 32 lines imported
Importing - calendar.txt - 13 lines imported
Importing - fare_attributes.txt - 2500 lines imported
Importing - fare_leg_rules.txt - No file found
Importing - fare_products.txt - No file found
Importing - fare_rules.txt - 2500 lines imported
Importing - fare_transfer_rules.txt - No file found
Importing - feed_info.txt - 1 lines imported
Importing - frequencies.txt - No file found
Importing - levels.txt - No file found
Importing - pathways.txt - No file found
Importing - routes.txt - 14 lines imported
Importing - shapes.txt - 38044 lines imported
Importing - stop_areas.txt - No file found
Importing - stop_times.txt - 61853 lines imported
Importing - stops.txt - 186 lines imported
Importing - transfers.txt - 41 lines imported
Importing - translations.txt - No file found
Importing - trips.txt - 4225 lines imported
Importing - calendar_attributes.txt - 11 lines imported
Importing - directions.txt - 14 lines imported
Importing - route_attributes.txt
Warning: Check route_attributes.txt for invalid data between lines 0 and 14.

file:///home/tmc/.nvm/versions/node/v16.19.1/lib/node_modules/gtfs-to-chart/node_modules/gtfs/lib/import.js:482
    throw error;
    ^
[  SqliteError: route_attributes.route_id

  - import.js:475 importLines
    [lib]/[gtfs-to-chart]/[gtfs]/lib/import.js:475:7

  - import.js:556 Parser.<anonymous>
    [lib]/[gtfs-to-chart]/[gtfs]/lib/import.js:556:11

  - node:events:513 Parser.emit
    node:events:513:28

  - readable:1358 endReadableNT
    node:internal/streams/readable:1358:12

  - task_queues:83 processTicksAndRejections
    node:internal/process/task_queues:83:21

  - task_queues:65 runNextTicks
    node:internal/process/task_queues:65:3

  - timers:437 processImmediate
    node:internal/timers:437:9

] {
  code: 'SQLITE_CONSTRAINT_PRIMARYKEY'
}

coding-to-music commented 1 year ago

I'm just noticing your reference to node v18, I am using v16

I will try node v18

coding-to-music commented 1 year ago

Looks like it is still occurring even with node v18

nvm use v18

Output

Now using node v18.15.0 (npm v9.5.0)

Run

gtfs-to-chart --configPath config.json

Output

gtfs-to-chart: command not found

Install

npm install gtfs-to-chart -g

Output

added 319 packages in 14s

49 packages are looking for funding
  run `npm fund` for details

Run

gtfs-to-chart --configPath config.json

Output

Starting GTFS import for 1 file using SQLite database at /tmp/gtfs
Downloading GTFS from http://www.bart.gov/dev/schedules/google_transit.zip
Download successful
Importing GTFS from /tmp/tmp-181995-1WN38QCkMYFO/gtfs.zip
Importing - agency.txt - 1 lines imported
Importing - areas.txt - No file found
Importing - attributions.txt - No file found
Importing - calendar_dates.txt - 32 lines imported
Importing - calendar.txt - 13 lines imported
Importing - fare_attributes.txt - 2500 lines imported
Importing - fare_leg_rules.txt - No file found
Importing - fare_products.txt - No file found
Importing - fare_rules.txt - 2500 lines imported
Importing - fare_transfer_rules.txt - No file found
Importing - feed_info.txt - 1 lines imported
Importing - frequencies.txt - No file found
Importing - levels.txt - No file found
Importing - pathways.txt - No file found
Importing - routes.txt - 14 lines imported
Importing - shapes.txt - 38044 lines imported
Importing - stop_areas.txt - No file found
Importing - stop_times.txt - 61853 lines imported
Importing - stops.txt - 186 lines imported
Importing - transfers.txt - 41 lines imported
Importing - translations.txt - No file found
Importing - trips.txt - 4225 lines imported
Importing - calendar_attributes.txt - 11 lines imported
Importing - directions.txt - 14 lines imported
Importing - route_attributes.txt
Warning: Check route_attributes.txt for invalid data between lines 0 and 14.

file:///home/tmc/.nvm/versions/node/v18.15.0/lib/node_modules/gtfs-to-chart/node_modules/gtfs/lib/import.js:482
    throw error;
    ^
[  SqliteError: route_attributes.route_id

  - import.js:475 importLines
    [lib]/[gtfs-to-chart]/[gtfs]/lib/import.js:475:7

  - import.js:556 Parser.<anonymous>
    [lib]/[gtfs-to-chart]/[gtfs]/lib/import.js:556:11

  - node:events:513 Parser.emit
    node:events:513:28

  - readable:1359 endReadableNT
    node:internal/streams/readable:1359:12

  - task_queues:82 process.processTicksAndRejections
    node:internal/process/task_queues:82:21

] {
  code: 'SQLITE_CONSTRAINT_PRIMARYKEY'
}

Node.js v18.15.0

coding-to-music commented 1 year ago

Great news - MBTA loads and makes charts - looks like the problem file route_attributes.txt does not exist in the MBTA file

big file with lots of rows

gtfs-to-chart --configPath config-mbta.json

Output

Starting GTFS import for 1 file using SQLite database at /tmp/gtfs
Downloading GTFS from https://cdn.mbta.com/MBTA_GTFS.zip
Download successful
Importing GTFS from /tmp/tmp-182845-5Sj01ZrXnN4T/gtfs.zip
Importing - agency.txt - 1 lines imported
Importing - areas.txt - 32 lines imported
Importing - attributions.txt - No file found
Importing - calendar_dates.txt - 54 lines imported
Importing - calendar.txt - 90 lines imported
Importing - fare_attributes.txt - No file found
Importing - fare_leg_rules.txt - 456 lines imported
Importing - fare_products.txt - 87 lines imported
Importing - fare_rules.txt - No file found
Importing - fare_transfer_rules.txt - 37 lines imported
Importing - feed_info.txt - 1 lines imported
Importing - frequencies.txt - No file found
Importing - levels.txt - 74 lines imported
Importing - pathways.txt - 7999 lines imported
Importing - routes.txt - 219 lines imported
Importing - shapes.txt - 258537 lines imported
Importing - stop_areas.txt - 758 lines imported
Importing - stop_times.txt - 1605036 lines imported
Importing - stops.txt - 9743 lines imported
Importing - transfers.txt - 5312 lines imported
Importing - translations.txt - No file found
Importing - trips.txt - 65139 lines imported
Importing - calendar_attributes.txt - 90 lines imported
Importing - directions.txt - 378 lines imported
Completed GTFS import for 1 agency
mbta: Generating charts [===================================----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 35/219
Warning: No trips found for route CR-Foxboro on May 5, 2023
mbta: Generating charts [============================================================================================================================-----------------------------------------------------------------------------------------------] 124/219
Warning: No trips found for route Shuttle-Generic-CommuterRail-South on May 5, 2023

Warning: No trips found for route Shuttle-Generic-CommuterRail-North on May 5, 2023

Warning: No trips found for route Shuttle-Generic-CommuterRail on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Blue on May 5, 2023

Warning: No trips found for route Shuttle-Generic on May 5, 2023
mbta: Generating charts [===============================================================================================================================================================================--------------------------------------------] 175/219
Warning: No trips found for route 195 on May 5, 2023
mbta: Generating charts [============================================================================================================================================================================================-------------------------------] 188/219
Warning: No trips found for route Shuttle-JFKPark on May 5, 2023
mbta: Generating charts [==============================================================================================================================================================================================-----------------------------] 190/219
Warning: No trips found for route Shuttle-JFKKendall on May 5, 2023
mbta: Generating charts [=================================================================================================================================================================================================--------------------------] 193/219
Warning: No trips found for route Shuttle-GovernmentCenterWonderland on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Worcester on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Red-Trunk on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Red-Braintree on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Red-Ashmont on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Red on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Providence on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Orange on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Newburyport on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Needham on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Middleborough on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Mattapan on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Lowell on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Kingston on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Haverhill on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Green-Trunk on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Green-E on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Green-D on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Green-C on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Greenbush on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Green-B on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Green on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Franklin on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Fitchburg on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Fairmount on May 5, 2023

Warning: No trips found for route Shuttle-Generic-Elevator on May 5, 2023
mbta: Generating charts [===========================================================================================================================================================================================================================] 219/219

mbta: charts created at /home/tmc/ap/vol6/gtfs-to-chart-stringlines/charts/mbta

coding-to-music commented 1 year ago

I am wondering the best way to view the charts? I can open the index file on my machine and browse each one, but I can't view them via GitHub at my repo

https://github.com/coding-to-music/gtfs-to-chart-stringlines/tree/main/charts/mbta

Is there an existing React or NextJS or Angular app that knows how to view the charts?

Also, when I look at a chart, it appears to only contain a single inbound and outbound row, perhaps from last night's last run of the day?

It does not show the trips throughout the day, such as this similar one I saw the other day on Twitter, not sure how it was generated https://twitter.com/ndhapple/status/1654169426944458773

coding-to-music commented 1 year ago

I tried out https://github.com/brendannee/whereisbart.com very cool

And magically was able to deploy my clone to vercel, works, amazing! https://whereisbart-leaflet-google-transit.vercel.app

I tried the same with this, I was able to deploy via vercel --prod and specify the file locations are in charts/mbta pretty cool, it works!! https://gtfs-to-chart-stringlines.vercel.app

I wonder why only the last runs of the day are visible?

For example

Green-B https://gtfs-to-chart-stringlines.vercel.app/charts/B.html
Green-C https://gtfs-to-chart-stringlines.vercel.app/charts/C.html
Green-D https://gtfs-to-chart-stringlines.vercel.app/charts/D.html
Green-E https://gtfs-to-chart-stringlines.vercel.app/charts/E.html
Orange Line https://gtfs-to-chart-stringlines.vercel.app/charts/Orange%20Line.html
Blue https://gtfs-to-chart-stringlines.vercel.app/charts/Blue%20Line.html
Nothing for the Red Line https://gtfs-to-chart-stringlines.vercel.app/charts/Red%20Line.html

BlinkTagInc / gtfs-to-chart