search

ibi-group / datatools-server

Server for IBI's GTFS data management platform.
MIT License
49 stars 52 forks source link

Cannot edit GTFS from scratch if no GTFS data has been loaded into PostgreSQL database #154

Closed dkinloch closed 5 years ago

dkinloch commented 5 years ago

Hi,

I'm getting this issue on dev branch.

`Error fetching GTFS entities via GraphQL

Any help on this?

Cheers,

ilarimikkonen commented 5 years ago

Hi,

I got the same on dev branch. server is logging: Jan 09 16:27:40.690 [qtp956634822-30] INFO c.c.d.common.utils.SparkUtils:221 - req ilari.mikkonen@apinf.io POST: http://84.20.150.166:9966/api/manager/secure/gtfs/graphqlbody content is longer than 500 characters, printing first 500 characters here: {"query":"\n query ($namespace: String) {\n feed(namespace: $namespace) {\n trip_counts {\n service_id {\n type\n count\n }\n pattern_id {\n type\n count\n }\n route_id {\n type\n count\n }\n }\n feed_info {\n id\n feed_publisher_name\n feed_publisher_url\n feed_lang\n
...and 949 more characters Jan 09 16:27:40.702 [qtp956634822-30] INFO graphql.schema.DataFetcher:37 - SQL: select * from feeds where namespace = 'fgyx_hjcjqluhekysurnixqyrky' Jan 09 16:27:40.703 [qtp956634822-30] WARN g.e.SimpleDataFetcherExceptionHandler:23 - Exception while fetching data (/feed) : org.postgresql.util.PSQLException: ERROR: relation "feeds" does not exist Position: 15 java.lang.RuntimeException: org.postgresql.util.PSQLException: ERROR: relation "feeds" does not exist Position: 15 at com.conveyal.gtfs.graphql.fetchers.FeedFetcher.get(FeedFetcher.java:57) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at com.conveyal.gtfs.graphql.fetchers.FeedFetcher.get(FeedFetcher.java:23) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at graphql.execution.ExecutionStrategy.fetchField(ExecutionStrategy.java:261) [dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at graphql.execution.ExecutionStrategy.resolveFieldWithInfo(ExecutionStrategy.java:202) [dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at graphql.execution.AsyncExecutionStrategy.execute(AsyncExecutionStrategy.java:70) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at graphql.execution.Execution.executeOperation(Execution.java:159) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at graphql.execution.Execution.execute(Execution.java:101) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at graphql.GraphQL.execute(GraphQL.java:573) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at graphql.GraphQL.parseValidateAndExecute(GraphQL.java:515) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at graphql.GraphQL.executeAsync(GraphQL.java:489) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at graphql.GraphQL.execute(GraphQL.java:420) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at com.conveyal.gtfs.GraphQLController.doQuery(GraphQLController.java:79) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at com.conveyal.gtfs.GraphQLController.postGraphQL(GraphQLController.java:59) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at spark.ResponseTransformerRouteImpl$1.handle(ResponseTransformerRouteImpl.java:47) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at spark.http.matching.Routes.execute(Routes.java:61) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at spark.http.matching.MatcherFilter.doFilter(MatcherFilter.java:130) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at spark.embeddedserver.jetty.JettyHandler.doHandle(JettyHandler.java:50) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1568) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at org.eclipse.jetty.server.Server.handle(Server.java:530) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na]

So, do we need to seed the database somehow? Also, subsequent saves to DB fail.

ilarimikkonen commented 5 years ago

or is dev branch unstable and shall we use some other branch?

landonreed commented 5 years ago

Does the PostgreSQL database you are referencing in env.yml exist? Creating that database and ensuring the application has access with username/password credentials should be the only prep you need to do for the database.

ilarimikkonen commented 5 years ago

The PostgreSQL database exists, I tested with giving wrong password and then it throws and password auth failed exception. If the database is incorrectly in env.yml, I get: Exception while fetching data (/feed) : org.postgresql.util.PSQLException: FATAL: database \"catalogueEEE\" does not exist"

I had accidentally used old config that had more values than the current one (https://github.com/conveyal/datatools-server/blob/dev/configurations/default/server.yml.tmp). I updated the file.

So the env.yml should be fine in that regard. I tested with both PostgreSQL ver 9.4 and 9.6. Same results.

When creating a new project and a feed to that I started seeing this :

Jan 10 14:42:22.701 [qtp1450237478-16] INFO c.c.d.common.utils.SparkUtils:221 - res (200) ilari.mikkonen@apinf.io GET: http://localhost:9966/api/manager/secure/feedsource?projectId=c1d05ae4-38ca-47f2-8863-f675e0afbc8b [ { "id" : "b6297e56-fc2d-4b6f-8559-c78ccad46957", "dateCreated" : 1547124142454, "projectId" : "c1d05ae4-38ca-47f2-8863-f675e0afbc8b", "name" : "fsdfsdfsdfdfsd", "isPublic" : false, "deployable" : false, "retrievalMethod" : "MANUALLY_UPLOADED", ...and 10 more lines Jan 10 14:42:22.814 [qtp1450237478-16] INFO c.c.d.common.utils.SparkUtils:221 - req ilari.mikkonen@apinf.io GET: http://localhost:9966/api/manager/secure/feedversion?feedSourceId=b6297e56-fc2d-4b6f-8559-c78ccad46957 null Jan 10 14:42:22.819 [qtp1450237478-16] INFO c.c.d.common.utils.SparkUtils:221 - res (200) ilari.mikkonen@apinf.io GET: http://localhost:9966/api/manager/secure/feedversion?feedSourceId=b6297e56-fc2d-4b6f-8559-c78ccad46957 [ ] Jan 10 14:42:22.831 [qtp1450237478-16] INFO c.c.d.common.utils.SparkUtils:221 - req ilari.mikkonen@apinf.io GET: http://localhost:9966/api/editor/secure/snapshot?feedId=b6297e56-fc2d-4b6f-8559-c78ccad46957 null Jan 10 14:42:22.836 [qtp1450237478-16] INFO c.c.d.common.utils.SparkUtils:221 - res (200) ilari.mikkonen@apinf.io GET: http://localhost:9966/api/editor/secure/snapshot?feedId=b6297e56-fc2d-4b6f-8559-c78ccad46957 [ ] Jan 10 14:42:22.838 [qtp1450237478-25] INFO c.c.d.common.utils.SparkUtils:221 - req ilari.mikkonen@apinf.io GET: http://localhost:9966/api/manager/secure/deployments?feedSourceId=b6297e56-fc2d-4b6f-8559-c78ccad46957 null Jan 10 14:42:22.839 [qtp1450237478-25] INFO c.c.d.common.utils.SparkUtils:112 - Halting with status code 404. Error message: No API route configured for this path.. Jan 10 14:42:22.839 [qtp1450237478-25] INFO c.c.d.common.utils.SparkUtils:221 - res (404) ilari.mikkonen@apinf.io GET: http://localhost:9966/api/manager/secure/deployments?feedSourceId=b6297e56-fc2d-4b6f-8559-c78ccad46957 { "result" : "ERR", "message" : "No API route configured for this path.", "code" : 404, "detail" : null } Jan 10 14:42:22.848 [qtp1450237478-25] INFO c.c.d.common.utils.SparkUtils:221 - req ilari.mikkonen@apinf.io GET: http://localhost:9966/api/manager/secure/feedversion?feedSourceId=b6297e56-fc2d-4b6f-8559-

Jan 10 14:44:56.713 [qtp1450237478-19] INFO c.c.d.common.utils.SparkUtils:221 - req ilari.mikkonen@apinf.io GET: http://localhost:9966/api/manager/secure/status/jobs null Jan 10 14:44:56.714 [qtp1450237478-19] INFO c.c.d.common.utils.SparkUtils:221 - res (200) ilari.mikkonen@apinf.io GET: http://localhost:9966/api/manager/secure/status/jobs [ ] Jan 10 14:44:57.354 [qtp1450237478-19] INFO c.c.d.common.utils.SparkUtils:221 - req ilari.mikkonen@apinf.io PUT: http://localhost:9966/api/editor/secure/lock/node0l46nak9h3lrc1spbyozp1j3gc2?feedId=b6297e56-fc2d-4b6f-8559-c78ccad46957 null Jan 10 14:44:57.355 [qtp1450237478-19] INFO c.c.d.common.utils.SparkUtils:221 - res (200) ilari.mikkonen@apinf.io PUT: http://localhost:9966/api/editor/secure/lock/node0l46nak9h3lrc1spbyozp1j3gc2?feedId=b6297e56-fc2d-4b6f-8559-c78ccad46957 { "result" : "OK", "message" : "Updating time for user ilari.mikkonen@apinf.io", "code" : 200, "feedId" : "b6297e56-fc2d-4b6f-8559-c78ccad46957" } Jan 10 14:45:01.100 [qtp1450237478-19] INFO c.c.d.common.utils.SparkUtils:221 - req ilari.mikkonen@apinf.io POST: http://localhost:9966/api/editor/secure/snapshot?feedId=b6297e56-fc2d-4b6f-8559-c78ccad46957 { "feedId" : "b6297e56-fc2d-4b6f-8559-c78ccad46957", "name" : "Blank" } Jan 10 14:45:01.122 [qtp1450237478-19] INFO c.c.d.common.utils.SparkUtils:221 - res (200) ilari.mikkonen@apinf.io POST: http://localhost:9966/api/editor/secure/snapshot?feedId=b6297e56-fc2d-4b6f-8559-c78ccad46957 { "jobId" : "64111216-d4f4-4047-83f1-5c9c388c9cf8", "message" : "Creating snapshot." } Jan 10 14:45:01.144 [qtp1450237478-19] INFO c.c.d.common.utils.SparkUtils:221 - req ilari.mikkonen@apinf.io GET: http://localhost:9966/api/manager/secure/status/jobs null Jan 10 14:45:01.146 [pool-1-thread-1] ERROR c.c.gtfs.loader.JdbcGtfsSnapshotter:435 - Exception while registering snapshot namespace in feeds table org.postgresql.util.PSQLException: ERROR: relation "feeds" does not exist Position: 13 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412) ~[dt-v3.2.0-313-g0ba07d3-dirty.jar:na]

And the errors mentioned in the previous comments keep popping up.

If I look at the database, it's "empty"

Attached are two setting files (never mind that .TXT in the end, github didn't like yml) I'm using server.yml.TXT settings.yml.TXT

ilarimikkonen commented 5 years ago

Ok,

figured it out. @dkinloch If you try to import some gtfs data (a zip file) to a project, this creates the needed database entiries. then you can create another project. Took me some time to sort this out...

let me know if this works for you!

abyrd commented 5 years ago

@landonreed I am seeing a similar error, but on a feed that's already been fetched.

Error is: Warning! Error fetching GTFS entities via GraphQL - Exception while fetching data (/feed) : Namespace prefix must contain only alpha, numeric, and underscores. ({"line":7,"column":3}). My public key is not in the known-hosts on the dev server, so I can't check the logs.

This happens after selecting a new route (19 in the Strasbourg feed) while viewing patterns.

Interestingly, going back a few minutes later, it now fetches the patterns with no problem.

landonreed commented 5 years ago

@ilarimikkonen, thanks for the follow-up. It does indeed look like the issue originally reported is that one cannot edit GTFS data in the Data Tools application if a GTFS zip file has never been imported into the PostgreSQL database. Essentially, JdbcGtfsSnapshotter expects that a feed already exists. I am going to change the name of this issue to reflect this underlying cause and will file this as a bug.

@abyrd, I'm not entirely sure what happened with your case, but it seems that perhaps somehow the namespace for the feed source was not updated before the GraphQL fetch happened. I'm not sure if you are able to consistently reproduce this, but if you uncover the steps to do so please share them here (we may need to create a separate ticket though).