Open isabelle-dr opened 2 years ago
Comment by barbeau May 11, 2017
As mentioned in #123 (comment):
If I try to run the Dutch feed (http://gtfs.openov.nl/gtfs/gtfs-openov-nl.zip) with -Xmx8g parameter on my machine (dual Xeon @ 2.5 GHz w/ 16GB RAM), I get this exception after it runs for a very long time (I left and came back an hour later):
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: >java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:423)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:386)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:334)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:800)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:313)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:626)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:546)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.glassfish.jersey.server.ContainerException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:256)
at org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:238)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:486)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:316)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:291)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1140)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:403)
... 17 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:68)
at java.lang.StringBuilder.<init>(StringBuilder.java:89)
at org.onebusaway.csv_entities.DelimitedTextParser.parse(DelimitedTextParser.java:65)
at org.onebusaway.csv_entities.CSVLibrary.parse(CSVLibrary.java:131)
at org.onebusaway.csv_entities.CsvTokenizerStrategy.parse(CsvTokenizerStrategy.java:34)
at org.onebusaway.csv_entities.CsvEntityReader.readEntities(CsvEntityReader.java:154)
at org.onebusaway.csv_entities.CsvEntityReader.readEntities(CsvEntityReader.java:120)
at org.onebusaway.csv_entities.CsvEntityReader.readEntities(CsvEntityReader.java:115)
at org.onebusaway.gtfs.serialization.GtfsReader.run(GtfsReader.java:172)
at org.onebusaway.gtfs.serialization.GtfsReader.run(GtfsReader.java:160)
at com.conveyal.gtfs.validator.json.FeedProcessor.load(FeedProcessor.java:73)
at com.conveyal.gtfs.validator.json.FeedProcessor.run(FeedProcessor.java:44)
at edu.usf.cutr.gtfsrtvalidator.api.resource.GtfsFeed.postGtfsFeed(GtfsFeed.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:308)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
So it looks like it's getting hung up in the static GTFS validation using the Conveyal gtfs-validator.
I tried running the same http://gtfs.openov.nl/gtfs/gtfs-openov-nl.zip feed through gtfs-lib on the command line (i.e., not integrated into gtfs-rt-validator - just running the built jar with java -jar target/gtfs-lib-2.2.0-SNAPSHOT-shaded.jar -validate gtfs-openov-nl.zip result.json) and it took a while but was able to process it without crashing. So that's one benefit of using/moving to gtfs-lib.
EDIT - Looks like I spoke a little too soon - it did get through the validation phase, but after about an hour still hasn't generated any JSON output. It's hung at:
[main] INFO com.conveyal.gtfs.GTFSFeed - TripTimesValidator finished in 154209 milliseconds.
[main] INFO com.conveyal.gtfs.GTFSFeed - UnusedStopValidator finished in 156 milliseconds.
[main] INFO com.conveyal.gtfs.GTFSFeed - 8 validators completed in 204791 milliseconds.
Comment by barbeau Apr 22, 2020
We should switch to using the MobilityData gtfs-validator as the static validator in this project: https://github.com/MobilityData/gtfs-validator
Comment by barbeau December 13, 2021
Note that the Maven configuration added in PR #403 should be removed when the transition to the MobilityData gtfs-validator is complete.
Issue by barbeau May 10, 2017 Originally opened as https://github.com/CUTR-at-USF/gtfs-realtime-validator/issues/193
Summary:
For static GTFS validation, the Conveyal team has moved away from using gtfs-validator and focused on a newer project gtfs-lib.
We should look at integrating this into our project, in addition to, and perhaps eventually in place of, gtfs-validator.
My comments from conveyal/gtfs-validator#40 (comment):