search

MobilityData / gtfs-realtime-validator

Java-based tool that validates General Transit Feed Specification (GTFS)-realtime feeds
Other
41 stars 10 forks source link

Speed up gtfs-validator component build #33

Open barbeau opened 2 years ago

barbeau commented 2 years ago

Issue by barbeau Tuesday Apr 25, 2017 at 16:43 GMT Originally opened as https://github.com/CUTR-at-USF/gtfs-realtime-validator/issues/152

Summary:

mvn package can take a while to execute - most of this time is due to pulling the components needed to build gtfs-validator-webapp (https://github.com/conveyal/gtfs-validator/tree/master/gtfs-validator-webapp) so we can display the results in our webapp.

We should try to speed this up. We are using https://github.com/eirslett/frontend-maven-plugin to build the webapp via npm and gulp + browserfy.

Possible options for speedup:

  1. Examine caching - does frontend-maven-plugin/npm/gulp/browserfy allow caching so it doesn't do a clean build every time?
  2. Examine execution phase for frontend-maven-plugin - in pom.xml, we have the <phase> as compile, but the default according to https://github.com/eirslett/frontend-maven-plugin#usage the default <phase> is generate-resources. Would changing to generate-resources speed up the build?
  3. Include an option to allow people to turn off building the gtfs-validator component if they don't need static GTFS validation.

Steps to reproduce:

Execute mvn package

Expected behavior:

Build fairly quickly

Observed behavior:

Builds take a while to complete - around 3 min 16 sec for mvn clean package, and about 2 min 9 sec for mvn package after that initial build on my dual CPU Xeon @ 2.5 GHz and 16 GB of RAM.

barbeau commented 2 years ago

Comment by barbeau Tuesday May 30, 2017 at 20:05 GMT

For speeding up NPM installs, see https://lincolnloop.com/blog/speeding-npm-installs/.

barbeau commented 2 years ago

Comment by barbeau Tuesday May 30, 2017 at 20:52 GMT

For caching with frontend-maven-plugin see discussion at https://github.com/eirslett/frontend-maven-plugin/issues/27, and potential solution at https://github.com/eirslett/frontend-maven-plugin/pull/118.

barbeau commented 2 years ago

Comment by barbeau Tuesday May 30, 2017 at 21:55 GMT

I tried the solution at https://github.com/eirslett/frontend-maven-plugin/pull/118, but it didn't seem to have any effect - I didn't see any new cache directory. I bumped the version of frontend-maven-plugin in https://github.com/CUTR-at-USF/gtfs-realtime-validator/commit/0d0f3811bcedae16320739b2645f23595ce02c94 just so we inherit any improvements there.

Running mvn -T 4 install to increase number of threads doesn't seem to have any speed up effect either.