search

heroku / predictionio-engine-ur

Universal Recommender optimized for deployment to Heroku
http://actionml.com/universal-recommender
Apache License 2.0
11 stars 10 forks source link

Intermittent `INSTANCE` error #2

Closed mars closed 7 years ago

mars commented 7 years ago

When this engine is deployed to Heroku, intermittently (~50% of time) the pio CLI commands will throw the error pasted below.

The effect is halting a new release (workaround with heroku releases:retry) or crashing the web process (workaround with heroku ps:restart; sometimes, heroku ps:stop is required to launch a dyno on a new underlying host.)

Exception in thread "main" org.apache.predictionio.data.storage.StorageClientException: INSTANCE
 at org.apache.predictionio.data.storage.elasticsearch.ESClient.open(StorageClient.scala:46)
 at org.apache.predictionio.data.storage.elasticsearch.ESEngineInstances.<init>(ESEngineInstances.scala:45)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at org.apache.predictionio.data.storage.Storage$.getDataObject(Storage.scala:309)
 at org.apache.predictionio.data.storage.Storage$.getDataObjectFromRepo(Storage.scala:269)
 at org.apache.predictionio.data.storage.Storage$.getMetaDataEngineInstances(Storage.scala:371)
 at org.apache.predictionio.tools.commands.Engine$.deploy(Engine.scala:220)
 at org.apache.predictionio.tools.console.Pio$.deploy(Pio.scala:102)
 at scala.Option.map(Option.scala:146)
 at org.apache.predictionio.tools.console.Console$$anonfun$main$1.apply(Console.scala:632)
 at org.apache.predictionio.tools.console.Console$$anonfun$main$1.apply(Console.scala:611)
 at org.apache.predictionio.tools.console.Console$.main(Console.scala:611)
 at org.apache.predictionio.tools.console.Console.main(Console.scala)
 at org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:53)
Caused by: java.lang.NoSuchFieldError: INSTANCE
 at org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:57)
 at org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory.<clinit>(DefaultHttpRequestWriterFactory.java:47)
 at org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.<init>(ManagedNHttpClientConnectionFactory.java:75)
 at org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.<init>(ManagedNHttpClientConnectionFactory.java:83)
 at org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.<clinit>(ManagedNHttpClientConnectionFactory.java:64)
 at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalConnectionFactory.<init>(PoolingNHttpClientConnectionManager.java:553)
 at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.<init>(PoolingNHttpClientConnectionManager.java:163)
 at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.<init>(PoolingNHttpClientConnectionManager.java:147)
 at org.apache.http.impl.nio.client.HttpAsyncClientBuilder.build(HttpAsyncClientBuilder.java:668)
 at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.<init>(PoolingNHttpClientConnectionManager.java:119)
 at org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:201)
 at org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:180)
 at org.apache.predictionio.data.storage.elasticsearch.ESClient.open(StorageClient.scala:43)
 ... 15 more
mars commented 7 years ago

Found several refs to this weird dependency problem: http://stackoverflow.com/questions/21864521/java-lang-nosuchfielderror-org-apache-http-message-basiclineformatter-instance http://stackoverflow.com/questions/27921077/nosuchfielderror-instance-at-org-apache-http-impl-io-defaulthttprequestwriterfac

mars commented 7 years ago

Fixed by stabilizing the sort order of "lib/spark/*" JARs in classpath.

The SNAPSHOT distribution of PredictionIO included with the buildpack now contains this change.