server becomes unstable after a request for bad user id

while testing i was making calls to the predictor and asked for a user id that is out of range, the server threw an error internally and then stopped sending valid responses to requests.

i was making calls against the /predictions/ranks endpoint, look for top entries for a user.

the error seen in the logs was:

18/02/16 22:05:13 INFO DAGScheduler: Job 22 finished: lookup at MatrixFactorizationModel.scala:168, took 0.192751 s
Process Process-1:                                                                             
Traceback (most recent call last):                                                                                                                    
  File "/opt/rh/python27/root/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap         
    self.run()                                                                                                                                        
  File "/opt/rh/python27/root/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run                                                       
    self._target(*self._args, **self._kwargs)                                                                                                         
  File "/opt/app-root/src/predictions.py", line 99, in loop                                                                                           
    int(req['topk']))                                                                                                   
  File "/opt/spark/python/lib/pyspark.zip/pyspark/mllib/recommendation.py", line 182, in recommendProducts                                            
    return list(self.call("recommendProducts", user, num))                                                                                            
  File "/opt/spark/python/lib/pyspark.zip/pyspark/mllib/common.py", line 146, in call              
    return callJavaFunc(self._sc, getattr(self._java_model, name), *a)                                                                                
  File "/opt/spark/python/lib/pyspark.zip/pyspark/mllib/common.py", line 123, in callJavaFunc                                                         
    return _java2py(sc, func(*args))                                       
  File "/opt/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__                                                       
    answer, self.gateway_client, self.target_id, self.name)                                                                                        
  File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco                                                                  
    return f(*a, **kw)                                                                                                          
  File "/opt/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value                                                    
    format(target_id, ".", name), value)                                                                                                              
Py4JJavaError: An error occurred while calling o53.recommendProducts.                                                                                 
: java.util.NoSuchElementException: next on empty iterator                                                                                            
        at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)                                                                                  
        at scala.collection.Iterator$$anon$2.next(Iterator.scala:37)                                                                                  
        at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:63)                                                                     
        at scala.collection.IterableLike$class.head(IterableLike.scala:107)                                                                           
        at scala.collection.mutable.WrappedArray.scala$collection$IndexedSeqOptimized$$super$head(WrappedArray.scala:35)                              
        at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOptimized.scala:126)                                                             
        at scala.collection.mutable.WrappedArray.head(WrappedArray.scala:35)                                                                          
        at org.apache.spark.mllib.recommendation.MatrixFactorizationModel.recommendProducts(MatrixFactorizationModel.scala:168)                       
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                                                
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)                                                              
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)                                                      
        at java.lang.reflect.Method.invoke(Method.java:498)                                                                                           
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)                                                                               
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)                                                                         
        at py4j.Gateway.invoke(Gateway.java:280)                                                                                                      
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)                                                                       
        at py4j.commands.CallCommand.execute(CallCommand.java:79)                                                                                     
        at py4j.GatewayConnection.run(GatewayConnection.java:214)                                                                                     
        at java.lang.Thread.run(Thread.java:748)                                                                                                      
172.17.0.1 - - [16/Feb/2018 22:05:14] "GET /predictions/ranks/fd3bb80ed43b4ab19fb139cf7454746b HTTP/1.1" 200 -

and all follow up calls, with good user ids return empty product sets, like this:

HTTP/1.0 200 OK
Cache-control: private
Connection: keep-alive
Content-Length: 96
Content-Type: application/json
Date: Fri, 16 Feb 2018 22:20:54 GMT
Server: Werkzeug/0.14.1 Python/2.7.13
Set-Cookie: 9980a7c9accc6bba40219a6c292148a7=86c1d6c030286ef743b6000825a7035b; path=/; HttpOnly

{
    "id": "00500b4dbab04617a53233802408d31f",
    "products": [],
    "topk": 5,
    "user": "1"
}

we probably just need to catch erroneous errors and figure out a way to keep the processing loop running.

radanalyticsio / jiminy-predictor

server becomes unstable after a request for bad user id #27