For high-volume environments like Hadoop and Storm, performing an API lookup for every record is a non-starter.
Therefore, it would be worth adding an LRU cache into the WeatherAPI. In Scala we normally use Twitter's LruMap but we could use the Java equivalent instead.
For high-volume environments like Hadoop and Storm, performing an API lookup for every record is a non-starter.
Therefore, it would be worth adding an LRU cache into the WeatherAPI. In Scala we normally use Twitter's LruMap but we could use the Java equivalent instead.
We take this approach in https://github.com/snowplow/scala-forex