Oculus is no longer actively maintained. Your mileage with patches may vary.
Oculus is the anomaly correlation component of Etsy's Kale system.
It lets you search for metrics, using your choice of comparison algorithms:
and shows you other metrics which are similar:
You can even save interesting metrics into a collection, complete with your own notes:
And then if Oculus finds matches in your saved collections, it'll show you this alongside your other search results:
Oculus consists of the following components:
Where recommended server specs have been mentioned in this document, they're based on Etsy's usage of Oculus which is based around 250k metrics. Adjust as necessary for your own metric volume.
It's recommended that you work through the section in this README in the following order:
Following the instructions in this order should result in a working Oculus setup with all the moving parts running and functioning correctly. Please note that these instructions are geared towards installing Oculus components on separate boxes. If you're installing everything on one box, then you can ignore replicated steps such as cloning the Oculus code - this only needs to be done once.
Also provided, but not necessary to get Oculus up and running are some utility scripts, in the section headed
For Oculus to function properly, you will ideally need at least two ElasticSearch servers in separate clusters, which Oculus will rotate through. Oculus requires the addition of custom scoring plugins, but otherwise uses ElasticSearch out of the box.
yum install jdk
)http://www.elasticsearch.org/download/
- here we'll assume /opt/elasticsearch
mkdir elasticsearch-oculus-plugin
/opt/oculus/resources/elasticsearch-oculus-plugin
to /opt/elasticsearch/elasticsearch-oculus-plugin
/opt/elasticsearch/elasticsearch-oculus-plugin
rake build
OculusPlugins.jar
OculusPlugins.jar
to /opt/elasticsearch/lib/OculusPlugins.jar
As noted above, the Elasticsearch servers which Oculus uses cannot be in the same cluster - Oculus needs at least two seperate servers (and clusters) to rotate between. To that end, the ElasticSearch configuration file needs to have the Cluster name and Node name manually specified. For simplicity, these instructions assume that you're going to name each cluster and node after the hostname of the server, but any name can be used.
The scoring plugins which Oculus uses for searching must also be added to the configuration file.
Change the following lines in /opt/elasticsearch/config/elasticsearch.yml
, changing the cluster and nodename to match the servers you're using:
cluster.name: oculussearch01.mydomain.com
node.name: oculussearch01.mydomain.com
Add the following lines to /opt/elasticsearch/config/elasticsearch.yml
, changing the cluster and nodename to match the servers you're using:
script.native:
oculus_euclidian.type: com.etsy.oculus.tsscorers.EuclidianScriptFactory
oculus_dtw.type: com.etsy.oculus.tsscorers.DTWScriptFactory
Once you've installed ElasticSearch, built the Oculus scoring plugins and edited the configuration file, you can start it up by running the following command from /opt/elasticsearch
bin/elasticsearch
You can verify that Elasticsearch is up and running by visiting http://locahost:9200
on the box you installed ElasticSearch on. If all is well, you should see JSON similar to the below:
{
"ok" : true,
"status" : 200,
"name" : "oculussearch01",
"version" : {
"number" : "0.20.5",
"snapshot_build" : false
},
"tagline" : "You Know, for Search"
}
Oculus' import and backend functions are handled by a cluster of Worker boxes running Resque (https://github.com/resque/resque). You'll need a Worker Master, running Redis (the central data store used by Resque) and some resque workers, and optionally some more Worker Slave boxes running extra Resque workers.
The first worker box you'll need to get up and running is the Master box running Redis.
Here's how to get it up and running:
yum install redis
)service redis start
)redis-cli
redis 127.0.0.1:6379>
exit
to return back to the Shell.bundle install
Rakefile
, changing the URL on line 66 to match the URL to your redis instancemkdir /var/run/oculus
and make sure the user you will run your workers as can read and write to it.mkdir /var/log/oculus
and make sure the user you will run your workers as can read and write to it.rake resque:start_workers
resque-web
from the binary directory of the resque Gem (for example /usr/lib64/ruby/gems/1.9.1/gems/resque-1.23.0/bin/resque-web
)rake resque:restart_workers
Once you've got your Worker Master set up and running Redis, you can also optionally set up any number of Worker Slave boxes (which just run Resque Workers) you want - we run 3 at Etsy.
bundle install
Rakefile
, changing the URL on line 66 to match the URL to your redis instance on the Worker Mastermkdir /var/run/oculus
and make sure the user you will run your workers as can read and write to it.mkdir /var/log/oculus
and make sure the user you will run your workers as can read and write to it.rake resque:start_workers
rake resque:restart_workers
Now that you've got your Elasticsearch servers all prepped and your Workers happily waiting for work, before we can start chucking data Oculus the next step is to set up the Oculus config file - config/config.yml
. Here's a sample config file:
results_explain: 0
elasticsearch:
servers:
- "http://oculussearch01.mycompany.com:9200"
- "http://oculussearch02.mycompany.com:9200"
index: "metrics"
timeout: 30
phrase_slop: 20
scorers:
dtw:
radius: 5
scale_points: 25
euclidian:
scale_points: 25
skyline:
host: "skyline.mycompany.com"
port: 6379
listener_port: 2015
redis:
host: "oculusredis.mycompany.com"
port: 6379
At this stage, all you need to do is the following:
elasticsearch->servers
sectionredis
section
skyline
section.You can ignore the other settings under elasticsearch for now - Have a look at the "Help" page in Oculus once you've gotten it up and running if you want to know more about these. For now, the defaults will suffice!
You should make sure that your config file is pushed out to all of your Oculus servers.
Now that you've got your search servers all ready to populate, you'll need to set up the script that imports data from Skyline into your search indexes. This can run on any of the servers you're using to run Oculus.
scripts
directory under the directory where you cloned Oculus../import.rb
Active ES Server: http://oculussearch01.mycompany.com:9200
Next ES Server: http://oculussearch02.mycompany.com:9200
Recreating indexes
Creating redis jobs...
Getting unique metric names
Found 250578 metric names
187 workers working
311 process_redis_metrics jobs left to run
187 workers working
309 process_redis_metrics jobs left to run
187 workers working
148 process_redis_metrics jobs left to run
187 workers working
113 process_redis_metrics jobs left to run
187 workers working
110 process_redis_metrics jobs left to run
153 workers working
0 process_redis_metrics jobs left to run
Setting active search server to http://oculussearch02.mycompany.com:9200
Oculus import finished in 36.634672763 seconds
*/2 * * * * /opt/oculus/scripts/import.rb > /var/log/oculus/import.log 2>&1
Now that we've got all of the moving parts set up and started updating our search indexes with metric data, the final step is to get the Oculus front end web app set up.
bundle install
config
directory under your Oculus checkoutthin start
>> Using rack adapter
>> Thin web server (v1.5.0 codename Knife)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:3000, CTRL+C to stop
http://yourserver:3000/admin, username admin and password admin
), then click "Reinitialize Collections." Click OK to confirm.http://yourserver:3000
- you can now start searching for metrics!!