search

kylefernandadams / box-elastic-events

The Box Elastic Events project is an open source data pipeline framework to pull Box Events and persist them to Elasticsearch.
Apache License 2.0
2 stars 1 forks source link

readme

Box Elastic Events

The Box Elastic Events project is an open source data pipeline framework to pull Box Events and persist them to Elasticsearch.

alt text

Components Used

  1. Box Java SDK: Java bindings for Box's REST API's
  2. Akka.io: A Framework based on the Actor Model and is used for building high-performance, elastic, resilient, and distributed applications.
    • Note that the current implementation is not using Akka's cluster implementation, but can be easily adapted to support it.
  3. Elasticsearch Java REST Client: Used for most GET and POST requests to Elasticsearch.
  4. Elasticsearch Java Transport Client: Used for managing Elasticserach Indices.
    • Note that if and when the Java REST Client is enhanced to support the index admin API's, the usage of the transport client will be deprecated.

All-in-one Package Installation

Download the Package

  1. Download and extract the self-contained package here or run the following wget command. 
    wget https://github.com/kylefernandadams/box-elastic-events/releases/download/v1.0/box-elastic-events-allinone.tar.gz
tar -xvf box-elastic-events-allinone.tar.gz

Package Contents:

Create a New Box Developer Application

  1. Create a new application in the Box Developer Console https://app.box.com/developers/services. alt text
  2. Choose the Enterprise Integration application type. alt text
  3. Choose the OAuth 2.0 with JWT (Server Authentication) authentication method. alt text
  4. Name your application. alt text
  5. In the configuration section of your newly created application, Copy the clientId and clientSecret OAuth credentials. alt text
  6. Enable the Manage enterprise properties check box. alt text

Generate RSA Keypair

  1. In the terminal, change directories to the unarchived box-elastic-events directory. 
    cd box-elastic-events
  2. Run the generate_rsa_keypair.sh script in your terminal app of choice to generate the public and private keys.
  3. Provide a password in the terminal 
    ./generate_rsa_keypair.sh
2017-02-16-11:50:36 - Starting RSA Keypair generation...
2017-02-16-11:50:36 - Generating private key...
Please enter a password for the RSA keypair
Enter Password:
  4. Copy the public_key.pem content that was printed in the console alt text
  5. In the Box Developer Console, Click the Add a Public Key button
    • Note that may recent a prompt to enable 2-factor authentication alt text
  6. Paste the public_key.pem text into the Public Key window and click the Verify and Save button alt text
  7. Copy the generated Public Key Id alt text

Authorize Application

  1. In the Admin Console's App configuration authorize your new application using the clientId. alt text

Add Application Configuration

  1. In the configuration/box-elastic-events.conf file, add the client.id in the box.platform section.
  2. Add the client.secret
  3. Add the enterprise.id. Note that your enterprise id can be retrieved from the Admin Console.
  4. Add the public.key.id
  5. Add the private.key.password
  6. Add the config.path for the configuration directory.
  7. Modify the lookback time value change how far back to capture enterprise events. 
    box {
platform {
client.id = ""
client.secret = ""
enterprise.id = ""
public.key.id = ""
private.key.file = "private_key.pem"
private.key.password = ""
max.cache.entries = 100
config.path = ""
# The lookback config is only used on initial creation of the ES index.
# If the ES index is already created, the last created_at date will be retrieved from ES.
# Use a lookback value of 0 if you would like to exclude a specific time frame such as days.
# Example: seconds = 60, minutess = 60, hours = 1, days = 0 to designate 1 hour
lookback {
  seconds = 60
  minutes = 60
  hours = 24
  days = 365
}
}
}
  8. In the elastic.enterprise section, modify the polling.interval value. The default value 2 minutes. 
    enterprise {
mapping = "enterprise-event-mapping.json"
type = "enterprise"
max.created.at = "get_last_es_doc.json"
# polling interval in minutes
polling.interval = 2
}

Startup Instructions

  1. Confirm Java is installed by running the following command. 
    java -version
  2. In the terminal, change directories to the unarchived box-elastic-events directory. 
    cd box-elastic-events
  3. Run the startup shell script. 
    ./startup_allinone.sh
  4. Tail the log file to monitor for errors. 
    ls logs
tail -f logs/TYPE_CURRENT_DATE_HERE.log

Validation Steps

  1. Check that Elasticsearch is running by navigating to http://localhost:9200
  2. Check that the "box" Elasticsearch index has been properly created by navigating to http://localhost:9200/box/_settings?pretty
  3. Execute a search GET request by navigating to http://localhost:9200/box/_search?q=*&pretty
    • CAUTION: This will return all docs contained the Elasticsearch box index. See the ES Search documentation to refine the search appropriately.
    • Navigate to Kibana at the following URL: http://localhost:5601

Kibana Configuration

  1. Navigate to Kibana at the following URL: http://localhost:5601
  2. Change the default index name to box and change the Time-field name to created_at alt text
  3. Switch to the Advanced Settings tab
  4. Filter the setting by timelion
  5. Change the timelion:timefield value to created_at
  6. Change the timelion:default_index value to box alt text
  7. Switch to the Saved Objects tab
  8. Click import and navigate to the configuration/kibana-configuration.json file alt text
  9. Navigate to the Kibana Dashboard Page and change the date/time range as needed. alt text

Shutdown Instructions

  1. Run the shutdown_allinone.sh script 
    ./shutdown_allinone.sh

How to Increase Ulimit on Mac OS X

Reference the following blog Ulimit Shenanigans on OS X El Capitan

Disclaimer

This project is an open source extension to the Box platform and is not officially supported by Box, Inc. Use at your own risk. If you encounter any problems, please log an issue.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.