elastic.py causing error message at psort.py

[robeweber]

I really loved the idea of using plaso with something other then excel as analysis tool. Love this approach. Installed ubuntu 14.04 yesterday, followed your install steps to the i as described in part 1 in your blog. Kibana and your efetch test case work fine until I add the new elastic.py from this repository into the mix. I figured out that I need this new elastic search output module as I am missing crucial fields in the output like ext.raw, dir etc. when I changed the elastic.py to this version I get:

quote: Traceback (most recent call last): File "/usr/bin/psort.py", line 31, in from plaso.cli.helpers import manager as helpers_manager File "/usr/lib/python2.7/dist-packages/plaso/cli/helpers/init.py", line 4, in from plaso.cli.helpers import dynamic_output File "/usr/lib/python2.7/dist-packages/plaso/cli/helpers/dynamic_output.py", line 7, in from plaso.output import dynamic File "/usr/lib/python2.7/dist-packages/plaso/output/init.py", line 14, in from plaso.output import timesketch_out File "/usr/lib/python2.7/dist-packages/plaso/output/timesketch_out.py", line 17, in from plaso.output.elastic import ElasticSearchHelper ImportError: cannot import name ElasticSearchHelper unquote:

I am not a newby to computer forensics, ubuntu, elasticsearch or kibana (and logstash), however I am not an expert in plaso. Maybe I am overlooking somethin?

[maurermj08]

As of Plaso 1.5 this module is no longer needed. Only the Python pyelasticsearch library is required. I believe I removed all the references to this module in my blog. If you need help returning the original elastic output module or have any other questions about exporting to Elasticsearch 2 let me know.

[maurermj08]

I went through the steps again as posted in my blog and I realize there was a problem. I still had "pip install elasticsearch", but it should have been "sudo pip install pyelasticsearch". I updated the post. Without installing pyelasticsearch the elastic output module option will not be available in psort.py. Thanks for the question, I believe it will definitely help some people out!

[robeweber]

uninstalled using pip uninstall elasticsearch, did sudo pip install pyelasticsearch.

What your blog describes works just fine in part 1. Part 2 however describes customizations in the dashboard visualization based on fields that do not exist in the standard psort output to elasticsearch. The field ext.raw for example used in the pichart visualization of blog part 2 does not exist in kibana and can therefore not be selected as term.

Also, what I liked in this github project was that you had the custom elastic.py output module generate the fantastic efetch fields and links as well which creates the efetch functionality when analysing an E01 image in a very intuitive way.

Can you elaborate where I might still be going wrong in my setup?

[maurermj08]

This output module was built and tested for Plaso 1.4 and does not work with Plaso 1.5. I personally use Plaso 1.4 still primarily due to this: https://github.com/log2timeline/plaso/issues/1073. However, the elastic output module that comes with 1.5 is more stable and more thoroughly tested than my output module.

I too believe the biggest disadvantage with using the Plaso 1.5 elastic output module is the lack of the directory and extension fields. However, I discussed with with Johan (output module creator) and we agreed that the output module is not the right place to add these fields. If more people want this feature an issue (enhancement) should be created on the Plaso github page requesting these fields be added to the filestat plugin.

  name = os.path.basename(filename)
  directory = os.path.dirname(filename) + '/'
  ext = os.path.splitext(name)[1][1:] or ""

As far as creating links to efetch, the "--raw_fields" option in Plaso 1.5 will create a pathspec.raw that can be used to easily link to efetch.

PS: I am working on a blog post that just happens to cover a few of these issues, but it probably won't be posted until next week

[robeweber]

I understand, thanks for the explenation. I have searched using apt-cache showpkg python-plaso to find an earlier version that I then could install using sudo apt-get install python-plaso=Version#, but without any luck. The only verion available in apt-cache is 1.5.1-1ppa1~trusty. Any chance you can give me a hint how to get 1.4.0 or 1.5.0 installed (preferably 1.4 so I can play with your custom elastic.py and learn)? By the way, I reverted back to Elasticsearch 1.7 and Kibana 4.1.6 as I assume that those are the best options to go with plaso 1.4.0. Appreciate it.

[maurermj08]

I actually do not know the proper way to install the previous Plaso version 1.4. I am interested to find that out myself, I will do some testing later this week.

I recommend using the most up to date 2.x version of Elasticsearch and the latest 4.x version of Kibana (prior to the 5 release) with Plaso 1.4 and the efetch elastic output module.

[robeweber]

I got 1.4.0 running now. log2timeline.py is busy at the moment (a lot longer then 1.5.1, which explains the complaints :-)) Will run psort.py tomorrow and report back. Where should I post the path to getting this installed on a barebone Ubuntu 14.04 virtual machine?

[robeweber]

Managed to install Plaso 1.4.0 and got your elastic output script to work propperly. Here is how I did it:

Get Google repository: add google repository sudo add-apt-repository ppa:gift/stable sudo apt-get update

Add packages required for Plaso 1.4.0: sudo apt-get install git python-pip build-essential autotools-dev libsqlite3-dev python-dev debhelper devscripts fakeroot quilt git mercurial python-setuptools libtool automake sleuthkit python-pytsk3 python-dateutil python-yaml python-protobuf python-protobuf.socketrpc python-zmq libregf-python python-binplist python-bencode liblnk-python libfwsi-python libesedb-python python-pyparsing python-hachoir-core python-hachoir-parser python-hachoir-regex python-hachoir-subfile python-hachoir-urwid python-hachoir-wx python-hachoir-metadata libmsiecf-python libolecf-python libevt-python libevtx-python python-psutil libsigscan-python python-construct libsmdev-python libfsntfs-python python-exif clamav sudo pip install pytz sudo pip install pefile

Get Plaso 1.4.0: wget https://github.com/log2timeline/plaso/releases/download/1.4.0/plaso-1.4.0.tar.gz sudo pip install plaso-1.4.0.tar.gz

Now Install Elasticsearch 2.3.3 and Kibana 4.5.1 sudo apt-get install default-jre sudo pip install pyelasticsearch wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.3.3/elasticsearch-2.3.3.deb sudo dpkg -i elasticsearch-2_.deb wget https://download.elastic.co/kibana/kibana/kibana_4.5.1_amd64.deb sudo dpkg -i kibana4.deb

Manually start elasticsearch and kibana sudo service elasticsearch start sudo service kibana start

Install Elasticsearch and Kibana as services so they run automatically at boot sudo update-rc.d elasticsearch defaults 95 10 [Kibana file downloaded earlier: "wget https://raw.githubusercontent.com/rlwmmw/kibana-4.2-init/master/etc/init.d/kibana" and changed run path in order to match this installation: PROG_PATH="/opt/kibana/bin" ] sudo cp kibana /etc/init.d/kibana sudo chmod +x /etc/init.d/kibana sudo update-rc.d kibana defaults 96 9

Install Efetch: wget --quiet -O - https://git.io/vPFgD | sudo bash -s

Important to install this one last, as somewhere before here a different, newer version of dfvfs is automatically installed and below version is the last that works with plaso 1.4.0 wget https://github.com/log2timeline/dfvfs/releases/download/20160108/dfvfs-20160108.tar.gz tar -xvf dfvfs-20160108.tar.gz cd dfvfs-20160108/ ./setup.py build sudo ./setup.py install

Download and install the Kibana json objects from this post (in Kibana).

USAGE +++ USAGE +++ USAGE Start efetch in terminal: efetch

Create plaso file: log2timeline.py casename.plaso /mnt/hgfs/peek/EfetchTestCase.E01

Output to Elasticsearch: psort.py -o elastic --case_name peek ~/casename.plaso

Worked like a charm for me - Thank you for this great Plaso contribution

[robeweber]

Oh, forgot to mention. For complete functionality as intended by the efetch kibana combination, modifications in the efetch output script as well as the Kibana dashboard are required. I am adding some more features into the dashboard (specific visualizations that help a digital forensic technitian to analyse a computer image) at present, to help me with a current investigation. Maybe I should write a blog of my own detailing the steps involved. But that would have to wait until early next year.

[maurermj08]

Thanks for the information and I am glad everything worked! There is definitely enough there to write a blog post! If you are interested, in the efetch git repo there is a Kibana plugin that you can try out. You just need to enter the URL, i.e. "http://localhost:8080/plugins/fa_timeline".