log2timeline.py is broken

[ekristen]

Unsure if this is a SIFT related problem or a GIFT related problem. Investigating.

[ekristen]

This is reproducible using the VM from the SANS website, but not from installing on a fresh ubuntu 14.04 box.

The exact error is below.

Traceback (most recent call last):
  File "/usr/bin/log2timeline.py", line 21, in <module>
    from plaso.cli import extraction_tool
  File "/usr/lib/python2.7/dist-packages/plaso/cli/extraction_tool.py", line 8, in <module>
    from plaso.cli import storage_media_tool
  File "/usr/lib/python2.7/dist-packages/plaso/cli/storage_media_tool.py", line 9, in <module>
    from dfvfs.analyzer import analyzer as dfvfs_analyzer
  File "/usr/lib/python2.7/dist-packages/dfvfs/analyzer/__init__.py", line 4, in <module>
    from dfvfs.analyzer import bde_analyzer_helper
  File "/usr/lib/python2.7/dist-packages/dfvfs/analyzer/bde_analyzer_helper.py", line 4, in <module>
    from dfvfs.analyzer import analyzer
  File "/usr/lib/python2.7/dist-packages/dfvfs/analyzer/analyzer.py", line 9, in <module>
    from dfvfs.resolver import resolver
  File "/usr/lib/python2.7/dist-packages/dfvfs/resolver/__init__.py", line 10, in <module>
    from dfvfs.resolver import cpio_resolver_helper
  File "/usr/lib/python2.7/dist-packages/dfvfs/resolver/cpio_resolver_helper.py", line 6, in <module>
    import dfvfs.vfs.cpio_file_system
  File "/usr/lib/python2.7/dist-packages/dfvfs/vfs/cpio_file_system.py", line 7, in <module>
    from dfvfs.lib import cpio
  File "/usr/lib/python2.7/dist-packages/dfvfs/lib/cpio.py", line 38, in <module>
    class CPIOArchiveFile(object):
  File "/usr/lib/python2.7/dist-packages/dfvfs/lib/cpio.py", line 52, in CPIOArchiveFile
    construct.UBInt16(u'signature'),
AttributeError: 'module' object has no attribute 'UBInt16'

Comparing both the SANS VM vs the fresh VM, the package names are the same, same version, same repo, something else if obviously going on, will have to investigate further.

[ekristen]

This should fix it.

sudo pip uninstall construct

/cc @fetchered

[amplify211]

Today, I downloaded the SIFT VM and did a fresh install.. both of which produce the same issue as described above. When I run "sudo pip uninstall construct", I get "Can't uninstall 'construct'. No files were found to uninstall." Any help would be appreciated.

[ekristen]

hi @amplify211 downloading the SIFT VM and a fresh install is contradictory.

What error are you actually seeing?

[amplify211]

What I meant is that I did both. I downloaded the Sift VM and tried to update..  AND I also created a new Ubuntu VM and ran the script to install Sift..Both VMS produce the same error with log2timeline

-------- Original message -------- From: Erik Kristensen notifications@github.com Date: 10/05/2016 6:10 PM (GMT-05:00) To: sans-dfir/sift sift@noreply.github.com Cc: amplify211 ryan.d.frampton@gmail.com, Mention mention@noreply.github.com Subject: Re: [sans-dfir/sift] log2timeline.py is broken (#128)

hi @amplify211 downloading the SIFT VM and a fresh install is contradictory.

What error are you actually seeing?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/sans-dfir/sift","title":"sans-dfir/sift","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/sans-dfir/sift"}},"updates":{"snippets":[{"icon":"PERSON","message":"@ekristen in #128: hi @amplify211 downloading the SIFT VM and a fresh install is contradictory. \r\n\r\nWhat error are you actually seeing? "}],"action":{"name":"View Issue","url":"https://github.com/sans-dfir/sift/issues/128#issuecomment-251814410"}}}

[ekristen]

Alright, please provide output of the command I provided. If you are indeed getting this error on 14.04, a newer version of construct has to be installed somewhere it shouldn't be.

[Fetchered]

@ekristen Just got to my computer and tried it out: Can't uninstall 'construct'. No files were found to uninstall. Tried apt-get remove construct (just in case) E: Unable to locate package construct sansforensics@siftworkstation:~$ which construct sansforensics@siftworkstation:~$ find / -name construct* Results for the find are in the attached text file.

construct.txt

[ekristen]

Please send me dpkg -l output and pip installed output.

[Fetchered]

Got it, here you go: dpkg.txt pip.txt

python-construct is listed in dpkg, and construct is listed in pip.

[ekristen]

construct is right there in the pip file. It needs to be uninstalled. Pretty sure the commmand I gave is right.

[Fetchered]

Yes the command you gave is right, and when it's run it gives you the error we returned earlier: Can't uninstall 'construct'. No files were found to uninstall. See if you can run the command on your VM. Do you want me to run it in a brand new VM first? Or the ones I already have created.

[ekristen]

It works just fine on my VM. Version2.8.3 is installed via pip and that's the problem. It needs to be removed. I removed it with that command that's why I know it's the fix. :/

[ekristen]

Maybe try installing it again via pip and then uninstall it. Perhaps there is some missing files for some reason.

http://stackoverflow.com/questions/13136060/why-cant-pip-uninstall-pysqlite

[Fetchered]

Problem is that once you've got the VM up and running, and have already run update-sift, then received the Error 100's, then you're at a point where you can't uninstall. The two VM's I've got right now; the brand new one from yesterday where I ran the commands in #106 that ended with update-sift; and the one that I've had for months which I've already done an update-sift quite a few times; both have that problem.

I'm going to install it again and try to remove it properly, then see what happens.

[ekristen]

We are confusing a lot of issues right now.

What are you stuck on exactly?

I've given fixes for both the 100 error on pytsk and mantaray on the other issue and for log2timeline in this issue.

If you've done the instructions in the other issue then update-sift works. However it doesn't fix this issue about log2timeline.

[amplify211]

I have a fresh Ubuntu 14 VM setup.. I am going to take a snapshot before anything else is done so I can revert if something goes awry... The next step to install SIFT is to execute: wget --quiet -O - https://raw.github.com/sans-dfir/sift-bootstrap/master/bootstrap.sh | sudo bash -s -- -i -s -y

Correct?

[ekristen]

@amplify211 that's correct

[Fetchered]

Correct. I'm working on fixing log2timeline. All I'm saying is that a lot of other errors that I've experienced over the last couple of months (pytsk3, Error 100) that we've just fixed, have all continually culminated.

The current issue is now log2timeline. But due to the errors and the previous update-sift commands that I've run, construct now no longer wants to uninstall, because it says it can't find the files. I tried the fixes in the already-built-for-months VM, and successfully managed to get construct uninstalled and am now receiving these errors: construct-2.txt

[Fetchered]

I'm building a brand new VM right now, trying the fix from #106 that we tried yesterday (removing google-chrome.list, mantaray, and the plaso/pytsk3/dfvfs/python-pytsk3 installs), running sudo pip uninstall construct, then doing an update-sift to see what that does.

[amplify211]

I am really puzzled... this is the third time that I have followed the same process... create a fresh Ubuntu 14 install, run the command wget --quiet -O - https://raw.github.com/sans-dfir/sift-bootstrap/master/bootstrap.sh | sudo bash -s -- -i -s -y to install SIFT.. and it WORKS (with some warnings from timesketch about flask_wtf.Form has been renamed... but that's at least tolerable).. I really did not do anything different....

@ekristen I really appreciate your time and help, hopefully I won't have to submit anything else :]

[Fetchered]

@ekristen Okay, so I've built a brand-new VM, and ran the following on first boot: sudo rm /etc/apt/sources.list.d/google-chrome.list sudo apt-get remove python-plaso python-pytsk3 mantaray python-dfvfs sudo apt-get remove python-plaso mantaray python-dfvfs (because I got the error 'python-pytsk3 was not found') sudo apt-get remove pytsk3 sudo apt-get update sudo pip uninstall construct sudo update-sift No errors up until this point. After the sift update completed, I tried log2timeline (legacy) and it's still there. I tried log2timeline.py (plaso) and got this: plaso-error.txt

Nothing else done up to now.

[ekristen]

Looks like it is working. I have a question into the plaso team about the forms.py warnings, but its just a warning. You should be good to go.

[Fetchered]

FYI: On the VM I've had for the last few months, I managed to get to this point by following my previously mentioned commands (thanks to @ekristen), but also by installing an older version of construct: sudo pip install construct==2.5.3 This is the latest version that will actually get log2timeline.py running at the same standard as the new VM previously mentioned. According to the log2timeline git, the version of construct that gets installed with update-sift is 'too new' and must be 2.6.0 or earlier. wget https://raw.githubusercontent.com/log2timeline/plaso/master/utils/check_dependencies.py

Hope this helps anyone else monitoring the situation.

[ekristen]

update-sift no longer installs construct from pip, only from apt which is the right version. Installing construct via pip is not supported by plaso.

Important: Please do not install construct from pip.

[Fetchered]

Possible fix, submitting for review: Currently installed version of Flask-WTF is likely 0.13 (it is for mine). Upgraded with pip to 0.13.1, log2timeline no longer gives me those errors mentioned above.

[Fetchered]

Looks like they might making a change to the install then. According to https://github.com/log2timeline/plaso/blob/master/requirements.txt, both pip and construct are requirements, but not python-construct. And https://github.com/log2timeline/plaso/blob/master/setup.cfg, says that python-construct >=2.5.2 is required.

Both were edited 'just now'.

[joachimmetz]

@Fetchered and according to: https://github.com/log2timeline/plaso/wiki/Running-plaso-in-virtualenv

Setting up plaso in virtualenv

For development purposes, Plaso can be installed using virtualenv.

Note that this is intended for development use only, and if you aren't comfortable debugging package installation, this is not for you.

Note that using pip outside virtualenv is not recommended since it ignores your systems package manager.

pip is NOT a preferred plaso installation method We recommend NOT to use pip and dpkg side by side without virtualenv

[joachimmetz]

Looks like it is working. I have a question into the plaso team about the forms.py warnings, but its just a warning.

Flask is not a plaso dependency

[ekristen]

Closing this issue as it has been inactive.