rekall appears to be broken in v2018.08.0

[jsolderitsch]

I did a fresh manual install using sift-cli into ubuntu 16.04.4. It completed without errors.

I have an old memory dump zeus.vmem that I want to process using rekall.

If I do rekall -v I get:

2018-03-30 02:04:21,439:DEBUG:rekall.1:Logging level set to 10 2018-03-30 02:04:21,441:DEBUG:rekall.1:Running plugin (shell) with args (()) kwargs ({'profile': None})

---

The Rekall Digital Forensic/Incident Response framework 1.7.2.rc1 (Hurricane Ridge).

"We can remember it for you wholesale!"

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License.

See http://www.rekall-forensic.com/docs/Manual/tutorial.html to get started.

But then if I run:

rekall -f zeus.vmem pslist

I get:

2018-03-30 02:06:45,529:WARNING:rekall.1:Inventory for repository "https://raw.githubusercontent.com/google/rekall-profiles" seems malformed. Are you behind a captive portal or proxy? If this is a custom repository, did you forget to create an inventory? You must use the tools/profiles/build_profile_repo.py tool with the --inventory flag. 2018-03-30 02:06:45,530:WARNING:rekall.1:Repository https://raw.githubusercontent.com/google/rekall-profiles will be disabled. 2018-03-30 02:06:45,530:WARNING:rekall.1:No usable repositories were found. Rekall Will attempt to use the local cache. This is likely to fail if profiles are missing locally! 2018-03-30 02:06:46,262:ERROR:rekall.1:No profiles match this image. Try specifying manually. Traceback (most recent call last):
File "/usr/local/bin/rekall", line 11, in load_entry_point('rekall-core==1.7.2rc1', 'console_scripts', 'rekall')() File "/opt/rekall/local/lib/python2.7/site-packages/rekall/rekal.py", line 98, in main user_session=user_session) File "/opt/rekall/local/lib/python2.7/site-packages/rekall/args.py", line 432, in parse_args command_metadata = user_session.plugins.Metadata(plugin_name) File "/opt/rekall/local/lib/python2.7/site-packages/rekall/session.py", line 142, in Metadata return self.plugin_db.GetActivePlugin(name) File "/opt/rekall/local/lib/python2.7/site-packages/rekall/plugin.py", line 982, in GetActivePlugin if plugin_cls.is_active(self.session): File "/opt/rekall/local/lib/python2.7/site-packages/rekall/plugin.py", line 434, in is_active profile = (session.profile != None and File "/opt/rekall/local/lib/python2.7/site-packages/rekall_lib/utils.py", line 1091, in get return super(safe_property, self).get(*args, **kwargs) File "/opt/rekall/local/lib/python2.7/site-packages/rekall/session.py", line 1062, in profile res = self.GetParameter("profile_obj") File "/opt/rekall/local/lib/python2.7/site-packages/rekall/session.py", line 768, in GetParameter result = self._RunParameterHook(item) File "/opt/rekall/local/lib/python2.7/site-packages/rekall/session.py", line 803, in _RunParameterHook result = hook.calculate() File "/opt/rekall/local/lib/python2.7/site-packages/rekall/plugins/guess_profile.py", line 764, in calculate "Unable to find a valid profile for this image. " RuntimeError: Unable to find a valid profile for this image. Try using -v for more details.

Note that if I install rekall into an un-sift-ified ubuntu VM using the virtualenv method rekall works.

Can anyone reproduce this or help me understand what I am doing wrong?

[jsolderitsch]

I am connected to the internet without a proxy when I try this. I just did the virtualenv style of rekall setup and I still get this error inside of sift.

[jsolderitsch]

Looks like ~/.rekallrc needs a tweak.

In the version of ubuntu where rekall worked I see these lines in .rekallrc:

repository_path:
- https://github.com/google/rekall-profiles/raw/master
- http://profiles.rekall-forensic.com

After changing the lines that were there in the sift vm:

repository_path:
  - https://raw.githubusercontent.com/google/rekall-profiles

to these, rekall once again works in the sift vm. This is true for the rekall version in this Sift release and for the one I installed using the virtualenv method.

Not sure who/where the change needs to be made.

[ekristen]

@jsolderitsch great info and troubleshooting. Thanks for this. I'm the person to talk to. I'm not sure why you are hitting this unless your SIFT install didn't work full, you are on an old version or something overrode the .rekallrc file.

It seems like our rc file is already setup to use that path ... https://github.com/sans-dfir/sift-saltstack/blob/9d08f3e40b11ec7d07559b280a4e414967e07c2a/sift/config/user/files/rekall-profile.txt

and we install it to the user that configures sift here ... https://github.com/sans-dfir/sift-saltstack/blob/9d08f3e40b11ec7d07559b280a4e414967e07c2a/sift/config/user/rekall.sls#L9

[jsolderitsch]

Thanks for the reply. I made a brand new install of Sift (the v2018.08.0 version) into a brand new Ubuntu instance using the Sift CLI. I did not experience any errors that were reported. I immediately ran rekall (the Sift installed one) and got the error I reported. Installing rekall fresh into another ubuntu 16.04 without Sift did not have the error. Checking .rekallrc in each showed the difference. I replace the Sift one with the working one and my error went away.

Don't know what else I can say.

[jsolderitsch]

Sift installs the wrong/broken one -- your first link shows that. So it seems Sift needs fixing. I think the github site for Rekall: [(https://github.com/google/rekall-profiles)] gives advice to use this one and that appears to also not work.

[ekristen]

Oh I read your profiles backwards. I'll take a look. Thanks.

[ekristen]

Resolved by https://github.com/sans-dfir/sift-saltstack/commit/265c354e771305c9613cd7296cde32517c06e6e8

[johnmccash]

I just upgraded to v2018.16.0, and this doesn't appear to be fixed...

[johnmccash]

Question: Is the rekall in Sift modified at all from the standard one? I was trying to get it to work with a memory dump from last year's DFIR Netwars tournament, at the DFIR Summit, and I can't get it to recognize it at all. I even tried using the rekall from last year's Win10 Sift workstation, and it fails as well. Wondering if they changed something and it doesn't work anymore, or if there was just something special in Sift that doesn't work anymore. Thoughts? The memory image is named Post_Malware.raw, sha256sum: c12f79231fefaf61d4cd7d8baf4238017a18f9a285ee9b1938afc4065a93e40b. I can share a link to a cloud drive of it if necessary.

[jsolderitsch]

I can try to run rekall in Sift with this memory image if you can link to it somehow. I have not updated my Sift since I reported the error and discovered the work-around I posted.

I used the rekall from SIft and the rekall installed using the virtualenv method and both worked once I fixed the broken .rekallrc

[ekristen]

It is not modified, it is installed per their instructions in a virtualenv

Sent from my iPhone

On Apr 26, 2018, at 17:11, jsolderitsch notifications@github.com wrote:

I can try to run rekall in Sift with this memory image if you can link to it somehow. I have not updated my Sift since I reported the error and discovered the work-around I posted.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

[johnmccash]

I can email you a link directly, but don't want to distribute the image publicly, as it's part of the Netwars content, and is probably entailed in some way by the user agreement. It definitely worked, at the time, with the version of SIFT distributed with FOR508 last June. But if I go back to that original SIFT image, it no longer works, because the http links to their online profiles aren't correct anymore. I'm unable to get it to work with any version of Rekall I've been able to dig up.

[jsolderitsch]

jsolderitsch<at>gmail><dot>com works if you want to proceed further

[jsolderitsch]

I was able to open the image in my copy of Sift.

My .rekallrc is:

$ more ~/.rekallrc
cache_dir: .rekall_cache
repository_path:
- https://github.com/google/rekall-profiles/raw/master
- http://profiles.rekall-forensic.com

[johnmccash]

That's odd... After replacing the contents of ~/.rekallrc in the current SIFT kit version with yours, I still get:

$ rekall -f Post_Malware.raw imageinfo Traceback (most recent call last): File "/usr/local/bin/rekall", line 11, in load_entry_point('rekall-core==1.7.2rc1', 'console_scripts', 'rekall')() File "/opt/rekall/local/lib/python2.7/site-packages/rekall/rekal.py", line 98, in main user_session=user_session) File "/opt/rekall/local/lib/python2.7/site-packages/rekall/args.py", line 432, in parse_args command_metadata = user_session.plugins.Metadata(plugin_name) File "/opt/rekall/local/lib/python2.7/site-packages/rekall/session.py", line 142, in Metadata return self.plugin_db.GetActivePlugin(name) File "/opt/rekall/local/lib/python2.7/site-packages/rekall/plugin.py", line 982, in GetActivePlugin if plugin_cls.is_active(self.session): File "/opt/rekall/local/lib/python2.7/site-packages/rekall/plugin.py", line 434, in is_active profile = (session.profile != None and File "/opt/rekall/local/lib/python2.7/site-packages/rekall_lib/utils.py", line 1091, in get return super(safe_property, self).get(*args, **kwargs) File "/opt/rekall/local/lib/python2.7/site-packages/rekall/session.py", line 1062, in profile res = self.GetParameter("profile_obj") File "/opt/rekall/local/lib/python2.7/site-packages/rekall/session.py", line 768, in GetParameter result = self._RunParameterHook(item) File "/opt/rekall/local/lib/python2.7/site-packages/rekall/session.py", line 803, in _RunParameterHook result = hook.calculate() File "/opt/rekall/local/lib/python2.7/site-packages/rekall/plugins/guess_profile.py", line 737, in calculate "filename %s." % filename) RuntimeError: Unable to instantiate physical_address_space from filename Post_Malware.raw.

Any idea what's going on? John

[ekristen]

Unfortunately I’m not a super rekall expert. Looks like the profile configs are matching now.

Erik

Sent from my iPhone

On Apr 27, 2018, at 12:39, johnmccash notifications@github.com wrote:

That's odd... After replacing the contents of ~/.rekallrc in the current SIFT kit version with yours, I still get:

$ rekall -f Post_Malware.raw imageinfo Traceback (most recent call last): File "/usr/local/bin/rekall", line 11, in load_entry_point('rekall-core==1.7.2rc1', 'console_scripts', 'rekall')() File "/opt/rekall/local/lib/python2.7/site-packages/rekall/rekal.py", line 98, in main user_session=user_session) File "/opt/rekall/local/lib/python2.7/site-packages/rekall/args.py", line 432, in parse_args command_metadata = user_session.plugins.Metadata(plugin_name) File "/opt/rekall/local/lib/python2.7/site-packages/rekall/session.py", line 142, in Metadata return self.plugin_db.GetActivePlugin(name) File "/opt/rekall/local/lib/python2.7/site-packages/rekall/plugin.py", line 982, in GetActivePlugin if plugin_cls.is_active(self.session): File "/opt/rekall/local/lib/python2.7/site-packages/rekall/plugin.py", line 434, in is_active profile = (session.profile != None and File "/opt/rekall/local/lib/python2.7/site-packages/rekall_lib/utils.py", line 1091, in get return super(safe_property, self).get(*args, **kwargs) File "/opt/rekall/local/lib/python2.7/site-packages/rekall/session.py", line 1062, in profile res = self.GetParameter("profile_obj") File "/opt/rekall/local/lib/python2.7/site-packages/rekall/session.py", line 768, in GetParameter result = self._RunParameterHook(item) File "/opt/rekall/local/lib/python2.7/site-packages/rekall/session.py", line 803, in _RunParameterHook result = hook.calculate() File "/opt/rekall/local/lib/python2.7/site-packages/rekall/plugins/guess_profile.py", line 737, in calculate "filename %s." % filename) RuntimeError: Unable to instantiate physical_address_space from filename Post_Malware.raw.

Any idea what's going on? John

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

[jsolderitsch]

$ rekall -f Post_Malware.raw imageinfo
        key          value
-------------------- -----
Kernel DTB           0x187000
NT Build             7601.win7sp1_rtm.101119-1850
NT Build Ex          7601.17514.amd64fre.win7sp1_rtm.101119-1850
Signed Drivers       True 
Time (UTC)           2015-12-30 20:39:32Z
Time (Local)         2015-12-30 15:39:32-0500
Sec Since Boot       516.6441118
NtSystemRoot         C:\Windows

I am also not an expert but the imageinfo command succeeds for me.

$ rekall --version
This is Rekall Version 1.7.1 (Hurricane Ridge)
{'pep440': u'1.7.1', u'rc': u'0', u'version': u'1.7.1', 'error': 'Not in a git repository.', u'codename': u'Hurricane Ridge', u'post': u'0'}

Perhaps wipe away the rekall cache and then try again?

[johnmccash]

OK, so are you running it from SIFT v2018.16.0, or From SIFT v2018.08.0, or from something earlier? I just went back to the SIFT version from my 508 class last June, and part of my problem in that instance is an apparent problem with the version of vmware tools, that's preventing access to the host filesystem, so I don't really know if it still works from there if you fix the .rekallrc file or not. I'll dig some more on that.

[jsolderitsch]

I am still using SIFT v2018.08.0 it would seem.

sudo sift list-upgrades
> sift-cli@1.5.1-beta.0-master.154cb2f
> sift-version: v2018.08.0

> List of available releases
  - v2018.16.0

Since I had some time I updated Sift to v2018.16.0 and rekall continues to work.

[johnmccash]

Then either the SIFT port of Rekall in the v2018.16.0 release has something more broken in it than the URLs in the .rekallrc file, or else the Rekall maintainers changed something between those releases that broke it.

[johnmccash]

Hmmmm.... Just noticed you upgraded to 16 and it's still working for you... Obviously there's something different between our two installs.

[jsolderitsch]

Definitely something is different for us. The latest Sift release as well as the prior one both have working rekall software for me with the updated .rekallrc as I have noted elsewhere.

[johnmccash]

Nevermind. I cleaned out the cache, and now it works. Sorry for all the kerfluffle. John

[jsolderitsch]

Good to hear. Erik can close the re-opened issue then.

[johnmccash]

as long as the .rekallrc file is correctly updated now.

[ekristen]

Sounds like it was a cache problem? So we are good to go?

Sent from my iPhone

On Apr 27, 2018, at 15:21, johnmccash notifications@github.com wrote:

as long as the .rekallrc file is correctly updated now.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

[johnmccash]

NTAC:3NS-20 Erik, I fixed the issue that remained after my successful upgrade from v2018.08.0 to v2018.p16.0 (which only worked because I manually downgraded pip to v9, and then used apt-get to reinstall pip3), by manually editing .rekallrc, and making the specified change. Assuming that the pip issue is resolved, and that the change you put in fixes the .rekallrc file we should be good.

            I'm guessing that the creation of the .rekallrc file is done the first time you run recall? If so, then the bad data in my file would have dated from before the v2018.p16.0 upgrade. You may want to make the execution of sift upgrade also fix any current .rekallrc files that are resident under user profiles on the upgraded system, just to avoid further problems.

                            John

From: Erik Kristensen [mailto:notifications@github.com] Sent: Friday, April 27, 2018 4:49 PM To: sans-dfir/sift Cc: John Mccash; Comment Subject: [EXT] Re: [sans-dfir/sift] rekall appears to be broken in v2018.08.0 (#247)

Sounds like it was a cache problem? So we are good to go?

Sent from my iPhone

On Apr 27, 2018, at 15:21, johnmccash notifications@github.com<mailto:notifications@github.com> wrote:

as long as the .rekallrc file is correctly updated now.

- You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

- You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_sans-2Ddfir_sift_issues_247-23issuecomment-2D385102945&d=DwMFaQ&c=K5gMqH44tVpW9Mb7NvpzqAFAhrpSdUITR819D8huNsU&r=sJCE6izyLG2FavzcHWpZ_Q&m=j09-h8bKTxsOVyifLAkvt-e-93WU3-2FhgbhzeAA11Q&s=ei6tHYKHVejLFKiLFsumoKgWVjbkT_FBOVt3tw1dcrs&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ACtvVHJqPtihZ57IZzpyshSJXKfnG2ITks5ts5JjgaJpZM4TBNoD&d=DwMFaQ&c=K5gMqH44tVpW9Mb7NvpzqAFAhrpSdUITR819D8huNsU&r=sJCE6izyLG2FavzcHWpZ_Q&m=j09-h8bKTxsOVyifLAkvt-e-93WU3-2FhgbhzeAA11Q&s=pqdJhrc18v6MI1xjkMibR4UtmiRnl7mfrpmN7d6aCT4&e=.

[ekristen]

We ensure the right rekallrc file is in place each run of the sift cli tool, but if dependencies fail it won't get fixed.

The pip to 9.0.3 is a manual thing unfortunately.

I have been unable to duplicate your pip3 problems but I'm glad you got it resolved.

I'll close this for now, and I'll probably send out a notice to the DFIR mailing list about pip.