gatopardos90
commented
6 years ago 
Closed gatopardos90 closed 6 years ago
gatopardos90
commented
6 years ago
ekristen
commented
6 years ago Looks like the same issues that others have reported. It seems that Machinae might have changed their dependencies.
Sent from my iPhone
On Jul 3, 2018, at 15:21, gatopardos90 notifications@github.com wrote:
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.
WhistleMaster
commented
6 years ago Same issue here, on a fresh 16.04 install, which seems to be linked to those issues as well: https://github.com/sans-dfir/sift/issues?utf8=%E2%9C%93&q=machinae
ekristen
commented
6 years ago I can't seem to reproduce this, this is why I haven't fixed it yet.
A clean install works, an the update on my system worked just fine.
Anyone experiencing this issue, please try and run the following and upload the output to the issue
sudo salt-call -l debug --local --file-root /var/cache/sift/cli/v2018.26.0/sift-saltstack-2018.26.0 state.sls sift.python-packages.machinae
darendarrow
commented
6 years ago $ sudo salt-call -l debug --local --file-root /var/cache/sift/cli/v2018.26.0/sift-saltstack-2018.26.0 state.sls sift.python-packages.machinae
[sudo] password for ddarrow:
Traceback (most recent call last):
File "/usr/bin/salt-call", line 11, in <module>
salt_call()
File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 391, in salt_call
import salt.cli.call
File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 9, in <module>
import salt.cli.caller
File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 19, in <module>
import salt.minion
File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 78, in <module>
import salt.pillar
File "/usr/lib/python2.7/dist-packages/salt/pillar/__init__.py", line 18, in <module>
import salt.fileclient
File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 29, in <module>
import salt.utils.templates
File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 31, in <module>
import salt.utils.http
File "/usr/lib/python2.7/dist-packages/salt/utils/http.py", line 71, in <module>
import requests
File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", line 84, in <module>
from urllib3.contrib import pyopenssl
File "/usr/local/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 46, in <module>
import OpenSSL.SSL
File "/usr/lib/python2.7/dist-packages/OpenSSL/__init__.py", line 8, in <module>
from OpenSSL import rand, crypto, SSL
File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 118, in <module>
SSL_ST_INIT = _lib.SSL_ST_INIT
AttributeError: 'module' object has no attribute 'SSL_ST_INIT'More info on my install: https://gist.github.com/darendarrow/490bd60ddd6c539d2edc5186b2e37550
ekristen
commented
6 years ago @darendarrow that would seem to indicate your python install is broken and or you have problems with your apt packages.
darendarrow
commented
6 years ago I agree, but it was a fresh install. I've blown away that image and will try again and see if it's the same.
ekristen
commented
6 years ago @darendarrow if it happens again, can you please try and take a look at just installing a few apt packages manually, same with python.
crahan
commented
6 years ago The issue that @darendarrow is experiencing (with pyOpenSSL) is what I'm also seeing after trying to install on 16.04. The install first fails on dnspython3 (which is installed as a dependency for machinae) and then when you try anything else you will get the pyOpenSSL error.
pip2 install -U pyOpenSSL will fix the second issue. Output for the salt-call command above for me is:
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: sift
[DEBUG ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Please install 'virt-what' to improve results of the 'virtual' grain.
[DEBUG ] Determining pillar cache
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded state.sls
[DEBUG ] LazyLoaded saltutil.is_running
[DEBUG ] LazyLoaded grains.get
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG ] Updating roots fileserver cache
[DEBUG ] Determining pillar cache
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[INFO ] Loading fresh modules for state activity
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] Could not find file 'salt://sift/python-packages/machinae.sls' in saltenv 'base'
[DEBUG ] Could not find file 'salt://sift/python-packages/machinae/init.sls' in saltenv 'base'
[DEBUG ] compile template: False
[ERROR ] Template was specified incorrectly: False
[DEBUG ] LazyLoaded highstate.output
local:
Data failed to compile:
----------
No matching sls found for 'sift.python-packages.machinae' in env 'base'@crahan that just means that the files are not present. Can you give me a list of /var/cache/sift/cli/. It sounds like you might not have tried to install or upgrade to the latest which would mean those files aren't present.
Thanks for the information on pyopenssl, I'm still trying to reproduce this locally, but haven't been able to. I'll be curious to see if now that you have fixed pyopenssl if the machinae state can be installed.
howlingshiba
commented
6 years ago I'm also having issues installing latest SIFT via sift-cli tool. It appears to be the issue with the machinae package which wants to install dnspython3 package, but fails to do so because the saltstack is trying to install dnspython3 using pip instead of pip3.
Based upon research and testing with different versions of saltstack, the error may be related to a bug in salt-minion which disregards the specified "bin_env" value since version 2017.7.6: https://github.com/saltstack/salt/issues/48122
I've done some simple test with using machinae package and saltstack versions: 2017.7.6, 2017.7.7, 2018.3.0, 2018.3.1, 2018.3.2 and they all fail to install correctly. Testing with saltstack version 2017.7.5 did not have issues installing machinae and dnspython3. However, I have not yet tested, if saltstack version 2017.7.5 works with sift-saltstack.
I believe the current sift-cli uses saltstack version 2017.7.7, is there a way to downgrade to version 2017.7.5?
crahan
commented
6 years ago I can confirm @howlingshiba's analysis. When trying to manually install machinae (and dnspython3) the error you get when using pip2 instead of pip3 is the same as what we're tracking in this thread. When you manually use pip3 then both dnspython3 and machinae are installed without issues. However as soon as you run sudo sift install it will use pip from python2 instead of python3 to install machinae.
@ekristen here's the file list you requested. Fixing pyOpenSSL does not resolve the issue however. Since the initial Sift install did not complete it tries to install from scratch which ends up in the same situation as before: dnspython3 being installed with pip2 instead of pip3.
crahan@sift:~$ ls -l /var/cache/sift/cli/
total 4
drwxr-xr-x 3 root root 4096 Jul 7 20:59 v2018.26.1
crahan@sift:~$ ls -l /var/cache/sift/cli/v2018.26.1/
total 11816
-rw-r--r-- 1 root root 338335 Jul 7 21:32 results.yml
-rw-r--r-- 1 root root 1837083 Jul 7 21:32 saltstack.log
drwxrwxr-x 5 root root 4096 Jun 25 13:40 sift-saltstack-2018.26.1
-rw-r--r-- 1 root root 9901009 Jul 7 20:59 sift-saltstack-v2018.26.1.tar.gz
-rw-r--r-- 1 root root 203 Jul 7 20:59 sift-saltstack-v2018.26.1.tar.gz.asc
-rw-r--r-- 1 root root 104 Jul 7 20:59 sift-saltstack-v2018.26.1.tar.gz.sha256
-rw-r--r-- 1 root root 356 Jul 7 20:59 sift-saltstack-v2018.26.1.tar.gz.sha256.asc
crahan@sift:~$ ls -l /var/cache/sift/cli/v2018.26.1/sift-saltstack-2018.26.1/
total 24
-rw-rw-r-- 1 root root 377 Jun 25 13:40 Dockerfile
-rw-rw-r-- 1 root root 1071 Jun 25 13:40 LICENSE.md
-rw-rw-r-- 1 root root 2294 Jun 25 13:40 README.md
drwxrwxr-x 2 root root 4096 Jun 25 13:40 scripts
drwxrwxr-x 9 root root 4096 Jun 25 13:40 sift
-rw-rw-r-- 1 root root 11 Jun 25 13:40 VERSION
crahan@sift:~$ ls -l /var/cache/sift/cli/v2018.26.1/sift-saltstack-2018.26.1/sift/
total 44
drwxrwxr-x 4 root root 4096 Jun 25 13:40 config
drwxrwxr-x 10 root root 4096 Jun 25 13:40 files
drwxrwxr-x 3 root root 12288 Jun 25 13:40 packages
-rw-rw-r-- 1 root root 297 Jun 25 13:40 pkgs.sls
drwxrwxr-x 2 root root 4096 Jun 25 13:40 python-packages
drwxrwxr-x 2 root root 4096 Jun 25 13:40 repos
drwxrwxr-x 2 root root 4096 Jun 25 13:40 scripts
drwxrwxr-x 2 root root 4096 Jun 25 13:40 tools
-rw-rw-r-- 1 root root 420 Jun 25 13:40 vm.sls@crahan change the command to use /var/cache/sift/cli/v2018.26.1/sift-saltstack-2018.26.1/ as the file-root
@howlingshiba interesting, thanks for the research. I can confirm that this is a bug specific with saltstack it seems, and my test environment is on a slightly older version, thus the inability to reproduce.
Thank you all for the hard work in troubleshooting this issue!
Based on the saltstack issues/pull requests this will be fixed in the next version of saltstack, 2017.7.8.
I will see what options we have in the short term to make a workaround/fix for this.
howlingshiba
commented
6 years ago After a bit more testing, I've come up with a sort of an obscure workaround to getting sift-cli installed without issues on a fresh 16.04 install. As stated previously, the problem appears to be a bug in saltstack versions (2017.7.6+) and sift-cli currently uses version 2017.7.7.
saltstack version 2017.7.5 does not have this problem, so the solution is to use saltstack version 2017.7.5.
Probably the best(?) temporary solution is to take sift-cli.js source code and change any reference of
http://repo.saltstack.com/apt/ubuntu/16.04/amd64/2017.7 xenial main to http://repo.saltstack.com/apt/ubuntu/16.04/amd64/archive/2017.7.5 xenial main and rebuild the binary. Unfortunately, I'm not really sure how to build sift-cli from source to linux binaries, so possible short term workaround is below.
Short term workaround on fresh 16.04 1) Install saltstack version 2017.7.5 first. You may need to enter password for sudo after the wget command.
wget -O - https://repo.saltstack.com/apt/ubuntu/16.04/amd64/archive/2017.7.5/SALTSTACK-GPG-KEY.pub | sudo apt-key add -
echo "deb http://repo.saltstack.com/apt/ubuntu/16.04/amd64/archive/2017.7.5 xenial main" | sudo tee /etc/apt/sources.list.d/saltstack.list
sudo apt-get update
sudo apt-get install salt-minion
sudo service salt-minion stop
Running `salt-minion --version` command should return `salt-minion 2017.7.5 (Nitrogen)`2) Next, we need to modify /etc/apt/sources.list.d/saltstack.list to have http://repo.saltstack.com/apt/ubuntu/16.04/amd64/2017.7 xenial main. This is because saltCheckVersion() function checks against this value and if it's different, sift-cli will install latest saltstack 2017.7 version which is 2017.7.7 (not what we want). So run the command below to make the change and prevent sift-cli from installing version 2017.7.7:
echo "deb http://repo.saltstack.com/apt/ubuntu/16.04/amd64/2017.7 xenial main" | sudo tee /etc/apt/sources.list.d/saltstack.list3) Install sift as usual
sudo sift install4) As sift is installing, after about two minutes or so to ensure it bypassed the saltstack version check, open another terminal window and run the following command to change back the /etc/apt/sources.list.d/saltstack.list to include version 2017.7.5. This is done because during the wine package installation, sift-cli does apt-get update and if we don't change /etc/apt/sources.list.d/saltstack.list back to 2017.7.5, it'll upgrade saltstack to version 2017.7.7 and fail to install machinae package later.
echo "deb http://repo.saltstack.com/apt/ubuntu/16.04/amd64/archive/2017.7.5 xenial main" | sudo tee /etc/apt/sources.list.d/saltstack.listThis should successfully install sift-cli without error. Hopefully, I didn't make any typos. I'll try to run this workaround again in the morning and see if I experience any issues.
crahan
commented
6 years ago Apologies @ekristen, totally looked over the version mismatch between the command and my install. Here is the correct output:
crahan@sift:~$ sudo salt-call -l debug --local --file-root /var/cache/sift/cli/v2018.26.1/sift-saltstack-2018.26.1 state.sls sift.python-packages.machinae
[sudo] password for crahan:
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: sift
[DEBUG ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Please install 'virt-what' to improve results of the 'virtual' grain.
[DEBUG ] Determining pillar cache
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded state.sls
[DEBUG ] LazyLoaded saltutil.is_running
[DEBUG ] LazyLoaded grains.get
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG ] Updating roots fileserver cache
[DEBUG ] Determining pillar cache
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[INFO ] Loading fresh modules for state activity
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] In saltenv 'base', looking at rel_path 'sift/python-packages/machinae.sls' to resolve 'salt://sift/python-packages/machinae.sls'
[DEBUG ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/sift/python-packages/machinae.sls' to resolve 'salt://sift/python-packages/machinae.sls'
[DEBUG ] compile template: /var/cache/salt/minion/files/base/sift/python-packages/machinae.sls
[DEBUG ] Jinja search path: ['/var/cache/salt/minion/files/base']
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/sift/python-packages/machinae.sls' using 'jinja' renderer: 0.0214250087738
[DEBUG ] Rendered data from file: /var/cache/salt/minion/files/base/sift/python-packages/machinae.sls:
# WEBSITE: https://github.com/HurricaneLabs/machinae
# LICENSE: MIT
include:
- sift.packages.python-pip
- sift.packages.python3-pip
sift-python-packages-machinae:
pip.installed:
- name: machinae
- bin_env: /usr/bin/pip3
- require:
- sls: sift.packages.python-pip
- sls: sift.packages.python3-pip
[DEBUG ] LazyLoaded config.get
[DEBUG ] Results of YAML rendering:
OrderedDict([('include', ['sift.packages.python-pip', 'sift.packages.python3-pip']), ('sift-python-packages-machinae', OrderedDict([('pip.installed', [OrderedDict([('name', 'machinae')]), OrderedDict([('bin_env', '/usr/bin/pip3')]), OrderedDict([('require', [OrderedDict([('sls', 'sift.packages.python-pip')]), OrderedDict([('sls', 'sift.packages.python3-pip')])])])])]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/sift/python-packages/machinae.sls' using 'yaml' renderer: 0.00650310516357
[DEBUG ] In saltenv 'base', looking at rel_path 'sift/packages/python-pip.sls' to resolve 'salt://sift/packages/python-pip.sls'
[DEBUG ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/sift/packages/python-pip.sls' to resolve 'salt://sift/packages/python-pip.sls'
[DEBUG ] compile template: /var/cache/salt/minion/files/base/sift/packages/python-pip.sls
[DEBUG ] Jinja search path: ['/var/cache/salt/minion/files/base']
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/sift/packages/python-pip.sls' using 'jinja' renderer: 0.00884199142456
[DEBUG ] Rendered data from file: /var/cache/salt/minion/files/base/sift/packages/python-pip.sls:
include:
- .python
python-pip:
pkg.installed
[DEBUG ] Results of YAML rendering:
OrderedDict([('include', ['.python']), ('python-pip', 'pkg.installed')])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/sift/packages/python-pip.sls' using 'yaml' renderer: 0.000671863555908
[DEBUG ] In saltenv 'base', looking at rel_path 'sift/packages/python.sls' to resolve 'salt://sift/packages/python.sls'
[DEBUG ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/sift/packages/python.sls' to resolve 'salt://sift/packages/python.sls'
[DEBUG ] compile template: /var/cache/salt/minion/files/base/sift/packages/python.sls
[DEBUG ] Jinja search path: ['/var/cache/salt/minion/files/base']
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/sift/packages/python.sls' using 'jinja' renderer: 0.0109560489655
[DEBUG ] Rendered data from file: /var/cache/salt/minion/files/base/sift/packages/python.sls:
python:
pkg.installed
[DEBUG ] Results of YAML rendering:
OrderedDict([('python', 'pkg.installed')])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/sift/packages/python.sls' using 'yaml' renderer: 0.000755071640015
[DEBUG ] In saltenv 'base', looking at rel_path 'sift/packages/python3-pip.sls' to resolve 'salt://sift/packages/python3-pip.sls'
[DEBUG ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/sift/packages/python3-pip.sls' to resolve 'salt://sift/packages/python3-pip.sls'
[DEBUG ] compile template: /var/cache/salt/minion/files/base/sift/packages/python3-pip.sls
[DEBUG ] Jinja search path: ['/var/cache/salt/minion/files/base']
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/sift/packages/python3-pip.sls' using 'jinja' renderer: 0.0165350437164
[DEBUG ] Rendered data from file: /var/cache/salt/minion/files/base/sift/packages/python3-pip.sls:
include:
- .python3
python3-pip:
pkg.installed
[DEBUG ] Results of YAML rendering:
OrderedDict([('include', ['.python3']), ('python3-pip', 'pkg.installed')])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/sift/packages/python3-pip.sls' using 'yaml' renderer: 0.00129103660583
[DEBUG ] In saltenv 'base', looking at rel_path 'sift/packages/python3.sls' to resolve 'salt://sift/packages/python3.sls'
[DEBUG ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/sift/packages/python3.sls' to resolve 'salt://sift/packages/python3.sls'
[DEBUG ] compile template: /var/cache/salt/minion/files/base/sift/packages/python3.sls
[DEBUG ] Jinja search path: ['/var/cache/salt/minion/files/base']
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/sift/packages/python3.sls' using 'jinja' renderer: 0.00943398475647
[DEBUG ] Rendered data from file: /var/cache/salt/minion/files/base/sift/packages/python3.sls:
python3:
pkg.installed
[DEBUG ] Results of YAML rendering:
OrderedDict([('python3', 'pkg.installed')])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/sift/packages/python3.sls' using 'yaml' renderer: 0.000540971755981
[DEBUG ] LazyLoaded pkg.install
[DEBUG ] LazyLoaded pkg.installed
[DEBUG ] LazyLoaded systemd.booted
[DEBUG ] DSC: Only available on Windows systems
[DEBUG ] Module PSGet: Only available on Windows systems
[DEBUG ] Could not LazyLoad pkg.ex_mod_init: 'pkg.ex_mod_init' is not available.
[INFO ] Running state [python] at time 08:45:46.840815
[INFO ] Executing state pkg.installed for [python]
[INFO ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/home/crahan'
[DEBUG ] Could not LazyLoad pkg.normalize_name: 'pkg.normalize_name' is not available.
[INFO ] All specified packages are already installed
[INFO ] Completed state [python] at time 08:45:47.426516 duration_in_ms=585.701
[INFO ] Running state [python-pip] at time 08:45:47.426787
[INFO ] Executing state pkg.installed for [python-pip]
[DEBUG ] Could not LazyLoad pkg.normalize_name: 'pkg.normalize_name' is not available.
[INFO ] All specified packages are already installed
[INFO ] Completed state [python-pip] at time 08:45:47.432114 duration_in_ms=5.328
[INFO ] Running state [python3] at time 08:45:47.432287
[INFO ] Executing state pkg.installed for [python3]
[DEBUG ] Could not LazyLoad pkg.normalize_name: 'pkg.normalize_name' is not available.
[INFO ] All specified packages are already installed
[INFO ] Completed state [python3] at time 08:45:47.437172 duration_in_ms=4.885
[INFO ] Running state [python3-pip] at time 08:45:47.437325
[INFO ] Executing state pkg.installed for [python3-pip]
[DEBUG ] Could not LazyLoad pkg.normalize_name: 'pkg.normalize_name' is not available.
[INFO ] All specified packages are already installed
[INFO ] Completed state [python3-pip] at time 08:45:47.445068 duration_in_ms=7.743
[DEBUG ] LazyLoaded boto_datapipeline.create_pipeline
[DEBUG ] lzma module is not available
[DEBUG ] Registered VCS backend: git
[DEBUG ] Registered VCS backend: hg
[DEBUG ] Registered VCS backend: svn
[DEBUG ] Registered VCS backend: bzr
[DEBUG ] LazyLoaded pip.installed
[INFO ] Running state [machinae] at time 08:45:47.884706
[INFO ] Executing state pip.installed for [machinae]
[DEBUG ] pip: Found python binary: /usr/bin/python
[INFO ] Executing command ['/usr/bin/python', '-m', 'pip', '--version'] in directory '/home/crahan'
[DEBUG ] stdout: pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)
[DEBUG ] Installed pip version: 9.0.1
[DEBUG ] pip: Found python binary: /usr/bin/python
[INFO ] Executing command ['/usr/bin/python', '-m', 'pip', 'freeze', '--all'] in directory '/home/crahan'
[DEBUG ] stdout: adium-theme-ubuntu==0.3.4
alembic==0.9.5
amqp==2.2.1
analyzeMFT==2.0.19
aniso8601==1.2.1
AppCompatProcessor==0.8.0
apsw==3.8.11.1.post1
artifacts==20180628
asn1crypto==0.24.0
astroid==1.5.3
atomicwrites==1.1.5
attrs==18.1.0
backports.functools-lru-cache==1.4
backports.lzma==0.0.12
bcrypt==3.1.3
BeautifulSoup==3.2.1
bencode.py==2.0.0
billiard==3.5.0.3
biplist==1.0.3
bitstring==3.1.5
blinker==1.4
celery==4.1.0
certifi==2017.7.27.1
cffi==1.11.5
CFPropertyList==0.0.1
chardet==3.0.4
click==6.7
colorama==0.3.9
configparser==3.5.0
construct==2.5.3
coverage==4.4.1
cryptography==2.2.2
cybox==2.1.0.17
datasketch==1.2.5
decorator==4.0.6
dfdatetime==20180606
dfvfs==20180510
dfwinreg==20180329
distorm3==3.3.4
dnspython==1.12.0
docopt==0.6.2
dpapick==0.3
dpkt==1.9.1
DSV==1.4.1
dtfabric==20180604
ecdsa==0.13
efilter==1!1.5
elasticsearch==5.4.0
enum34==1.1.6
Flask==0.12.2
Flask-Bcrypt==0.7.1
Flask-Login==0.4.0
Flask-Migrate==2.1.1
Flask-RESTful==0.3.6
Flask-Script==2.0.5
Flask-SQLAlchemy==2.2
Flask-Testing==0.6.2
Flask-WTF==0.14.2
flowgrep==0.9
funcsigs==1.0.2
fuse-python==0.2.1
future==0.16.0
futures==3.0.5
geoip2==2.9.0
gunicorn==19.7.1
hachoir-core==1.3.3
hachoir-metadata==1.3.3
hachoir-parser==1.3.4
haystack==0.42
hexdump==3.3
idna==2.7
ioc-writer==0.3.3
ipaddress==1.0.22
ipython==2.4.1
isort==4.2.15
itsdangerous==0.24
Jinja2==2.9.6
kombu==4.1.0
lazy-object-proxy==1.3.1
libregf-python==20170130
lxml==4.2.3
M2Crypto==0.30.1
Mako==1.0.7
MarkupSafe==1.0
maxminddb==1.4.1
mccabe==0.6.1
mixbox==1.0.3
mock==2.0.0
more-itertools==4.2.0
msgpack-python==0.4.6
ndg-httpsclient==0.4.0
neo4jrestclient==2.1.1
nose==1.3.7
ntdsxtract==1.2b0
numpy==1.13.3
ordered-set==3.0.0
ordereddict==1.1
parameterized==0.6.1
paramiko==1.16.0
pbr==3.1.1
pefile==2017.11.5
pexpect==4.0.1
pip==9.0.1
plaso==20180630
pluggy==0.6.0
psutil==5.4.5
ptyprocess==0.5
py==1.5.4
pyasn1==0.4.3
pycoin==0.80
pycparser==2.18
pycrypto==2.6.1
pycurl==7.43.0
pycypher==0.5.9
PyJWT==1.6.4
pylint==1.7.2
PyMySQL==0.7.2
pynids==0.6.1
pyOpenSSL==18.0.0
pyparsing==2.2.0
pyserial==3.0.1
PySocks==1.6.8
pysqlite==2.7.0
pytest==3.6.3
pytest-cov==2.5.1
python-apt==1.1.0b1+ubuntu0.16.4.1
python-dateutil==2.6.1
python-editor==1.0.3
python-evtx==0.6.1
python-Levenshtein==0.12.0
python-magic==0.4.15
python-ptrace==0.9.3
python-registry==1.0.4
python-systemd==231
pytsk3==20180225
pytz==2017.2
PyYAML==4.1
pyzmq==17.0.0
redis==2.10.6
requests==2.18.4
salt==2017.7.7
setuptools==39.2.0
simplegeneric==0.8.1
simplejson==3.16.0
singledispatch==3.4.0.3
six==1.11.0
SQLAlchemy==1.1.13
stix==1.2.0.6
stix-validator==2.5.0
termcolor==1.1.0
timesketch==20180613
tornado==4.2.1
typing==3.6.4
unicodecsv==0.14.1
urllib3==1.22
usnparser==4.0.3
vine==1.1.4
virtualenv==15.0.1
virustotal-api==1.1.10
volatility==2.6
weakrefmethod==1.0.3
Werkzeug==0.12.2
wheel==0.31.1
windowsprefetch==3.0.5
wrapt==1.10.11
WTForms==2.1
wxPython==3.0.2.0
wxPython-common==3.0.2.0
xdot==0.6
xlrd==1.1.0
XlsxWriter==1.0.5
yara-python==3.7.0
[DEBUG ] stderr: The directory '/home/crahan/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
You are using pip version 9.0.1, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[DEBUG ] pip: Found python binary: /usr/bin/python
[DEBUG ] CLEANUP_REQUIREMENTS: []
[DEBUG ] TRY BLOCK: end of pip.install -- cmd: ['/usr/bin/python', '-m', 'pip', 'install', 'machinae'], cmd_kwargs: {'runas': None, 'use_vt': False, 'saltenv': 'base'}
[INFO ] Executing command ['/usr/bin/python', '-m', 'pip', 'install', 'machinae'] in directory '/home/crahan'
[ERROR ] Command '['/usr/bin/python', '-m', 'pip', 'install', 'machinae']' failed with return code: 1
[ERROR ] stdout: Collecting machinae
Downloading https://files.pythonhosted.org/packages/e1/32/fcf173886cb51cbba843a9eb4e6550d71c8f5f9aefec08dc08359232e022/machinae-1.4.0.tar.gz
Collecting dnspython3 (from machinae)
Downloading https://files.pythonhosted.org/packages/f0/bb/f41cbc8eaa807afb9d44418f092aa3e4acf0e4f42b439c49824348f1f45c/dnspython3-1.15.0.zip
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-fnipNA/dnspython3/setup.py", line 25
"""+"="*78, file=sys.stdout)
^
SyntaxError: invalid syntax
----------------------------------------
[ERROR ] stderr: The directory '/home/crahan/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/crahan/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-fnipNA/dnspython3/
You are using pip version 9.0.1, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[ERROR ] retcode: 1
[ERROR ] Failed to install packages: machinae. Error: Collecting machinae
Downloading https://files.pythonhosted.org/packages/e1/32/fcf173886cb51cbba843a9eb4e6550d71c8f5f9aefec08dc08359232e022/machinae-1.4.0.tar.gz
Collecting dnspython3 (from machinae)
Downloading https://files.pythonhosted.org/packages/f0/bb/f41cbc8eaa807afb9d44418f092aa3e4acf0e4f42b439c49824348f1f45c/dnspython3-1.15.0.zip
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-fnipNA/dnspython3/setup.py", line 25
"""+"="*78, file=sys.stdout)
^
SyntaxError: invalid syntax
---------------------------------------- The directory '/home/crahan/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/crahan/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-fnipNA/dnspython3/
You are using pip version 9.0.1, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[INFO ] Completed state [machinae] at time 08:45:50.431319 duration_in_ms=2546.613
[DEBUG ] File /var/cache/salt/minion/accumulator/140229255975248 does not exist, no need to cleanup.
[DEBUG ] LazyLoaded highstate.output
local:
----------
ID: python
Function: pkg.installed
Result: True
Comment: All specified packages are already installed
Started: 08:45:46.840815
Duration: 585.701 ms
Changes:
----------
ID: python-pip
Function: pkg.installed
Result: True
Comment: All specified packages are already installed
Started: 08:45:47.426786
Duration: 5.328 ms
Changes:
----------
ID: python3
Function: pkg.installed
Result: True
Comment: All specified packages are already installed
Started: 08:45:47.432287
Duration: 4.885 ms
Changes:
----------
ID: python3-pip
Function: pkg.installed
Result: True
Comment: All specified packages are already installed
Started: 08:45:47.437325
Duration: 7.743 ms
Changes:
----------
ID: sift-python-packages-machinae
Function: pip.installed
Name: machinae
Result: False
Comment: Failed to install packages: machinae. Error: Collecting machinae
Downloading https://files.pythonhosted.org/packages/e1/32/fcf173886cb51cbba843a9eb4e6550d71c8f5f9aefec08dc08359232e022/machinae-1.4.0.tar.gz
Collecting dnspython3 (from machinae)
Downloading https://files.pythonhosted.org/packages/f0/bb/f41cbc8eaa807afb9d44418f092aa3e4acf0e4f42b439c49824348f1f45c/dnspython3-1.15.0.zip
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-fnipNA/dnspython3/setup.py", line 25
"""+"="*78, file=sys.stdout)
^
SyntaxError: invalid syntax
---------------------------------------- The directory '/home/crahan/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/crahan/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-fnipNA/dnspython3/
You are using pip version 9.0.1, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Started: 08:45:47.884706
Duration: 2546.613 ms
Changes:
Summary for local
------------
Succeeded: 4
Failed: 1
------------
Total states run: 5
Total run time: 3.150 sOnce again, thank you to everyone that helped troubleshoot this issue. It's awesome to have members of the community helping out.
I have released a release candidate (RC) of the SIFT CLI https://github.com/sans-dfir/sift-cli/releases/tag/v1.7.1-rc2 as well as a new version of the salt states that decouple the CLI from it. This allows the CLI to be updated independently of the salt states from running.
If you go download the 1.7.2-rc1 and then proceed to install or upgrade using the --pre-release flag you will get the new install and this should also result in a successful machinae install.
The 1.7.2-rc1 release pins SaltStack to 2017.7.5 until the fix that was recently committed makes it into 2017.7.8.
Please report back success or failure. Thank you!
ekristen
commented
6 years ago Please use 1.7.1-rc2 instead.
crahan
commented
6 years ago Looks like so far, so good. Just passed the Machinae step of the install.
Update: install successful!!!
darendarrow
commented
6 years ago Using rc2 I'm getting a new issue. Apt shows issues with open-iscsi and docker-engine.
Get:1 http://ppa.launchpad.net/gift/stable/ubuntu xenial/main amd64 python-certifi all 2018.4.16-1ppa1~xenial [138 kB] Unpacking python-requests (2.19.1-1ppa1~xenial) over (2.9.1-3) ... Preparing to unpack .../python-urllib3_1.23-1ppa1~xenial_all.deb ... Unpacking python-urllib3 (1.23-1ppa1~xenial) over (1.13.1-2ubuntu0.16.04.1) ... Setting up open-iscsi (2.0.873+git0.3b4b4500-14ubuntu3.4) ... Job for iscsid.service failed because a configured resource limit was exceeded. See "systemctl status iscsid.service" and "journalctl -xe" for details. invoke-rc.d: initscript iscsid, action "start" failed. ● iscsid.service - iSCSI initiator daemon (iscsid) Loaded: loaded (/lib/systemd/system/iscsid.service; enabled; vendor preset: enabled) Active: failed (Result: resources) since Mon 2018-07-09 22:05:27 UTC; 14ms ago Docs: man:iscsid(8) Process: 614312 ExecStart=/sbin/iscsid (code=exited, status=0/SUCCESS) Process: 614305 ExecStartPre=/lib/open-iscsi/startup-checks.sh (code=exited, status=0/SUCCESS) dpkg: error processing package open-iscsi (--configure): subprocess installed post-installation script returned error exit status 1 Setting up docker-engine (17.05.0~ce-0~ubuntu-xenial) ... Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details. invoke-rc.d: initscript docker, action "start" failed.
● docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Mon 2018-07-09 22:05:29 UTC; 13ms ago Docs: https://docs.docker.com Process: 614425 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=1/FAILURE) Main PID: 614425 (code=exited, status=1/FAILURE) dpkg: error processing package docker-engine (--configure): subprocess installed post-installation script returned error exit status 1 Setting up python-certifi (2018.4.16-1ppa1~xenial) ... Setting up python-urllib3 (1.23-1ppa1~xenial) ... Setting up python-requests (2.19.1-1ppa1~xenial) ... Processing triggers for initramfs-tools (0.122ubuntu8.11) ... Errors were encountered while processing: open-iscsi docker-engine E: Sub-process /usr/bin/dpkg returned an error code (1)
Completed: dconf write /org/compiz/profiles/unity/plugins/unityshell/icon-size 32 (Took: 202.982 ms) Completed: dconf write /org/compiz/profiles/unity/plugins/unityshell/launcher-minimize-window true (Took: 193.847 ms) Completed: Etc/UTC (Took: 92.49 ms) Completed: /cases (Took: 2.136 ms) Completed: /mnt/usb (Took: 1.155 ms) Completed: /mnt/vss (Took: 1.079 ms) Completed: /mnt/shadow_mount (Took: 1.094 ms) Completed: /mnt/windows_mount (Took: 1.092 ms) Completed: /mnt/ewf_mount (Took: 1.155 ms) Completed: /mnt/e01 (Took: 1.074 ms) Completed: /mnt/aff (Took: 1.065 ms) Completed: /mnt/ewf (Took: 1.025 ms) Completed: /mnt/bde (Took: 1.019 ms) Completed: /mnt/iscsi (Took: 1.1 ms) Completed: /mnt/windows_mount1 (Took: 1.028 ms) Completed: /mnt/windows_mount2 (Took: 1.017 ms) Completed: /mnt/windows_mount3 (Took: 1.024 ms) Completed: /mnt/windows_mount4 (Took: 1.015 ms) Completed: /mnt/windows_mount5 (Took: 1.016 ms) Completed: /mnt/shadow_mount/vss1 (Took: 1.057 ms) Completed: /mnt/shadow_mount/vss2 (Took: 1.056 ms) Completed: /mnt/shadow_mount/vss3 (Took: 1.049 ms) Completed: /mnt/shadow_mount/vss4 (Took: 1.05 ms) Completed: /mnt/shadow_mount/vss5 (Took: 1.091 ms) Completed: /mnt/shadow_mount/vss6 (Took: 1.042 ms) Completed: /mnt/shadow_mount/vss7 (Took: 1.057 ms) Completed: /mnt/shadow_mount/vss8 (Took: 1.061 ms) Completed: /mnt/shadow_mount/vss9 (Took: 1.12 ms) Completed: /mnt/shadow_mount/vss10 (Took: 1.058 ms) Completed: /mnt/shadow_mount/vss11 (Took: 1.039 ms) Completed: /mnt/shadow_mount/vss12 (Took: 1.06 ms) Completed: /mnt/shadow_mount/vss13 (Took: 1.07 ms) Completed: /mnt/shadow_mount/vss14 (Took: 1.029 ms) Completed: /mnt/shadow_mount/vss15 (Took: 1.034 ms) Completed: /mnt/shadow_mount/vss16 (Took: 1.058 ms) Completed: /mnt/shadow_mount/vss17 (Took: 1.027 ms) Completed: /mnt/shadow_mount/vss18 (Took: 1.041 ms) Completed: /mnt/shadow_mount/vss19 (Took: 1.094 ms) Completed: /mnt/shadow_mount/vss20 (Took: 1.047 ms) Completed: /mnt/shadow_mount/vss21 (Took: 1.07 ms) Completed: /mnt/shadow_mount/vss22 (Took: 1.029 ms) Completed: /mnt/shadow_mount/vss23 (Took: 1.069 ms) Completed: /mnt/shadow_mount/vss24 (Took: 1.049 ms) Completed: /mnt/shadow_mount/vss25 (Took: 1.021 ms) Completed: /mnt/shadow_mount/vss26 (Took: 1.02 ms) Completed: /mnt/shadow_mount/vss27 (Took: 1.019 ms) Completed: /mnt/shadow_mount/vss28 (Took: 1.038 ms) Completed: /mnt/shadow_mount/vss29 (Took: 1.03 ms) Completed: /mnt/shadow_mount/vss30 (Took: 1.011 ms) Completed: salt-minion (Took: 1892.589 ms) Completed: /etc/foremost.conf (Took: 7.225 ms) Completed: /usr/local/etc/foremost.conf (Took: 2.591 ms) Completed: sift-config-tools (Took: 0.404 ms)
Incomplete due to Failures -- Success: 306, Failure: 232
List of Failures (first 10 only)
NOTE: First failure is generally the root cause. IMPORTANT: If opening a ticket, please include this information. - ID: docker-engine SLS: sift.packages.docker-engine Run#: 44 Comment: Problem encountered installing package(s). Additional info follows:errors:
- Running scope as unit run-r343708878ecc434b9aca626433aaff2b.scope. E: Sub-process /usr/bin/dpkg returned an error code (1)
- ID: driftnet SLS: sift.packages.driftnet Run#: 45 Comment: Problem encountered installing package(s). Additional info follows:
errors:
- Running scope as unit run-r8692fdd06af044d68fbd14f48d33160c.scope. E: Sub-process /usr/bin/dpkg returned an error code (1)
- ID: dsniff SLS: sift.packages.dsniff Run#: 46 Comment: Problem encountered installing package(s). Additional info follows:
errors:
- Running scope as unit run-r506e9a590a094048915a98c1fa04240d.scope. E: Sub-process /usr/bin/dpkg returned an error code (1)
- ID: dumbpig SLS: sift.packages.dumbpig Run#: 47 Comment: Problem encountered installing package(s). Additional info follows:
errors:
- Running scope as unit run-rcdeee34e43de47fb9a82391e71d0add5.scope. E: Sub-process /usr/bin/dpkg returned an error code (1)
- ID: e2fslibs-dev SLS: sift.packages.e2fslibs-dev Run#: 48 Comment: Problem encountered installing package(s). Additional info follows:
errors:
- Running scope as unit run-rdec9f5630a60486c9dae4f75b9a8e2ff.scope. E: Sub-process /usr/bin/dpkg returned an error code (1)
- ID: ent SLS: sift.packages.ent Run#: 49 Comment: Problem encountered installing package(s). Additional info follows:
errors:
- Running scope as unit run-r7c614c9547d84f4ebcc75d2038255003.scope. E: Sub-process /usr/bin/dpkg returned an error code (1)
- ID: epic5 SLS: sift.packages.epic5 Run#: 50 Comment: Problem encountered installing package(s). Additional info follows:
errors:
- Running scope as unit run-ra0c61eea55ed49b9affdbbefe40afe59.scope. E: Sub-process /usr/bin/dpkg returned an error code (1)
- ID: etherape SLS: sift.packages.etherape Run#: 51 Comment: Problem encountered installing package(s). Additional info follows:
errors:
- Running scope as unit run-r17153ffeba4e4976b750b9533ddd10a4.scope. E: Sub-process /usr/bin/dpkg returned an error code (1)
- ID: ettercap-graphical SLS: sift.packages.ettercap-graphical Run#: 52 Comment: Problem encountered installing package(s). Additional info follows:
errors:
- Running scope as unit run-re1499f84417d4b7dacd1368a04a8791a.scope. E: Sub-process /usr/bin/dpkg returned an error code (1)
- ID: exfat-fuse SLS: sift.packages.exfat-fuse Run#: 53 Comment: Problem encountered installing package(s). Additional info follows:
errors:
- Running scope as unit run-rbb1300cd3c0c42ef8a9ee2d650c0c94f.scope. E: Sub-process /usr/bin/dpkg returned an error code (1)
ekristen
commented
6 years ago These seem to be problems with the host which is unfortunately outside the control of what I can control with the installation system.
I would check on the health of apt.
darendarrow
commented
6 years ago It's a new image. I've destroyed and rebuilt it a few times before posting the messages.
darendarrow
commented
6 years ago I'm attempting again in a different infrastructure with an ISO image source for Ubuntu. Should note the URL provided for RC2 is wrong in the release notes.
The version was moved to 1.7.2-rc1 instead of 1.7.1-rc2.
ekristen
commented
6 years ago @darendarrow thanks for pointing the flaw out in the release notes, I've fixed it.
I appreciate that it's a new image, but that doesn't negate the fact that there could be still issues with APT or the underlying services. I've found that ubuntu can be very aggressive about updating apt and can cause conflicts, especially around locks.
ekristen
commented
6 years ago Running scope as unit run-re1499f84417d4b7dacd1368a04a8791a.scope. E: Sub-process /usr/bin/dpkg returned an error code (1)
This means there's something wrong with dpkg or apt.
darendarrow
commented
6 years ago It completed fine with the ISO based instance. odd.
ekristen
commented
6 years ago I've found the default settings for apt and security updates can cause lots of problems. Because during the install we do not hold a lock on apt, it allows for other tools to grab the lock and that usually ends poorly for the install unfortunately ;(
howlingshiba
commented
6 years ago I can also confirm that version v1.7.1-rc2 installs without any issue. Thanks for the update! :-)
However, I did run into issue running log2timeline. It gave the following error:
AttributeError: 'module' object has no attribute 'SSL_ST_INIT'As previously suggested regarding SSL_ST_INIT error, this issue was resolved with updating pyOpenSSL via sudo pip install -U pyOpenSSL. This upgraded from pyOpenSSL 15.1 to 18.0
ekristen
commented
6 years ago @howlingshiba Can you please open a new issue for the pyOpenSSL and it's related problems. Thank you!
I'm going to close this issue for now.