search

certtools / intelmq

IntelMQ is a solution for IT security teams for collecting and processing security feeds using a message queuing protocol.
https://docs.intelmq.org/latest/
GNU Affero General Public License v3.0
975 stars 296 forks source link

Installation: Pip install intelmq #819

Closed SYNchroACK closed 7 years ago

SYNchroACK commented 7 years ago

On documentation there is a mention of 'pip install intelmq' vs 'pip install .' which I would like to suggest to replace with pip3. I test the "intelmq" way -> 'pip3 install intelmq' but the /opt/intelmq directory was not created.

sebix commented 7 years ago

On documentation there is a mention of 'pip install intelmq' vs 'pip install .' which I would like to suggest to replace with pip3.

Depends on the OS if it's called pip, pip3, pip-3.5 or similar. Which OS do you use?

On pip install intelmq vs pip install .: They have a different meaning, the first installs from pypi, the second from the local dir, useful for development:

I test the "intelmq" way -> 'pip3 install intelmq' but the /opt/intelmq directory was not created.

What output has been shown during installation? Which OS?

geometric-networks commented 7 years ago

I know this might be old, but why do we need to do 'pip install intelmq' if nothing is created in the /opt folder where IntelMQ Manager searches?

sebix commented 7 years ago

I know this might be old, but why do we need to do 'pip install intelmq' if nothing is created in the /opt folder where IntelMQ Manager searches?

I don't understand your question. /opt/intelmq should be created when using the pypi packages. For deb/rpm packages we use the correct paths only (/usr, /var, /etc, ...).

See https://github.com/certtools/intelmq/blob/master/setup.py#L18

geometric-networks commented 7 years ago

Sebix,

I confirmed in a separate Ubuntu 16.04 build that using the pypi package doesn't create an 'intelmq' folder under the /opt directory automatically, but rather under /usr/local/lib/python2.7/dist-packages/opt/intelmq. You also run into an issue where only the intelmq /etc and /var folders are under the .../opt/intelmq directory, but the /bin /bots /lib /tests folders from the github repo aren't present. To my understanding, the /bots folder is critical to have under '/opt/intelmq' as that's where you would be setting your API keys for your bot tie-ins.

sebix commented 7 years ago

Hi,

I confirmed in a separate build that using the pypi package doesn't create an 'intelmq' folder under the /opt directory automatically, but rather under /usr/local/lib/python2.7/dist-packages/opt/intelmq. We don't support Python < 3.3. but the /bin /bots /lib /tests folders aren't present. No, they should not be there. To my understanding, the /bots folder is critical to have under '/opt/intelmq' as that's where you would be setting your API keys. No, your settings reside in /opt/intelmq/etc/runtime.conf. Have a look at our user guide: https://github.com/certtools/intelmq/blob/master/docs/User-Guide.md#configuration

geometric-networks commented 7 years ago

Ah, apologies. You are correct on the runtime.conf file. Yea, I dunno about that 2.7 part. It's what I got from Ubuntu's package repo after requesting python3 installed.

dmth commented 7 years ago

On systems with python2.7 and python3.x installed, its very likely you need to use pip3, because pipwill default to python2.7

SYNchroACK commented 7 years ago

System Used

root@server:~# uname -a
Linux server 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

root@server:~# cat /etc/*release* | grep _NAME
PRETTY_NAME="Ubuntu 16.04.1 LTS"

Installation via 'pip3 install intelmq'

Installation and Verification

root@server:~# history
  ...
   14  apt-get update
   15  apt-get upgrade -y
   16  reboot
  ...
   20  apt-get install git build-essential libcurl4-gnutls-dev libffi-dev python3 python3-pip redis-server python3-dev
   21  pip3 install intelmq

root@server:~# find / -iname "*intelmq*"
/usr/local/lib/python3.5/dist-packages/intelmq-1.0.0.dev5.dist-info
/usr/local/lib/python3.5/dist-packages/opt/intelmq
/usr/local/lib/python3.5/dist-packages/intelmq
/usr/local/lib/python3.5/dist-packages/intelmq/tests/bin/test_intelmqctl.py
/usr/local/lib/python3.5/dist-packages/intelmq/tests/bin/__pycache__/test_intelmqctl.cpython-35.pyc
/usr/local/lib/python3.5/dist-packages/intelmq/tests/bin/__pycache__/test_intelmqdump.cpython-35.pyc
/usr/local/lib/python3.5/dist-packages/intelmq/tests/bin/test_intelmqdump.py
/usr/local/lib/python3.5/dist-packages/intelmq/tests/bots/outputs/intelmqmailer
/usr/local/lib/python3.5/dist-packages/intelmq/bin/intelmqdump.py
/usr/local/lib/python3.5/dist-packages/intelmq/bin/intelmq_psql_initdb.py
/usr/local/lib/python3.5/dist-packages/intelmq/bin/__pycache__/intelmq_gen_harm_docs.cpython-35.pyc
/usr/local/lib/python3.5/dist-packages/intelmq/bin/__pycache__/intelmqdump.cpython-35.pyc
/usr/local/lib/python3.5/dist-packages/intelmq/bin/__pycache__/intelmq_psql_initdb.cpython-35.pyc
/usr/local/lib/python3.5/dist-packages/intelmq/bin/__pycache__/intelmqctl.cpython-35.pyc
/usr/local/lib/python3.5/dist-packages/intelmq/bin/intelmqctl.py
/usr/local/lib/python3.5/dist-packages/intelmq/bin/intelmq_gen_harm_docs.py
/usr/local/lib/python3.5/dist-packages/intelmq/bots/outputs/intelmqmailer
/usr/local/bin/intelmqdump
/usr/local/bin/intelmqctl
/usr/local/bin/intelmq_psql_initdb

root@server:~# ls -la /usr/local/bin/
total 44
drwxr-xr-x  2 root root 4096 Dec 28 22:37 .
drwxr-xr-x 10 root root 4096 Dec 20 03:19 ..
-rwxr-xr-x  1 root root  224 Dec 28 22:37 intelmqctl
-rwxr-xr-x  1 root root  225 Dec 28 22:37 intelmqdump
-rwxr-xr-x  1 root root  233 Dec 28 22:37 intelmq_psql_initdb
-rwxr-xr-x  1 root root  205 Dec 28 22:37 pip
-rwxr-xr-x  1 root root  205 Dec 28 22:37 pip3
-rwxr-xr-x  1 root root  205 Dec 28 22:37 pip3.5
-rwxr-xr-x  1 root root 1098 Dec 28 22:37 update-asn-data
-rwxr-xr-x  1 root root  864 Dec 28 22:37 update-geoip-data
-rwxr-xr-x  1 root root  838 Dec 28 22:37 update-tor-nodes

root@server:~# ls -la /usr/local/lib/python3.5/dist-packages/
total 80
drwxrwsr-x 19 root staff 4096 Dec 28 22:37 .
drwxrwsr-x  3 root staff 4096 Dec 20 03:19 ..
drwxr-sr-x  5 root staff 4096 Dec 28 22:37 dateutil
drwxr-sr-x  4 root staff 4096 Dec 28 22:37 dns
drwxr-sr-x  2 root staff 4096 Dec 28 22:37 dnspython-1.15.0.dist-info
drwxr-sr-x  8 root staff 4096 Dec 28 22:37 intelmq
drwxr-sr-x  2 root staff 4096 Dec 28 22:37 intelmq-1.0.0.dev5.dist-info
drwxr-sr-x  3 root staff 4096 Dec 28 22:37 opt
drwxr-sr-x 11 root staff 4096 Dec 28 22:37 pip
drwxr-sr-x  2 root staff 4096 Dec 28 22:37 pip-9.0.1.dist-info
drwxr-sr-x  4 root staff 4096 Dec 28 22:37 psutil
drwxr-sr-x  2 root staff 4096 Dec 28 22:37 psutil-5.0.1.dist-info
drwxr-sr-x  2 root staff 4096 Dec 28 22:37 __pycache__
drwxr-sr-x  2 root staff 4096 Dec 28 22:37 python_dateutil-2.6.0.dist-info
drwxr-sr-x  2 root staff 4096 Dec 28 22:37 python_termstyle-0.1.10.dist-info
drwxr-sr-x  4 root staff 4096 Dec 28 22:37 pytz
drwxr-sr-x  2 root staff 4096 Dec 28 22:37 pytz-2016.10.dist-info
drwxr-sr-x  3 root staff 4096 Dec 28 22:37 redis
drwxr-sr-x  2 root staff 4096 Dec 28 22:37 redis-2.10.5.dist-info
-rw-r--r--  1 root staff 3195 Dec 28 22:37 termstyle.py

root@server:~# ls -la /usr/local/lib/python3.5/dist-packages/intelmq/
total 44
drwxr-sr-x  8 root staff 4096 Dec 28 22:37 .
drwxrwsr-x 19 root staff 4096 Dec 28 22:37 ..
drwxr-sr-x  3 root staff 4096 Dec 28 22:37 bin
drwxr-sr-x  7 root staff 4096 Dec 28 22:37 bots
drwxr-sr-x  2 root staff 4096 Dec 28 22:37 etc
-rw-r--r--  1 root staff  634 Dec 28 22:37 __init__.py
drwxr-sr-x  3 root staff 4096 Dec 28 22:37 lib
drwxr-sr-x  2 root staff 4096 Dec 28 22:37 __pycache__
drwxr-sr-x  6 root staff 4096 Dec 28 22:37 tests
-rw-r--r--  1 root staff 1146 Dec 28 22:37 untitled2.py
-rw-r--r--  1 root staff   84 Dec 28 22:37 version.py

root@server:~# ls -la /usr/local/lib/python3.5/dist-packages/opt/intelmq/
total 16
drwxr-sr-x 4 root staff 4096 Dec 28 22:37 .
drwxr-sr-x 3 root staff 4096 Dec 28 22:37 ..
drwxr-sr-x 3 root staff 4096 Dec 28 22:37 etc
drwxr-sr-x 3 root staff 4096 Dec 28 22:37 var

root@server:~# ls -la /opt/
total 8
drwxr-xr-x  2 root root 4096 Jul 19 21:43 .
drwxr-xr-x 23 root root 4096 Dec 20 03:54 ..

Analysis

  1. '/usr/local/bin/' does not contain the "link" to bots (e.g. /usr/local/bin/intelmq.bots.parsers.phishtank.parser) which is correct because is not a development installation.
  2. '/opt/intelmq' does not exist. Instead, the folder was wrongly (I suppose) created in '/usr/local/lib/python3.5/dist-packages/opt/intelmq'

Proposal

Installation via 'pip3 install .'

Installation and Verification

root@server:~# history
  ...
   14  apt-get update
   15  apt-get upgrade -y
   16  reboot
  ...
   20  apt-get install git build-essential libcurl4-gnutls-dev libffi-dev python3 python3-pip redis-server python3-dev
   21  pip3 install -r REQUIREMENTS
   22  pip3 install .
   23  useradd -d /opt/intelmq -U -s /bin/bash intelmq
   24  chmod -R 0770 /opt/intelmq
   25  chown -R intelmq.intelmq /opt/intelmq
   26  rm -rf /tmp/intelmq

root@server:~# find / -iname "*intelmq*"
/opt/intelmq
/usr/local/lib/python3.5/dist-packages/intelmq
/usr/local/lib/python3.5/dist-packages/intelmq/tests/bin/test_intelmqctl.py
/usr/local/lib/python3.5/dist-packages/intelmq/tests/bin/__pycache__/test_intelmqctl.cpython-35.pyc
/usr/local/lib/python3.5/dist-packages/intelmq/tests/bin/__pycache__/test_intelmqdump.cpython-35.pyc
/usr/local/lib/python3.5/dist-packages/intelmq/tests/bin/test_intelmqdump.py
/usr/local/lib/python3.5/dist-packages/intelmq/bin/intelmqdump.py
/usr/local/lib/python3.5/dist-packages/intelmq/bin/intelmq_psql_initdb.py
/usr/local/lib/python3.5/dist-packages/intelmq/bin/__pycache__/intelmq_gen_harm_docs.cpython-35.pyc
/usr/local/lib/python3.5/dist-packages/intelmq/bin/__pycache__/intelmqdump.cpython-35.pyc
/usr/local/lib/python3.5/dist-packages/intelmq/bin/__pycache__/intelmq_psql_initdb.cpython-35.pyc
/usr/local/lib/python3.5/dist-packages/intelmq/bin/__pycache__/intelmqctl.cpython-35.pyc
/usr/local/lib/python3.5/dist-packages/intelmq/bin/intelmqctl.py
/usr/local/lib/python3.5/dist-packages/intelmq/bin/intelmq_gen_harm_docs.py
/usr/local/lib/python3.5/dist-packages/intelmq-1.0.0.dev5.egg-info
/usr/local/bin/intelmq.bots.parsers.malwaredomainlist.parser
/usr/local/bin/intelmq.bots.experts.ripencc_abuse_contact.expert
/usr/local/bin/intelmq.bots.experts.certat_contact.expert
/usr/local/bin/intelmq.bots.parsers.misp.parser
/usr/local/bin/intelmq.bots.outputs.udp.output
/usr/local/bin/intelmq.bots.parsers.dshield.parser_domain
/usr/local/bin/intelmq.bots.collectors.misp.collector
/usr/local/bin/intelmq.bots.parsers.abusech.parser_ransomware
/usr/local/bin/intelmq.bots.parsers.cymru_full_bogons.parser
/usr/local/bin/intelmq.bots.outputs.amqptopic.output
/usr/local/bin/intelmq.bots.parsers.turris.parser
/usr/local/bin/intelmq.bots.parsers.openbl.parser
/usr/local/bin/intelmq.bots.collectors.rt.collector_rt
/usr/local/bin/intelmq.bots.experts.gethostbyname.expert
/usr/local/bin/intelmq.bots.collectors.xmpp.collector
/usr/local/bin/intelmq.bots.parsers.malwaredomains.parser
/usr/local/bin/intelmq.bots.collectors.n6.collector_stomp
/usr/local/bin/intelmq.bots.parsers.dshield.parser_asn
/usr/local/bin/intelmqdump
/usr/local/bin/intelmq.bots.parsers.json.parser
/usr/local/bin/intelmq.bots.parsers.dyn.parser
/usr/local/bin/intelmq.bots.experts.filter.expert
/usr/local/bin/intelmq.bots.parsers.blueliv.parser_crimeserver
/usr/local/bin/intelmq.bots.parsers.bitsight.parser
/usr/local/bin/intelmq.bots.parsers.bambenek.parser
/usr/local/bin/intelmq.bots.parsers.hphosts.parser
/usr/local/bin/intelmq.bots.parsers.generic.parser_csv
/usr/local/bin/intelmq.bots.parsers.phishtank.parser
/usr/local/bin/intelmq.bots.outputs.restapi.output
/usr/local/bin/intelmq.bots.experts.modify.expert
/usr/local/bin/intelmq.bots.outputs.redis.output
/usr/local/bin/intelmq.bots.parsers.spamhaus.parser_drop
/usr/local/bin/intelmqctl
/usr/local/bin/intelmq.bots.parsers.urlvir.parser
/usr/local/bin/intelmq.bots.collectors.bitsight.collector
/usr/local/bin/intelmq.bots.collectors.http.collector_http
/usr/local/bin/intelmq.bots.parsers.alienvault.parser
/usr/local/bin/intelmq.bots.parsers.abusech.parser_ip
/usr/local/bin/intelmq.bots.experts.cymru_whois.expert
/usr/local/bin/intelmq.bots.experts.tor_nodes.expert
/usr/local/bin/intelmq.bots.collectors.mail.collector_mail_attach
/usr/local/bin/intelmq.bots.outputs.postgresql.output
/usr/local/bin/intelmq.bots.parsers.danger_rulez.parser
/usr/local/bin/intelmq.bots.parsers.cleanmx.parser
/usr/local/bin/intelmq.bots.outputs.file.output
/usr/local/bin/intelmq.bots.outputs.tcp.output
/usr/local/bin/intelmq.bots.collectors.alienvault_otx.collector
/usr/local/bin/intelmq.bots.parsers.abusech.parser_domain
/usr/local/bin/intelmq.bots.parsers.spamhaus.parser_cert
/usr/local/bin/intelmq.bots.parsers.taichung.parser
/usr/local/bin/intelmq.bots.parsers.nothink.parser
/usr/local/bin/intelmq.bots.collectors.mail.collector_mail_url
/usr/local/bin/intelmq.bots.parsers.malc0de.parser
/usr/local/bin/intelmq.bots.collectors.file.collector_file
/usr/local/bin/intelmq.bots.parsers.fraunhofer.parser_dga
/usr/local/bin/intelmq.bots.experts.url2fqdn.expert
/usr/local/bin/intelmq.bots.parsers.malwarepatrol.parser_dansguardian
/usr/local/bin/intelmq.bots.parsers.shadowserver.parser
/usr/local/bin/intelmq.bots.experts.abusix.expert
/usr/local/bin/intelmq.bots.parsers.ci_army.parser
/usr/local/bin/intelmq.bots.experts.generic_db_lookup.expert
/usr/local/bin/intelmq.bots.experts.rfc1918.expert
/usr/local/bin/intelmq.bots.parsers.vxvault.parser
/usr/local/bin/intelmq.bots.parsers.n6.parser_n6stomp
/usr/local/bin/intelmq.bots.experts.deduplicator.expert
/usr/local/bin/intelmq.bots.parsers.autoshun.parser
/usr/local/bin/intelmq.bots.experts.taxonomy.expert
/usr/local/bin/intelmq.bots.parsers.alienvault.parser_otx
/usr/local/bin/intelmq.bots.collectors.blueliv.collector_crimeserver
/usr/local/bin/intelmq.bots.parsers.bitcash.parser
/usr/local/bin/intelmq.bots.experts.maxmind_geoip.expert
/usr/local/bin/intelmq.bots.outputs.mongodb.output
/usr/local/bin/intelmq.bots.parsers.dshield.parser_block
/usr/local/bin/intelmq.bots.parsers.dataplane.parser
/usr/local/bin/intelmq_psql_initdb
/usr/local/bin/intelmq.bots.parsers.blocklistde.parser
/usr/local/bin/intelmq.bots.experts.reverse_dns.expert
/usr/local/bin/intelmq.bots.parsers.openphish.parser
/usr/local/bin/intelmq.bots.parsers.netlab_360.parser
/usr/local/bin/intelmq.bots.outputs.xmpp.output
/usr/local/bin/intelmq.bots.experts.asn_lookup.expert

root@server:~# ls -la /usr/local/bin/
total 344
drwxr-xr-x  2 root root 4096 Dec 28 22:47 .
drwxr-xr-x 10 root root 4096 Dec 20 03:19 ..
-rwxr-xr-x  1 root root  399 Dec 28 22:47 intelmq.bots.collectors.alienvault_otx.collector
-rwxr-xr-x  1 root root  387 Dec 28 22:47 intelmq.bots.collectors.bitsight.collector
-rwxr-xr-x  1 root root  409 Dec 28 22:47 intelmq.bots.collectors.blueliv.collector_crimeserver
-rwxr-xr-x  1 root root  389 Dec 28 22:47 intelmq.bots.collectors.file.collector_file
-rwxr-xr-x  1 root root  389 Dec 28 22:47 intelmq.bots.collectors.http.collector_http
-rwxr-xr-x  1 root root  403 Dec 28 22:47 intelmq.bots.collectors.mail.collector_mail_attach
-rwxr-xr-x  1 root root  397 Dec 28 22:47 intelmq.bots.collectors.mail.collector_mail_url
-rwxr-xr-x  1 root root  379 Dec 28 22:47 intelmq.bots.collectors.misp.collector
-rwxr-xr-x  1 root root  387 Dec 28 22:47 intelmq.bots.collectors.n6.collector_stomp
-rwxr-xr-x  1 root root  381 Dec 28 22:47 intelmq.bots.collectors.rt.collector_rt
-rwxr-xr-x  1 root root  379 Dec 28 22:47 intelmq.bots.collectors.xmpp.collector
-rwxr-xr-x  1 root root  371 Dec 28 22:47 intelmq.bots.experts.abusix.expert
-rwxr-xr-x  1 root root  379 Dec 28 22:47 intelmq.bots.experts.asn_lookup.expert
-rwxr-xr-x  1 root root  387 Dec 28 22:47 intelmq.bots.experts.certat_contact.expert
-rwxr-xr-x  1 root root  381 Dec 28 22:47 intelmq.bots.experts.cymru_whois.expert
-rwxr-xr-x  1 root root  383 Dec 28 22:47 intelmq.bots.experts.deduplicator.expert
-rwxr-xr-x  1 root root  371 Dec 28 22:47 intelmq.bots.experts.filter.expert
-rwxr-xr-x  1 root root  393 Dec 28 22:47 intelmq.bots.experts.generic_db_lookup.expert
-rwxr-xr-x  1 root root  385 Dec 28 22:47 intelmq.bots.experts.gethostbyname.expert
-rwxr-xr-x  1 root root  385 Dec 28 22:47 intelmq.bots.experts.maxmind_geoip.expert
-rwxr-xr-x  1 root root  371 Dec 28 22:47 intelmq.bots.experts.modify.expert
-rwxr-xr-x  1 root root  381 Dec 28 22:47 intelmq.bots.experts.reverse_dns.expert
-rwxr-xr-x  1 root root  373 Dec 28 22:47 intelmq.bots.experts.rfc1918.expert
-rwxr-xr-x  1 root root  401 Dec 28 22:47 intelmq.bots.experts.ripencc_abuse_contact.expert
-rwxr-xr-x  1 root root  375 Dec 28 22:47 intelmq.bots.experts.taxonomy.expert
-rwxr-xr-x  1 root root  377 Dec 28 22:47 intelmq.bots.experts.tor_nodes.expert
-rwxr-xr-x  1 root root  375 Dec 28 22:47 intelmq.bots.experts.url2fqdn.expert
-rwxr-xr-x  1 root root  377 Dec 28 22:47 intelmq.bots.outputs.amqptopic.output
-rwxr-xr-x  1 root root  367 Dec 28 22:47 intelmq.bots.outputs.file.output
-rwxr-xr-x  1 root root  373 Dec 28 22:47 intelmq.bots.outputs.mongodb.output
-rwxr-xr-x  1 root root  379 Dec 28 22:47 intelmq.bots.outputs.postgresql.output
-rwxr-xr-x  1 root root  369 Dec 28 22:47 intelmq.bots.outputs.redis.output
-rwxr-xr-x  1 root root  373 Dec 28 22:47 intelmq.bots.outputs.restapi.output
-rwxr-xr-x  1 root root  365 Dec 28 22:47 intelmq.bots.outputs.tcp.output
-rwxr-xr-x  1 root root  365 Dec 28 22:47 intelmq.bots.outputs.udp.output
-rwxr-xr-x  1 root root  367 Dec 28 22:47 intelmq.bots.outputs.xmpp.output
-rwxr-xr-x  1 root root  387 Dec 28 22:47 intelmq.bots.parsers.abusech.parser_domain
-rwxr-xr-x  1 root root  379 Dec 28 22:47 intelmq.bots.parsers.abusech.parser_ip
-rwxr-xr-x  1 root root  395 Dec 28 22:47 intelmq.bots.parsers.abusech.parser_ransomware
-rwxr-xr-x  1 root root  379 Dec 28 22:47 intelmq.bots.parsers.alienvault.parser
-rwxr-xr-x  1 root root  387 Dec 28 22:47 intelmq.bots.parsers.alienvault.parser_otx
-rwxr-xr-x  1 root root  375 Dec 28 22:47 intelmq.bots.parsers.autoshun.parser
-rwxr-xr-x  1 root root  375 Dec 28 22:47 intelmq.bots.parsers.bambenek.parser
-rwxr-xr-x  1 root root  373 Dec 28 22:47 intelmq.bots.parsers.bitcash.parser
-rwxr-xr-x  1 root root  375 Dec 28 22:47 intelmq.bots.parsers.bitsight.parser
-rwxr-xr-x  1 root root  381 Dec 28 22:47 intelmq.bots.parsers.blocklistde.parser
-rwxr-xr-x  1 root root  397 Dec 28 22:47 intelmq.bots.parsers.blueliv.parser_crimeserver
-rwxr-xr-x  1 root root  373 Dec 28 22:47 intelmq.bots.parsers.ci_army.parser
-rwxr-xr-x  1 root root  373 Dec 28 22:47 intelmq.bots.parsers.cleanmx.parser
-rwxr-xr-x  1 root root  393 Dec 28 22:47 intelmq.bots.parsers.cymru_full_bogons.parser
-rwxr-xr-x  1 root root  383 Dec 28 22:47 intelmq.bots.parsers.danger_rulez.parser
-rwxr-xr-x  1 root root  377 Dec 28 22:47 intelmq.bots.parsers.dataplane.parser
-rwxr-xr-x  1 root root  381 Dec 28 22:47 intelmq.bots.parsers.dshield.parser_asn
-rwxr-xr-x  1 root root  385 Dec 28 22:47 intelmq.bots.parsers.dshield.parser_block
-rwxr-xr-x  1 root root  387 Dec 28 22:47 intelmq.bots.parsers.dshield.parser_domain
-rwxr-xr-x  1 root root  365 Dec 28 22:47 intelmq.bots.parsers.dyn.parser
-rwxr-xr-x  1 root root  387 Dec 28 22:47 intelmq.bots.parsers.fraunhofer.parser_dga
-rwxr-xr-x  1 root root  381 Dec 28 22:47 intelmq.bots.parsers.generic.parser_csv
-rwxr-xr-x  1 root root  373 Dec 28 22:47 intelmq.bots.parsers.hphosts.parser
-rwxr-xr-x  1 root root  367 Dec 28 22:47 intelmq.bots.parsers.json.parser
-rwxr-xr-x  1 root root  373 Dec 28 22:47 intelmq.bots.parsers.malc0de.parser
-rwxr-xr-x  1 root root  393 Dec 28 22:47 intelmq.bots.parsers.malwaredomainlist.parser
-rwxr-xr-x  1 root root  387 Dec 28 22:47 intelmq.bots.parsers.malwaredomains.parser
-rwxr-xr-x  1 root root  411 Dec 28 22:47 intelmq.bots.parsers.malwarepatrol.parser_dansguardian
-rwxr-xr-x  1 root root  367 Dec 28 22:47 intelmq.bots.parsers.misp.parser
-rwxr-xr-x  1 root root  379 Dec 28 22:47 intelmq.bots.parsers.n6.parser_n6stomp
-rwxr-xr-x  1 root root  379 Dec 28 22:47 intelmq.bots.parsers.netlab_360.parser
-rwxr-xr-x  1 root root  373 Dec 28 22:47 intelmq.bots.parsers.nothink.parser
-rwxr-xr-x  1 root root  371 Dec 28 22:47 intelmq.bots.parsers.openbl.parser
-rwxr-xr-x  1 root root  377 Dec 28 22:47 intelmq.bots.parsers.openphish.parser
-rwxr-xr-x  1 root root  377 Dec 28 22:47 intelmq.bots.parsers.phishtank.parser
-rwxr-xr-x  1 root root  383 Dec 28 22:47 intelmq.bots.parsers.shadowserver.parser
-rwxr-xr-x  1 root root  385 Dec 28 22:47 intelmq.bots.parsers.spamhaus.parser_cert
-rwxr-xr-x  1 root root  385 Dec 28 22:47 intelmq.bots.parsers.spamhaus.parser_drop
-rwxr-xr-x  1 root root  375 Dec 28 22:47 intelmq.bots.parsers.taichung.parser
-rwxr-xr-x  1 root root  371 Dec 28 22:47 intelmq.bots.parsers.turris.parser
-rwxr-xr-x  1 root root  371 Dec 28 22:47 intelmq.bots.parsers.urlvir.parser
-rwxr-xr-x  1 root root  373 Dec 28 22:47 intelmq.bots.parsers.vxvault.parser
-rwxr-xr-x  1 root root  323 Dec 28 22:47 intelmqctl
-rwxr-xr-x  1 root root  325 Dec 28 22:47 intelmqdump
-rwxr-xr-x  1 root root  341 Dec 28 22:47 intelmq_psql_initdb
-rwxr-xr-x  1 root root 1103 Dec 28 22:46 update-asn-data
-rwxr-xr-x  1 root root  864 Dec 28 22:46 update-geoip-data
-rwxr-xr-x  1 root root  838 Dec 28 22:46 update-tor-nodes

root@server:~# ls -la /usr/local/lib/python3.5/dist-packages/
total 68
drwxrwsr-x 16 root staff 4096 Dec 28 22:47 .
drwxrwsr-x  3 root staff 4096 Dec 20 03:19 ..
drwxr-sr-x  5 root staff 4096 Dec 28 22:46 dateutil
drwxr-sr-x  4 root staff 4096 Dec 28 22:46 dns
drwxr-sr-x  2 root staff 4096 Dec 28 22:46 dnspython-1.15.0.dist-info
drwxr-sr-x  8 root staff 4096 Dec 28 22:47 intelmq
drwxr-sr-x  2 root staff 4096 Dec 28 22:47 intelmq-1.0.0.dev5.egg-info
drwxr-sr-x  4 root staff 4096 Dec 28 22:46 psutil
drwxr-sr-x  2 root staff 4096 Dec 28 22:46 psutil-5.0.1.dist-info
drwxr-sr-x  2 root staff 4096 Dec 28 22:46 __pycache__
drwxr-sr-x  2 root staff 4096 Dec 28 22:46 python_dateutil-2.6.0.dist-info
drwxr-sr-x  2 root staff 4096 Dec 28 22:46 python_termstyle-0.1.10.dist-info
drwxr-sr-x  4 root staff 4096 Dec 28 22:46 pytz
drwxr-sr-x  2 root staff 4096 Dec 28 22:46 pytz-2016.10.dist-info
drwxr-sr-x  3 root staff 4096 Dec 28 22:46 redis
drwxr-sr-x  2 root staff 4096 Dec 28 22:46 redis-2.10.5.dist-info
-rw-r--r--  1 root staff 3195 Dec 28 22:46 termstyle.py

root@server:~# ls -la /usr/local/lib/python3.5/dist-packages/intelmq/
total 40
drwxr-sr-x  8 root staff 4096 Dec 28 22:47 .
drwxrwsr-x 16 root staff 4096 Dec 28 22:47 ..
drwxr-sr-x  3 root staff 4096 Dec 28 22:47 bin
drwxr-sr-x  7 root staff 4096 Dec 28 22:47 bots
drwxr-sr-x  2 root staff 4096 Dec 28 22:47 etc
-rw-r--r--  1 root staff  679 Dec 28 22:46 __init__.py
drwxr-sr-x  3 root staff 4096 Dec 28 22:47 lib
drwxr-sr-x  2 root staff 4096 Dec 28 22:47 __pycache__
drwxr-sr-x  6 root staff 4096 Dec 28 22:47 tests
-rw-r--r--  1 root staff   84 Dec 28 22:46 version.py

root@server:~# ls -la /usr/local/lib/python3.5/dist-packages/opt/intelmq/
ls: cannot access '/usr/local/lib/python3.5/dist-packages/opt/intelmq/': No such file or directory

root@server:~# ls -la /opt/
total 12
drwxr-xr-x  3 root    root    4096 Dec 28 22:47 .
drwxr-xr-x 23 root    root    4096 Dec 20 03:54 ..
drwxrwx---  4 intelmq intelmq 4096 Dec 28 22:47 intelmq

root@server:~# ls -la /opt/intelmq/
total 16
drwxrwx--- 4 intelmq intelmq 4096 Dec 28 22:47 .
drwxr-xr-x 3 root    root    4096 Dec 28 22:47 ..
drwxrwx--- 3 intelmq intelmq 4096 Dec 28 22:47 etc
drwxrwx--- 4 intelmq intelmq 4096 Dec 28 22:47 var

Analysis

  1. '/usr/local/bin/' contains the "link" to bots (e.g. /usr/local/bin/intelmq.bots.parsers.phishtank.parser) which is correct because is a development installation.
  2. '/opt/intelmq' exist and was correctly set up. Although, there is a path '/usr/local/lib/python3.5/dist-packages/intelmq/etc/' which was created (may be was intentionally configured in setup.py) but IMHO creates some confusion to the users when they test the system via command line. The reason why is due the fact that even configurations still stored in '/opt/intelmq/etc/examples/' which are not read by IntelMQ, the system will use the configurations stored in '/usr/local/lib/python3.5/dist-packages/intelmq/etc/' without user notice.

Proposal

ghost commented 7 years ago

I uploaded a new version (dev6) to pypi, ignoring the wheel. It should work now.

SYNchroACK commented 7 years ago

Installation via 'pip3 install intelmq'

Proposal:

  • Installation via 'pip3 install intelmq' should not create '/usr/local/lib/python3.5/dist-packages/opt/intelmq' folder, instead should create on '/opt/intelmq'.

@wagner-certat I tried again minutes ago and still not working like we talked 3 days ago. :/

Installation via 'pip3 install .'

Proposal:

  • Installation via 'pip3 install .' should not create '/usr/local/lib/python3.5/dist-packages/intelmq/etc/' folder and rely 100% on '/opt/intelmq/etc/' folder.

Issue stills.

SYNchroACK commented 7 years ago

As we (me and @wagner-certat ) discussed today, this is not important since on v1.0 users will use packages (.deb and .rpm). This issue should be revisited in the end of work regarding v1.0....

ghost commented 7 years ago

I'm not sure if we can actually solve this issue at all.

SYNchroACK commented 6 years ago

Tests Results

pip3 install intelmq

The command pip3 install intelmq will not create the directory and configuration files on /opt/intelmq. It will create the opt/intelmq directory in python3 -c "import site; print(site.getsitepackages()[0])"

pip3 install .

The command pip3 install . will create the directory and configuration files on /opt/intelmq.