search

viper-framework / viper

Binary analysis and management framework
Other
1.54k stars 351 forks source link

Clarification - supported DB backends? #650

Open frennkie opened 6 years ago

frennkie commented 6 years ago

There are some issues that refer to database backends other than SQLite (#583 #573 #302).

The Viper documentation (https://github.com/viper-framework/viper/blob/master/docs/source/usage/concepts.rst) states that:

You can create as many projects as you want and you can easily switch from one to another. Each project will have its own local repositories of binary files, a SQLite database containing metadata

I tested this on PostgreSQL and also looked through the code. Having more than one database currently only works automatically with SQLite. A workaround is to manually edit the viper.conf, replace the connection string and restart Viper. viper-update currently also only takes care of SQLite files.

So either we explicitly state that users MUST use SQLite or we add the missing things to the roadmap. What do you think @botherder @Rafiot @deralexxx ?

frennkie commented 6 years ago

Here is a log of a session using a PostgreSQL database and trying to use different projects:

$: tree /tmp/testing
/tmp/testing
├── viper_case1
│   ├── file5
│   ├── file6
│   └── file7
├── viper_case2
│   ├── file8
│   └── file9
└── viper_default
    ├── file1
    ├── file2
    ├── file3
    └── file4

3 directories, 9 files

$: ./viper-cli
/home/vipert/work/viper/venv/lib/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8;
in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
         _
        (_)
   _   _ _ ____  _____  ____
  | | | | |  _ \| ___ |/ ___)
   \ V /| | |_| | ____| |
    \_/ |_|  __/|_____)_| v1.3-dev
          |_|

You have 0 files in your default repository
viper > about
+----------------+-------------------------------------------------+
| About          |                                                 |
+----------------+-------------------------------------------------+
| Viper Version  | 1.3-dev                                         |
| Python Version | 3.5.2                                           |
| Homepage       | https://viper.li                                |
| Issue Tracker  | https://github.com/viper-framework/viper/issues |
+----------------+-------------------------------------------------+
+--------------------+--------------------------------------------------+
| Configuration      |                                                  |
+--------------------+--------------------------------------------------+
| Configuration File | /home/vipert/work/viper/viper/viper.conf         |
| Active Project     | default                                          |
| Storage Path       | /home/vipert/.viper                              |
| Database Path      | postgresql://viper:changeme@localhost:5432/viper |
+--------------------+--------------------------------------------------+

viper > store -f /tmp/testing/viper_default
[+] Stored file "file4" to /home/vipert/.viper/binaries/d/6/d/1/d6d17244b216e2490b565c70683624accf42f72a30873bd5ed8a457e16eebb0e
[*] Session opened on /home/vipert/.viper/binaries/d/6/d/1/d6d17244b216e2490b565c70683624accf42f72a30873bd5ed8a457e16eebb0e
[*] Running command "yara scan -t"
[*] Scanning file4 (d6d17244b216e2490b565c70683624accf42f72a30873bd5ed8a457e16eebb0e)
[*] Running command "triage"
[+] Stored file "file3" to /home/vipert/.viper/binaries/7/4/6/5/7465b8337e40016ec17c5e3be6377ec4358cafb35db81d6b24dbd2ff29447c36
[*] Session opened on /home/vipert/.viper/binaries/7/4/6/5/7465b8337e40016ec17c5e3be6377ec4358cafb35db81d6b24dbd2ff29447c36
[*] Running command "yara scan -t"                                                                                                                           
[*] Scanning file3 (7465b8337e40016ec17c5e3be6377ec4358cafb35db81d6b24dbd2ff29447c36)
[*] Running command "triage"
[+] Stored file "file1" to /home/vipert/.viper/binaries/8/e/c/0/8ec0e7ae18a08cb4bcbb659e6efcecae30f6be5e16e6cfb425d938d2b1701801
[*] Session opened on /home/vipert/.viper/binaries/8/e/c/0/8ec0e7ae18a08cb4bcbb659e6efcecae30f6be5e16e6cfb425d938d2b1701801
[*] Running command "yara scan -t"
[*] Scanning file1 (8ec0e7ae18a08cb4bcbb659e6efcecae30f6be5e16e6cfb425d938d2b1701801)
[*] Running command "triage"
viper > projects -l
[*] The projects directory does not exist yet
viper > projects -s case1
[*] Switched to project case1
case1 viper > store -f /tmp/testing/viper_case1
[+] Stored file "file7" to /home/vipert/.viper/projects/case1/binaries/b/9/1/4/b914d94373e431658832a84a46712d77d7acaff93c6eacbc1c996886c0f25dae
[*] Session opened on /home/vipert/.viper/projects/case1/binaries/b/9/1/4/b914d94373e431658832a84a46712d77d7acaff93c6eacbc1c996886c0f25dae
[*] Running command "yara scan -t"
[*] Scanning file7 (b914d94373e431658832a84a46712d77d7acaff93c6eacbc1c996886c0f25dae)
[*] Running command "triage"
[+] Stored file "file6" to /home/vipert/.viper/projects/case1/binaries/4/c/f/f/4cfffcc1f1d82d642c9325ecab7168055cf0dde16e7a872a27a256aed8976d86
[*] Session opened on /home/vipert/.viper/projects/case1/binaries/4/c/f/f/4cfffcc1f1d82d642c9325ecab7168055cf0dde16e7a872a27a256aed8976d86
[*] Running command "yara scan -t"
[*] Scanning file6 (4cfffcc1f1d82d642c9325ecab7168055cf0dde16e7a872a27a256aed8976d86)
[*] Running command "triage"
[+] Stored file "file5" to /home/vipert/.viper/projects/case1/binaries/4/5/4/0/454012b57d582deb8d6b048cec77cd0f331c0e8c5742638caeab899345f275ae
[*] Session opened on /home/vipert/.viper/projects/case1/binaries/4/5/4/0/454012b57d582deb8d6b048cec77cd0f331c0e8c5742638caeab899345f275ae
[*] Running command "yara scan -t"
[*] Scanning file5 (454012b57d582deb8d6b048cec77cd0f331c0e8c5742638caeab899345f275ae)
[*] Running command "triage"
case1 viper > projects -s case2
[*] Switched to project case2
case2 viper > store -f /tmp/testing/viper_case2
[+] Stored file "file9" to /home/vipert/.viper/projects/case2/binaries/8/3/0/d/830d0a80fb1df2d8f9a78e6d223c3d1e0b7add927cfcaf8897e650bc216af0da
[*] Session opened on /home/vipert/.viper/projects/case2/binaries/8/3/0/d/830d0a80fb1df2d8f9a78e6d223c3d1e0b7add927cfcaf8897e650bc216af0da
[*] Running command "yara scan -t"
[*] Scanning file9 (830d0a80fb1df2d8f9a78e6d223c3d1e0b7add927cfcaf8897e650bc216af0da)
[*] Running command "triage"
[+] Stored file "file8" to /home/vipert/.viper/projects/case2/binaries/5/8/6/6/5866438985c4f41a1fe70bd08cf018fcaa6dd6ee075efb8269c813abe99eded7
[*] Session opened on /home/vipert/.viper/projects/case2/binaries/5/8/6/6/5866438985c4f41a1fe70bd08cf018fcaa6dd6ee075efb8269c813abe99eded7
[*] Running command "yara scan -t"
[*] Scanning file8 (5866438985c4f41a1fe70bd08cf018fcaa6dd6ee075efb8269c813abe99eded7)
[*] Running command "triage"
case2 viper >
case2 viper > projects -l
[*] Projects Available:
+--------------+--------------------------+---------+
| Project Name | Creation Time            | Current |
+--------------+--------------------------+---------+
| case1        | Sun Feb 18 14:52:22 2018 |         |
| case2        | Sun Feb 18 14:52:37 2018 | Yes     |
+--------------+--------------------------+---------+
case2 viper > projects -s default
[*] Switched to project default
default viper > projects -l
[*] Projects Available:
+--------------+--------------------------+---------+
| Project Name | Creation Time            | Current |
+--------------+--------------------------+---------+
| case1        | Sun Feb 18 14:52:22 2018 |         |
| case2        | Sun Feb 18 14:52:37 2018 |         |
+--------------+--------------------------+---------+
default viper > find all
+---+-------+--------------------------+----------------------------------+------+
| # | Name  | Mime                     | MD5                              | Tags |
+---+-------+--------------------------+----------------------------------+------+
| 1 | file4 | application/octet-stream | 2a70104edc73f0ea5e450dab92fc5a11 |      |
| 2 | file3 | application/octet-stream | 13e277d58cfb358da9a6e25634910d7d |      |
| 3 | file1 | application/octet-stream | addcac07a641ea4f657b39996768e4c9 |      |
| 4 | file7 | application/octet-stream | 2631254381f21571b1728b8eaa5fd040 |      |
| 5 | file6 | application/octet-stream | b01a627f828a400e3eefa69373281e23 |      |
| 6 | file5 | application/octet-stream | 88f3ef847927c0c3a16aa6a509eca111 |      |
| 7 | file9 | application/octet-stream | 0d1bd8cdcb8e41370cfc24ea6f255de7 |      |
| 8 | file8 | application/octet-stream | 50bbc877fb1a0a7c2104935e763ae9eb |      |
+---+-------+--------------------------+----------------------------------+------+
default viper > projects -s case1
[*] Switched to project case1
case1 viper > find all
+---+-------+--------------------------+----------------------------------+------+
| # | Name  | Mime                     | MD5                              | Tags |
+---+-------+--------------------------+----------------------------------+------+
| 1 | file4 | application/octet-stream | 2a70104edc73f0ea5e450dab92fc5a11 |      |
| 2 | file3 | application/octet-stream | 13e277d58cfb358da9a6e25634910d7d |      |
| 3 | file1 | application/octet-stream | addcac07a641ea4f657b39996768e4c9 |      |
| 4 | file7 | application/octet-stream | 2631254381f21571b1728b8eaa5fd040 |      |
| 5 | file6 | application/octet-stream | b01a627f828a400e3eefa69373281e23 |      |
| 6 | file5 | application/octet-stream | 88f3ef847927c0c3a16aa6a509eca111 |      |
| 7 | file9 | application/octet-stream | 0d1bd8cdcb8e41370cfc24ea6f255de7 |      |
| 8 | file8 | application/octet-stream | 50bbc877fb1a0a7c2104935e763ae9eb |      |
+---+-------+--------------------------+----------------------------------+------+
case1 viper > open -l 3
[!] You have to open a session on a path or on a misp event.
case1 viper > open -l 9
case1 viper > info
case1 viper > projects -s case2
[*] Switched to project case2
case2 viper > open -l 9
case2 viper > info
case2 viper > projects -s default
[*] Switched to project default
default viper > open -l 9
default viper > info
default viper > about
+----------------+-------------------------------------------------+
| About          |                                                 |
+----------------+-------------------------------------------------+
| Viper Version  | 1.3-dev                                         |
| Python Version | 3.5.2                                           |
| Homepage       | https://viper.li                                |
| Issue Tracker  | https://github.com/viper-framework/viper/issues |
+----------------+-------------------------------------------------+
+--------------------+--------------------------------------------------+
| Configuration      |                                                  |
+--------------------+--------------------------------------------------+
+--------------------+--------------------------------------------------+
| Configuration File | /home/vipert/work/viper/viper/viper.conf         |
| Active Project     | default                                          |
| Storage Path       | /home/vipert/.viper                              |
| Database Path      | postgresql://viper:changeme@localhost:5432/viper |
+--------------------+--------------------------------------------------+
default viper > open -l 3
[*] Session opened on /home/vipert/.viper/binaries/8/e/c/0/8ec0e7ae18a08cb4bcbb659e6efcecae30f6be5e16e6cfb425d938d2b1701801
default viper file1 > info
+----------+----------------------------------------------------------------------------------------------------------------------------------+
| Key      | Value                                                                                                                            |
+----------+----------------------------------------------------------------------------------------------------------------------------------+
| Name     | file1                                                                                                                            |
| Tags     |                                                                                                                                  |
| Path     | /home/vipert/.viper/binaries/8/e/c/0/8ec0e7ae18a08cb4bcbb659e6efcecae30f6be5e16e6cfb425d938d2b1701801                            |
| Size     | 2097152                                                                                                                          |
| Type     | data                                                                                                                             |
| Mime     | application/octet-stream                                                                                                         |
| MD5      | addcac07a641ea4f657b39996768e4c9                                                                                                 |
| SHA1     | fe367a4d8e166b7072ab579fc46ed66a4c643c49                                                                                         |
| SHA256   | 8ec0e7ae18a08cb4bcbb659e6efcecae30f6be5e16e6cfb425d938d2b1701801                                                                 |
| SHA512   | e5944e845b826a69985c2305de8740969951b34262694a2bd62416c5ff55d7a85eb385dc9242f4b58d033f7cb38ae352b3fe3c1f28193aa2b901d98d8a4beb0c |
| SSdeep   | 49152:lhUkYoVnJFOm1x0PSyCFCbGPIT8kTPj7KW+Rw4jNlQQuS:Z/nfNxOhbGPIYqr8lQQuS                                                        |
| CRC32    | 7AE90063                                                                                                                         |
| Parent   |                                                                                                                                  |
| Children |                                                                                                                                  |
+----------+----------------------------------------------------------------------------------------------------------------------------------+
default viper file1 >

$: tree /home/vipert/.viper
/home/vipert/.viper
├── binaries
│   ├── 7
│   │   └── 4
│   │       └── 6
│   │           └── 5
│   │               └── 7465b8337e40016ec17c5e3be6377ec4358cafb35db81d6b24dbd2ff29447c36
│   ├── 8
│   │   └── e
│   │       └── c
│   │           └── 0
│   │               └── 8ec0e7ae18a08cb4bcbb659e6efcecae30f6be5e16e6cfb425d938d2b1701801
│   └── d
│       └── 6
│           └── d
│               └── 1
│                   └── d6d17244b216e2490b565c70683624accf42f72a30873bd5ed8a457e16eebb0e
├── history
├── projects
│   ├── case1
│   │   └── binaries
│   │       ├── 4
│   │       │   ├── 5
│   │       │   │   └── 4
│   │       │   │       └── 0
│   │       │   │           └── 454012b57d582deb8d6b048cec77cd0f331c0e8c5742638caeab899345f275ae
│   │       │   └── c
│   │       │       └── f
│   │       │           └── f
│   │       │               └── 4cfffcc1f1d82d642c9325ecab7168055cf0dde16e7a872a27a256aed8976d86
│   │       └── b
│   │           └── 9
│   │               └── 1
│   │                   └── 4
│   │                       └── b914d94373e431658832a84a46712d77d7acaff93c6eacbc1c996886c0f25dae
│   └── case2
│       └── binaries
│           ├── 5
│           │   └── 8
│           │       └── 6
│           │           └── 6
│           │               └── 5866438985c4f41a1fe70bd08cf018fcaa6dd6ee075efb8269c813abe99eded7
│           └── 8
│               └── 3
│                   └── 0
│                       └── d
│                           └── 830d0a80fb1df2d8f9a78e6d223c3d1e0b7add927cfcaf8897e650bc216af0da
├── scraper
└── viper.log

38 directories, 10 files

So this is definitely broken.

jaegeral commented 6 years ago

Agree, we should than change the documentation to sqlite is the only one at the moment really working

botherder commented 6 years ago

I'm not sure I understand the problem. If by changing the connection configuration to specify another type of DBMS everything works fine, it is intended behavior. No?

frennkie commented 6 years ago

Hm.. I think my main concern is that the behavior when using SQLite is documented and easily to understand... you run ./viper-cli and can switch between projects. Each project has it's own database and the binaries are in a separate folder per project.

But if you use a different DBMS you have to exit viper, change the .conf file and then run viper-cli -p caseA to make sure that the selected database matches the binaries found in the project dir.

If this is the intended behavior then this should at least be documented (currently there is not mention of changing connection in the concepts.rst doc).