search

malwaredb / malwaredb-rs

MalwareDB: bookkeeping for malware, goodware, and unknown files with relationship discovery
https://malwaredb.net/
Apache License 2.0
30 stars 4 forks source link

Client doesn't upload samples to server #398

Closed dmknght closed 4 months ago

dmknght commented 4 months ago

I'm using compiled file from release version for both client and server. My configuration:

rjzak commented 4 months ago

Thank you for this. From what you've shared, the file should have been saved and reflected in the stats. I'm investigating.

rjzak commented 4 months ago

It seems to be working, I set up a fresh server based on the latest code from Git 9e9695f0158a111c6190b4cdd4e103c70b725592 and have been loading in data from VirusShare (as you were doing).

Command: ./target/debug/mdb_client submit-samples -s 1 -d ~/Downloads/VirusShare_00000.zip -p infected

Stats:

❯ /usr/local/sbin/mdb_server admin -c /usr/local/etc/mdb_server/mdb_config.toml stats
Number of samples: 588
Number of users: 1
Number of groups: 2
Number of sources: 487
Database size: 13 MB
Database version: PostgreSQL 16.2 on amd64-portbld-freebsd14.0, compiled by clang version 15.0.7, 64-bit, no similarity hashing extensions are installed
File counts by type:
Office97: 22
PDF: 3
PE32: 562
ELF: 1

It's possible there's a bug in the latest release, and it's possible that the PE32 parser for that particular sample failed. That's an area that needs more work, ensuring the parsers don't fail. Right now, loading in a bunch of files from a Zip, I can see some panics because of arrays having an index that's out of bounds. That would bring down the parsing of that file only.

dmknght commented 4 months ago

Hello. Sorry that I didn't give full info (I forgot about it when I created this issue). The file in the command I gave about was a PE32 file. I also tested with many ELF files and all of them failed. For some reasons, I tried some zip files and the result showed Submitted 0 files (on the client-side). Is it possible that on my server (debian testing), I'm missing some runtime libraries hence errors happened? BTW the reason I used release version is because rustc failed to build something. I'm giving it a try again

rjzak commented 4 months ago

There's no runtime requirements which could be a surprise, if something was missing, there'd be an OS error. You have MDB configured to store files in /home/test/samples, does the user account running the server have write access to that directory?

BTW the reason I used release version is because rustc failed to build something.

dmknght commented 4 months ago

You have MDB configured to store files in /home/test/samples, does the user account running the server have write access to that directory?

Folder samples has default permission 755. I tested 2 cases: owned by root and owned by test. Both gave the same error nothing is there. I'm running server at port 80, so it requires root permission to starts, so I assume owner of this folder is not a problem.

What error did you get?

Ah just casual compatible Rust compiler version on Debian.

Can you try the latest binary from the recent Github Actions? https://github.com/malwaredb/malwaredb-rs/actions/runs/9259947401

Thank you I'm trying it. I had a ton of errors with the older rustc version on Debian (on the server) so it helps me a lot. I'm trying right now.

Update: I'm still seeing 0 samples. I'm using both client and server from your actions link. At this point I hope I didn't configure something wrong?

dmknght commented 4 months ago

Update: I tried uploading file with mdb_server command:

  1. Copy 2 sample files, 1 zip and 1 ELF to samples
  2. Run command: mdb_server admin -c /etc/mdb_server/mdb_config.toml bulk-add --source-id 1 --user-id 0 /home/test/samples/
  3. Result: segment fault

Edit: strace output of this segment fault:

  1. Server tried reading file's header (i suppose) image
  2. Segment fault after some mmap and munmap image
rjzak commented 4 months ago

Could you install latest Rust (based on instructions on the Rust website) and recompile? The debug info would likely be more useful. Not that it should matter, but what version of Debian are you using? I test and develop on Debian 12, and test with a FreeBSD server and on a Mac.

Can you run cargo test --workspace after installing latest Rust? The unit tests cover loading files, and if it has a problem there, it might have more useful output.

I'm sorry this hasn't been a smooth experience for you; I haven't encountered these issues, but I'm sure we can figure it out.

dmknght commented 4 months ago

Could you install latest Rust (based on instructions on the Rust website) and recompile?

Okay i'm doing it. I'm using Debian trixie (i believe it's testing). Let me switch to unstable branch and see how it goes.

I'm sorry this hasn't been a smooth experience for you; I haven't encountered these issues, but I'm sure we can figure it out.

No worries LoL. I mean it's a program. Bugs are expected

dmknght commented 4 months ago

Can you run cargo test --workspace after installing latest Rust? The unit tests cover loading files, and if it has a problem there, it might have more useful output.

Well Debian is having rustc 1.72.1 on unstable, and older on other branches. I tested and cargo build gave me so many erros like package xxx requires rustc 1.74 or newer. I hope i didn't do something wrong here. Meanwhile is there any possible reason that causes crashes with the build version in action?

rjzak commented 4 months ago

I can't think of any reason why you'd have this issue or why the release builds from GitHub Actions would fail, especially since the tests passed and the tests have a few places where files are added then checked on disk and in the database to ensure the file & records exist.

The Rust website has installation instructions: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh.

https://www.rust-lang.org/tools/install

I'll try MalwareDB on Debian Trixie/Testing this evening and see if I encounter any issues.

dmknght commented 4 months ago

I have good news:Install latest ruts helped. I am having mdb_server with features. I did simple test case:

  1. Create a separate folder contains malware at /home/test/malware_list. It has an ELF and a ZIP file.
  2. The dir for server's configuration should be at /home/test/samples as before
  3. Use admin bulk-add to upload malicious files in malware_list. I tested with both server didn't run and server ran.
  4. Instead of segment fault, I got error Error: uid 0 not allowed to upload to sid 1. The value of uid and sid is from the command line. I tried several values like uid = 0, 1, 1000 and i have the same error.

Update: quick grep showed me it's error in pg.rs and sqlite.rs so likely this one is some kind of database error? Edit: uploading file from client still made no new file in samples. I forgot mentioning it.

rjzak commented 4 months ago

MDB keeps track of samples by user and group, and permissions for collections of samples (sources) are also by group. The admin commands let you create a group and source, link them, then add the user to the group. Then you can upload the samples to the group with the relationship to the uploading user. Did you set that up earlier? If not, I wonder if there was some condition I didn't check which caused weird or silent errors when you tried adding file(s).

dmknght commented 4 months ago

Did you set that up earlier?

I believe I didn't. Everything I configured was the config file only. I suppose this is the same error that caused crash.

rjzak commented 4 months ago

Check out these commands and make a test source and group, and add the relations.

I added a few entries for VirusShare, and I see:

❯ ./mdb_client whoami
UserID: 0
You're part of 2 groups:
    admin
    default
You have access to 2 sources:
    VirusShare_00000.zip
    VirusShare_00001.zip
dmknght commented 4 months ago

Creating group gave me error database is running. I think at this point i'm going to create a server from scratch again.

rjzak commented 4 months ago

Wow, that is weird. Maybe it's a VM issue? I haven't tried MDB in a VM, nor with Trixie/Testing yet (didn't have time the other night). Maybe try it with SQLite first while you're getting a feel for it?

dmknght commented 4 months ago

Wow, that is weird. Maybe it's a VM issue? I haven't tried MDB in a VM, nor with Trixie/Testing yet (didn't have time the other night). Maybe try it with SQLite first while you're getting a feel for it?

I'm not sure what i did wrong. I created a postgres DB and created a connection. I think it's better to do everything from scratch first, from build with latest rustc version.

rjzak commented 4 months ago

Hopefully there will be lessons learned so we can make this an easier, smoother, simpler process in the future.

dmknght commented 4 months ago

Hopefully there will be lessons learned so we can make this an easier, smoother, simpler process in the future.

I decided drop db, remove configs and other stuff instead of making a server fully from scratch to save time. I'm more familiar with MySQL syntax so playing with Postgres wasn't really fun at all LoL.

Anyway If I managed to make it, I would like to list my steps as an instruction because I personally feel like a detailed tutorial is better.

dmknght commented 4 months ago

Good news: Everything is working. I think the error of this topic was my mistake didn't add user, group to source. That's my bad skipping 4th step for some reason lol. Kinda bad news: When I uploaded a big file, server gave me error: image

File is ELF, hash: 00ae07c9fe63b080181b8a6d59c6b3b6f9913938858829e5a42ab90fb72edf7a, size: 7.8 MB Update:

P/S: I think the code logic should be updated when admin didn't set source too.

rjzak commented 4 months ago

A new issue would be great, and the code should display a useful error when the needed info isn't present. An issue for that would also help so I can try to replicate that.

My hope was to make it so you don't have to do much to get things working with Postgres, so that's something I can work on starting with better documentation. But some things can't be helped since admin rights are needed to make a user and create a database.