search

CERT-Polska / ursadb

Trigram database written in C++, suited for malware indexing
BSD 3-Clause "New" or "Revised" License
124 stars 25 forks source link

Remove files without rebuilding datasets #216

Open pavb-fr opened 10 months ago

pavb-fr commented 10 months ago

Currently the deletion of files is not supported by Ursadb as mentioned in the documentation https://github.com/CERT-Polska/ursadb/blob/master/docs/datasets.md#add-and-remove-files

But we have some use cases where we want to add a lot of temporary files for a short duration. We would like to be able to remove them after use without reindexing the database which could be very time consuming.

The tag feature on dataset seems very interesting for such use cases but it is not supported by the solution MalwareDB / MQuery.

pavb-fr commented 10 months ago

Maybe it can be linked to this issue on Mwdb https://github.com/CERT-Polska/mwdb-core/issues/560

If we can have a different ursadb tag by mwdb bucket

msm-code commented 10 months ago

But we have some use cases where we want to add a lot of temporary files for a short duration.

Hi, do I understand correctly that you want to index some files temporarily and later remove them all? You can remove datasets (groups of files) (as you already know from documentation you linked), just not individual files.

The only problem is that it's hard to know which datasets to remove after indexing, but it's actually very simple to fix. You can group files during indexing by adding a specified tag (known as taint in ursadb world), for example:

index "/opt" with taints ["temporary123"]

This will create datasets with tags, similarly to the screen below:

image

Datasets with one set of tags will never merge with datasets that have different tags. So after your temporary work is done, you can just query for all datasets with your tag temporary123 and issue dataset "xxxx" drop commands.

I think this is even easier than keeping track of which files to remove after processing. And the added benefit is that (if you want to) you may use the tags to scope your mquery queries to for example only query your temporary files. Would this work for you?

pavb-fr commented 10 months ago

Yes I think it should be fine for my use case when the issue https://github.com/CERT-Polska/mwdb-core/issues/560 will be closed. Indeed I need a way to split the sample files by tag to be able to index them correctly. Having a different s3 bucket or a different root path for the samples should fixed that.

On the other hand, I still have errors in mquery with missing files. But as described in the documentation mquery should ignored them. Maybe the issue should be open on mquery rather than here.

msm-code commented 10 months ago

On the other hand, I still have errors in mquery with missing files. But as described in the documentation mquery should ignored them. Maybe the issue should be open on mquery rather than here.

If you remove the dataset from ursadb it shouldn't be an issue - the files will be gone from mquery as well. But if you just remove files from disk then you'll get errors.

I agree that https://github.com/CERT-Polska/mwdb-core/issues/560 may help with implementing this in practice