search

simsong / bulk_extractor

This is the development tree. Production downloads are at:
https://github.com/simsong/bulk_extractor/releases
Other
1.11k stars 187 forks source link

As soon as 2.0 is stable, make main the main development branch. #161

Closed jonstewart closed 3 years ago

jonstewart commented 3 years ago

std::thread works great and is widely portable. Refactoring threadpool to use it should simplify the code greatly.

==EDIT==

Complete the following:

simsong commented 3 years ago

This work is already completed and then the slg-dev branch.

jonstewart commented 3 years ago

Ah, sorry, I was looking at the release-2.0-dev branch.

simsong commented 3 years ago

As soon as 2.0 is stable, make main the main development branch.

simsong commented 3 years ago

@jonstewart - can you give release-2.0-dev a try?

jonstewart commented 3 years ago

It builds successfully on my Mac. Two notes to add to build instructions from the git repo would be to run git submodules update --init --recursive (I forgot the recursive bit and got an error, but easy to suss out) and then of course to run ./bootstrap.sh.

Will play around with runs tonight.

jonstewart commented 3 years ago

No warnings either! Impressive.

simsong commented 3 years ago

It builds successfully on my Mac. Two notes to add to build instructions from the git repo would be to run git submodules update --init --recursive (I forgot the recursive bit and got an error, but easy to suss out) and then of course to run ./bootstrap.sh.

Will play around with runs tonight.

You're welcome to add a check to bootstrap.sh and have it give an error if the submodule is not checked out (or just do a git submodules update --init --recursive if it isn't).

jonstewart commented 3 years ago

I think it’s fine to make this the main branch. Compiled without a hitch and easy to run.

In the past I’ve always been particular about which scanners to run, but I thought it’d be good to test a default run. Unfortunately I had to suspend the job because I started to run out of disk space—a 28GB test image from SANS resulted in over 75GB of jpeg carves. Many of them were 20MB on the filesystem and I assume did not have an identifiable footer. I was just running on a laptop and didn’t think I’d need so much free space; not sure whether any change in behavior is needed, but it was a surprise nonetheless.

With the in-progress output, it wasn’t clear to me how much of the image had been processed and how much remained. V1 would also try to estimate a completion time, wouldn’t it?

I wonder with the in-progress output whether it would be a nicer user experience to update the same console lines rather than outputting new. I know there’s some trick to doing so, will look it up.

I’m on vacation for the week with poor internet access, but lots of free time. I will fix scan_lightgrep and submit a PR for it.

On Aug 11, 2021, at 7:28 PM, Simson L. Garfinkel @.***> wrote:

 It builds successfully on my Mac. Two notes to add to build instructions from the git repo would be to run git submodules update --init --recursive (I forgot the recursive bit and got an error, but easy to suss out) and then of course to run ./bootstrap.sh.

Will play around with runs tonight.

You're welcome to add a check to bootstrap.sh and have it give an error if the submodule is not checked out (or just do a git submodules update --init --recursive if it isn't).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

simsong commented 3 years ago

Unfortunately I had to suspend the job because I started to run out of disk space—a 28GB test image from SANS resulted in over 75GB of jpeg carves. Many of them were 20MB on the filesystem and I assume did not have an identifiable footer.

There is a bug in the jpeg carver: it's not trimming the JPEGs. See https://github.com/simsong/bulk_extractor/issues/199

Clearly, that needs to be fixed. Honestly, I want to get the path_printer operational and restart operational. I'll update the checklist at the top of this issue.

simsong commented 3 years ago

We're mostly there. @jonstewart - can you give it a try again?