search

simsong / bulk_extractor

This is the development tree. Production downloads are at:
https://github.com/simsong/bulk_extractor/releases
Other
1.11k stars 187 forks source link

Make bulk_extractor compile under Debian #452

Open simsong opened 9 months ago

simsong commented 9 months ago

@zdavatz -This is for you. @simsong will use this distribution: https://aws.amazon.com/marketplace/pp/prodview-fznsw3f7mq7to

zdavatz commented 9 months ago

Yes, also on Gentoo the compilation from source is not straight forward.

simsong commented 9 months ago

@zdavatz - Can you try out b1e500f?

zdavatz commented 9 months ago

sure! Can I just do a git pull for the latest update?

I done a git clone this time. Now I get:

~/.software/bulk_extractor> ./bootstrap.sh
submodule be20_api is not present.
simsong commented 9 months ago

When you do a git clone, you need to add the --recursive flag, because there are submodules.

Alternatively, you can do a git submodule init and git submodule update

zdavatz commented 9 months ago

I done that all, but still no luck:

~/.software/bulk_extractor> git submodule init
~/.software/bulk_extractor> git submodule update
Klone nach '/home/zeno/.software/bulk_extractor/dfxml_schema'...
Klone nach '/home/zeno/.software/bulk_extractor/src/be20_api'...
Submodul-Pfad 'dfxml_schema': 'f2a702e401dd15802ec8558b42c496a1184a0486' ausgecheckt
Submodul-Pfad 'src/be20_api': 'c031194542beed802c6303ac7c50904d10a5b67e' ausgecheckt
~/.software/bulk_extractor> ./bootstrap.sh
submodule be20_api/dfxml_cpp is not present.
zdavatz commented 9 months ago

Ok, this worked: git clone --recursive https://github.com/simsong/bulk_extractor.git

this did not work:

git submodule init
git submodule update
simsong commented 9 months ago

Hm. that has a sub module too. And then that sub module has another sub module. Which is why I recommend a git clone recursive. I think that the init and update commands also have recurse flags but that they are different.

On Sun, Jan 28, 2024 at 7:12 AM Zeno R.R. Davatz @.***> wrote:

I done that all, but still no luck:

~/.software/bulk_extractor> git submodule init ~/.software/bulk_extractor> git submodule update Klone nach '/home/zeno/.software/bulk_extractor/dfxml_schema'... Klone nach '/home/zeno/.software/bulk_extractor/src/be20_api'... Submodul-Pfad 'dfxml_schema': 'f2a702e401dd15802ec8558b42c496a1184a0486' ausgecheckt Submodul-Pfad 'src/be20_api': 'c031194542beed802c6303ac7c50904d10a5b67e' ausgecheckt ~/.software/bulk_extractor> ./bootstrap.sh submodule be20_api/dfxml_cpp is not present.

— Reply to this email directly, view it on GitHub https://github.com/simsong/bulk_extractor/issues/452#issuecomment-1913575572, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMFHLFAAI3GAS34EYRUZ63YQY6BDAVCNFSM6AAAAABCLPEOR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJTGU3TKNJXGI . You are receiving this because you were assigned.Message ID: @.***>

zdavatz commented 9 months ago

Ok, configure goes through, but make -j9 still fails.

 495 |         if ( std::filesystem::exists( sc.outdir/"report.xml" )){
      |                   ^~~~~~~~~~
bulk_extractor.cpp:495:42: Fehler: »struct scanner_config« hat kein Element namens »outdir«
  495 |         if ( std::filesystem::exists( sc.outdir/"report.xml" )){
      |                                          ^~~~~~
bulk_extractor.cpp:504:37: Fehler: »struct scanner_config« hat kein Element namens »input_fname«
  504 |         p = image_process::open( sc.input_fname, cfg.opt_recurse, cfg.opt_pagesize, cfg.opt_marginsize );
      |                                     ^~~~~~~~~~~
bulk_extractor.cpp:504:104: Fehler: »image_process::open« kann nicht als Funktion verwendet werden
  504 |         p = image_process::open( sc.input_fname, cfg.opt_recurse, cfg.opt_pagesize, cfg.opt_marginsize );
      |                                                                                                        ^
bulk_extractor.cpp:507:73: Fehler: »struct scanner_config« hat kein Element namens »input_fname«
  507 |         cerr << "error: file " << e.what() << " is in directory " << sc.input_fname << std::endl;
      |                                                                         ^~~~~~~~~~~
bulk_extractor.cpp:511:71: Fehler: »struct scanner_config« hat kein Element namens »input_fname«
  511 |         cerr << "       then place them in a sub directory of " << sc.input_fname << std::endl;
      |                                                                       ^~~~~~~~~~~
bulk_extractor.cpp:539:50: Fehler: »struct scanner_config« hat kein Element namens »outdir«
  539 |     dfxml_writer *xreport = new dfxml_writer( sc.outdir / Phase1::REPORT_FILENAME, false ); // do not make DTD
      |                                                  ^~~~~~
bulk_extractor.cpp:547:23: Fehler: »struct scanner_config« hat kein Element namens »input_fname«
  547 |     validate_path( sc.input_fname );
      |                       ^~~~~~~~~~~
bulk_extractor.cpp:547:5: Fehler: »validate_path« wurde in diesem Gültigkeitsbereich nicht definiert
  547 |     validate_path( sc.input_fname );
      |     ^~~~~~~~~~~~~
bulk_extractor.cpp:565:38: Fehler: »struct scanner_config« hat kein Element namens »input_fname«
  565 |         cout << "Input file: " << sc.input_fname << std::endl ;
      |                                      ^~~~~~~~~~~
bulk_extractor.cpp:566:44: Fehler: »struct scanner_config« hat kein Element namens »outdir«
  566 |         cout << "Output directory: " << sc.outdir << std::endl ;
      |                                            ^~~~~~
bulk_extractor.cpp:600:46: Fehler: »struct scanner_config« hat kein Element namens »input_fname«
  600 |     xreport->xmlout( "provided_filename", sc.input_fname ); // save this information
      |                                              ^~~~~~~~~~~
make[2]: *** [Makefile:1438: scan_aes.o] Fehler 1
make[2]: *** [Makefile:1438: bulk_extractor_scanners.o] Fehler 1
make[2]: *** [Makefile:1438: notify_thread.o] Fehler 1
make[2]: *** [Makefile:1438: bulk_extractor.o] Fehler 1
make[2]: Verzeichnis „/home/zeno/.software/bulk_extractor/src“ wird verlassen
make[1]: *** [Makefile:526: all-recursive] Fehler 1
make[1]: Verzeichnis „/home/zeno/.software/bulk_extractor“ wird verlassen
make: *** [Makefile:466: all] Fehler 2
zdavatz commented 9 months ago

@zdavatz - Can you try out b1e500f?

this commit is in the latest git pull right?

My repo is here: 88c2d72efcc92776075f2a18855e2d518ccfd3f1

zdavatz commented 9 months ago

I do not have libewf installed, but AFAICT that should not be a problem.

simsong commented 9 months ago

Correct, you should not need libewf.

It appears that git submodule update has a --recursive flag but git init does not. So you will need to use the git submodule foreach --recursive git init command (or something like that in addition to git init if you do not wish to do a new clone with git clone --recursive, which is what I recommend.

zdavatz commented 9 months ago

ok, git clone --recursive works but I still get a make error: https://github.com/simsong/bulk_extractor/issues/452#issuecomment-1913577328

simsong commented 9 months ago

Try switching to the branch rel-2.1.1 ?

simsong commented 9 months ago

Can you post your full config.log file? I can try to spin up a Kali Linux somewhere. Is it available on AWS? Is there a way to run Kali linux on Github actions? Kali Linux is based on Debian. There is no github actions running that provides debian linux. Do you know if I can run it under docker?

zdavatz commented 9 months ago

Can you post your full config.log file? I can try to spin up a Kali Linux somewhere. Is it available on AWS? Is there a way to run Kali linux on Github actions? Kali Linux is based on Debian. There is no github actions running that provides debian linux. Do you know if I can run it under docker?

you can definitely do a github actions build from source. Config log is attached.

config.log

Configure runs fine for me.

simsong commented 9 months ago

I have GitHub actions that build from source, but the GitHub runners are Ubuntu, not Debian. Do you have an example of setting up Debian with docker to run an action?

On Mon, Jan 29, 2024 at 2:57 AM Zeno R.R. Davatz @.***> wrote:

Can you post your full config.log file? I can try to spin up a Kali Linux somewhere. Is it available on AWS? Is there a way to run Kali linux on Github actions? Kali Linux is based on Debian. There is no github actions running that provides debian linux. Do you know if I can run it under docker?

you can definitely do a github actions build from source. Config log is attached.

config.log https://github.com/simsong/bulk_extractor/files/14081257/config.log

Configure runs fine for me.

— Reply to this email directly, view it on GitHub https://github.com/simsong/bulk_extractor/issues/452#issuecomment-1914148405, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMFHLF5IYYWZZACUHTH7OTYQ5I7HAVCNFSM6AAAAABCLPEOR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJUGE2DQNBQGU . You are receiving this because you were assigned.Message ID: @.***>

zdavatz commented 9 months ago

does this help: https://github.com/sdolenc/debian-matrix/blob/master/.circleci/config.yml https://github.com/sdolenc/debian-matrix

simsong commented 9 months ago

Thanks. It does. I looked through your config file. You are compiling with C++17. Nothing obviously is wrong. I'll give this a whirl and see what happens.

zdavatz commented 9 months ago

Thank you!

simsong commented 9 months ago

I'm trying to compile under a debian container and I can't get that to work either. This is a deeper problem. Somebody who knows Debian or Kali Linux, which is based on Debian, needs to fix this.

zdavatz commented 9 months ago

On which Linux do you normally build?

simsong commented 9 months ago

We build on both Ubuntu with amazon actions. Fedora to build the windows executable. Also Amazon Linux. Debian has always been a mystery to me.

On Tue, Jan 30, 2024 at 1:49 AM Zeno R.R. Davatz @.***> wrote:

On which Linux do you normally build?

— Reply to this email directly, view it on GitHub https://github.com/simsong/bulk_extractor/issues/452#issuecomment-1916183043, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMFHLEU5SPRKC5BJBO7KALYRCJYDAVCNFSM6AAAAABCLPEOR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJWGE4DGMBUGM . You are receiving this because you were assigned.Message ID: @.***>