simsong / bulk_extractor

This is the development tree. Production downloads are at:
https://github.com/simsong/bulk_extractor/releases
Other
1.04k stars 184 forks source link

`error: no matching function for call to 'bulk_extractor_restarter::restart()` #391

Closed zdavatz closed 1 year ago

zdavatz commented 1 year ago
  1. git clone https://github.com/simsong/bulk_extractor.git
  2. git submodule update --init --recursive
  3. ./configure runs fine.
  4. Running make results in
                              ^~~
    bulk_extractor.cpp: In function 'int bulk_extractor_main(std::ostream&, std::ostream&, int, char* const*)':
    bulk_extractor.cpp:486:23: error: no matching function for call to 'bulk_extractor_restarter::restart()'
             r.restart();                    // load the restart file and rename report.xml
                       ^
    In file included from bulk_extractor.cpp:49:
simsong commented 1 year ago

This has been reported before. You don't have the expat-dev library installed. How did you configure the VM? It's listed as one of the requirements.

What platform are you compiling on?

zdavatz commented 1 year ago

I am compiling on Oracle Linux with AMD EPYC 7551 32-Core Processor. Super great tool btw 🙏🙏

zdavatz commented 1 year ago

that seems to have worked. Next error:

image_process.cpp: In member function 'virtual int process_ewf::open()':
image_process.cpp:154:54: error: cannot convert 'const value_type*' {aka 'const char*'} to 'const wchar_t*'
     if (LIBEWF_GLOB(fname.c_str(), STRLEN(fname.c_str()), LIBEWF_FORMAT_UNKNOWN,
                                           ~~~~~~~~~~~^~
In file included from /usr/include/c++/8/cwchar:44,
                 from /usr/include/c++/8/bits/postypes.h:40,
                 from /usr/include/c++/8/bits/char_traits.h:40,
                 from /usr/include/c++/8/string:40,
                 from /usr/include/c++/8/stdexcept:39,
                 from image_process.cpp:33:
/usr/include/wchar.h:222:38: note:   initializing argument 1 of 'size_t wcslen(const wchar_t*)'
 extern size_t wcslen (const wchar_t *__s) __THROW __attribute_pure__;
                       ~~~~~~~~~~~~~~~^~~
make[2]: *** [Makefile:1379: image_process.o] Error 1
make[2]: *** Waiting for unfinished jobs....

libewf-devel and libewf are both installed.

simsong commented 1 year ago

How did you install libewf? How did you prepare the VM? Did you start with a clean VM and use one of our configure scripts? The libewf that you get from the package system is very out of date.

Are you cross-compiling?

zdavatz commented 1 year ago
  1. sudo yum install libewf
  2. No I am not cross-compiling.
  3. I started with a clean server.
  4. I installed all the missing stuff ./configure was complaining about.
  5. Which configure script do you recommend to use for Oracle Linux?
simsong commented 1 year ago

Any luck?


From: Zeno R.R. Davatz @.> Sent: Monday, March 13, 2023 3:06 PM To: simsong/bulk_extractor @.> Cc: @. @.>; Comment @.***> Subject: Re: [simsong/bulk_extractor] error: no matching function for call to 'bulk_extractor_restarter::restart() (Issue #391)

  1. sudo yum install libewf
  2. No I am not cross-compiling.
  3. I started with a clean server.

— Reply to this email directly, view it on GitHubhttps://github.com/simsong/bulk_extractor/issues/391#issuecomment-1466791484, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAMFHLAX3VKVMIVLZ25OEYLW35V3XANCNFSM6AAAAAAVZOGAJM. You are receiving this because you commented.Message ID: @.***>

zdavatz commented 1 year ago

No, so far not yet, thank you for asking.

simsong commented 1 year ago

Did you try installing libewf from source using the VM prep script, as is recommended?


Sent from my phone.

On Mar 15, 2023, at 1:39 PM, Zeno R.R. Davatz @.***> wrote:



No, so far not yet, thank you for asking.

— Reply to this email directly, view it on GitHubhttps://github.com/simsong/bulk_extractor/issues/391#issuecomment-1470469277, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAMFHLCQHLJRL3RTOAWLEK3W4H427ANCNFSM6AAAAAAVZOGAJM. You are receiving this because you commented.Message ID: @.***>

zdavatz commented 1 year ago

where is the documentation for those steps?

simsong commented 1 year ago

It looks like your system is similar to Fedora, so you should be able to modify this script to work. When you do, please send me a PR so I can add it to the repo:

https://github.com/simsong/bulk_extractor/blob/main/etc/CONFIGURE_FEDORA36.bash

Separately, I should make bulk_extractor work with the libewf you get from sudo yum install libewf. However, @joachimmetz has repeatedly told me that this is an old version that is buggy and quite out-of-date, so I have been hesitant to make it work.

zdavatz commented 1 year ago

Yes, Oracle Linux is a fork of Red Had linux, without the Red Hat logos ;)

zdavatz commented 1 year ago

https://github.com/simsong/bulk_extractor/blob/main/etc/CONFIGURE_FEDORA36.bash

this script does not show the download URL, AFAICT.

I found this: https://github.com/libyal/libewf

but ./configure does not work there.

Yes, it would be great if libewf installed via apt-get install or yum install would just work out of the box.

joachimmetz commented 1 year ago

but ./configure does not work there.

have a look at https://github.com/libyal/libewf/wiki/Building#read-first

Yes, it would be great if libewf installed via apt-get install or yum install would just work out of the box.

you need to reach out to the maintainer of the distro you are using, the upstream project has no influence on this

joachimmetz commented 1 year ago

this script does not show the download URL, AFAICT.

Have a look at https://github.com/simsong/bulk_extractor/blob/main/etc/paths.bash

which gets included here https://github.com/simsong/bulk_extractor/blob/main/etc/CONFIGURE_FEDORA36.bash#L2

zdavatz commented 1 year ago

have a look at https://github.com/libyal/libewf/wiki/Building#read-first

Yes thank you! Some steps to do not work: https://github.com/libyal/libewf/issues/184

zdavatz commented 1 year ago

this script does not show the download URL, AFAICT.

Have a look at https://github.com/simsong/bulk_extractor/blob/main/etc/paths.bash

but this uses a Release from 2014? Why not just make the version from yum install libewf work?

which gets included here https://github.com/simsong/bulk_extractor/blob/main/etc/CONFIGURE_FEDORA36.bash#L2

Thank you!

joachimmetz commented 1 year ago

but this uses a Release from 2014?

Have a look at the actual date of the release, you are conflating the release version (which looks like a date) with an actual date

zdavatz commented 1 year ago

Separately, I should make bulk_extractor work with the libewf you get from sudo yum install libewf.

Yes, that would be great!

joachimmetz commented 1 year ago

@zdavatz you can help by filing an issue with your distro maintainers to have them update libewf to the latest stable version. So that this project can build against the latest version provided by your distro.

simsong commented 1 year ago

Separately, I should make bulk_extractor work with the libewf you get from sudo yum install libewf.

Yes, that would be great!

Indeed. I'm really torn here. The problem is that the version you get with sudo yum install libewf is so out of date and buggy that you might get a bad outcome with it. What do you think? Would you rather have it compile, link, and produce erroneous results, perhaps silently? Or would you rather know of the problem?

joachimmetz commented 1 year ago

the issue is that the same issue applies for other distros, here Dockerizing can help, this is an easy way to provide an up to date version even if the distro does not provide it

zdavatz commented 1 year ago

Separately, I should make bulk_extractor work with the libewf you get from sudo yum install libewf.

Yes, that would be great!

Indeed. I'm really torn here. The problem is that the version you get with sudo yum install libewf is so out of date and buggy that you might get a bad outcome with it. What do you think? Would you rather have it compile, link, and produce erroneous results, perhaps silently? Or would you rather know of the problem?

Oracle Linux gives me this:

sudo yum list installed | grep ewf
libewf.x86_64                                    20140608-16.el8                               @epel             
libewf-devel.x86_64                              20140608-16.el8                               @epel    

Kali gives me this:

└─$ apt list libewf2    
Listing... Done
libewf2/kali-rolling,now 20140813-1+b1 amd64 [installed,automatic]

┌──(zeno㉿kali)-[~]
└─$ apt list libewf-dev
Listing... Done
libewf-dev/kali-rolling 20140813-1+b1 amd64

┌──(zeno㉿kali)-[~]
└─$ apt list ewf-tools 
Listing... Done
ewf-tools/kali-rolling 20140813-1+b1 amd64

not really far apart. What really confuses me is that both run on 2014 version.

joachimmetz commented 1 year ago

2014xxxx version, is correct here are the versions and release dates https://github.com/libyal/libewf-legacy/releases

zdavatz commented 1 year ago

On Kali Linux bulk_extractor works out of the box, no problem.

zdavatz commented 1 year ago

2014xxxx version, is correct here are the versions and release dates https://github.com/libyal/libewf-legacy/releases

Ok, but then nothing speaks against using the yum install libewf-dev libewf packages.

zdavatz commented 1 year ago

Separately, I should make bulk_extractor work with the libewf you get from sudo yum install libewf. However, @joachimmetz has repeatedly told me that this is an old version that is buggy and quite out-of-date, so I have been hesitant to make it work.

Why are you saying it is an old version then, @joachimmetz ?

joachimmetz commented 1 year ago

Ok, but then nothing speaks against using the yum install libewf-dev libewf packages. Why are you saying it is an old version then, @joachimmetz ?

Do you see 20140608 on https://github.com/libyal/libewf-legacy/releases ?

simsong commented 1 year ago

@zdavatz - this may be the source of your confusion: image

Specifically, the July 16, 2022 release is a bugfix to version 20140814. The version number did not increment with the bugfix. This release of libewf may not use a version number system with which you are familiar.

joachimmetz commented 1 year ago

that's what i told you earlier

Have a look at the actual date of the release, you are conflating the release version (which looks like a date) with an actual date
joachimmetz commented 1 year ago

And 20140608 is very likely from 2014 before the move to Github and work on the new experimental version started.

simsong commented 1 year ago

2014xxxx version, is correct here are the versions and release dates https://github.com/libyal/libewf-legacy/releases

Ok, but then nothing speaks against using the yum install libewf-dev libewf packages.

Hi @zdavatz ---- Your command yum install libewf-dev libewf has different behavior on different operating systems with different configuration files running in different data centers and is dependent upon various environment variables that you may have set. In your case, this command does not produce the correct install because the maintainer of your particular variant of Linux has not updated the version of libewf in their repo. This is why @joachimmetz recommended that you speak with the maintainers of your version of Linux.

simsong commented 1 year ago

On Kali Linux bulk_extractor works out of the box, no problem.

Which version do you get?

joachimmetz commented 1 year ago

from a previous comment looks like Kali ships with 20140813, which is from 2022

simsong commented 1 year ago

I thought OP meant that it came with bulk_extractor pre-installed.


Sent from my phone.

On Mar 16, 2023, at 7:21 AM, Joachim Metz @.***> wrote:



from a previous comment looks like Kali ships with 20140813, which is from 2022

— Reply to this email directly, view it on GitHubhttps://github.com/simsong/bulk_extractor/issues/391#issuecomment-1471768546, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAMFHLDDNSY35LZTGFUXY33W4LZUBANCNFSM6AAAAAAVZOGAJM. You are receiving this because you commented.Message ID: @.***>

joachimmetz commented 1 year ago

I thought OP meant that it came with bulk_extractor pre-installed.

Ah in the sense of which bulk_extractor version.

zdavatz commented 1 year ago

On Kali Linux bulk_extractor works out of the box, no problem. Which version do you get?

lsb_release -r No LSB modules are available. Release: 2022.4

Kali Linux Packet management is really well organised.

zdavatz commented 1 year ago

@zdavatz - this may be the source of your confusion: image

Specifically, the July 16, 2022 release is a bugfix to version 20140814. The version number did not increment with the bugfix.

this is indeed very confusing!

joachimmetz commented 1 year ago

this is indeed very confusing!

it is very simple, these are release numbers nothing more, nothing less. I don't understand what is hard to grasp, it is nothing different than 1, 2, 3, ... or 1.0, 1.1, 1.2, ...

zdavatz commented 1 year ago

this is indeed very confusing!

it is very simple, these are release numbers nothing more, nothing less. I don't understand what is hard to grasp, it is nothing different than 1, 2, 3, ... or 1.0, 1.1, 1.2, ...

I could be Year 2014, August 14th ;)

zdavatz commented 1 year ago

The Oracle Linux Version I am using is: Oracle Linux Server release 8.7

zdavatz commented 1 year ago

And 20140608 is very likely from 2014 before the move to Github and work on the new experimental version started.

Ok, so the Fedora, Cent OS packages are also out-of-date?

joachimmetz commented 1 year ago

Ok, so the Fedora, Cent OS packages are also out-of-date?

Is very well possible

zdavatz commented 1 year ago

Ok, but then nothing speaks against using the yum install libewf-dev libewf packages. Why are you saying it is an old version then, @joachimmetz ?

Do you see 20140608 on https://github.com/libyal/libewf-legacy/releases ?

I do seet it there yes.

Why do you create a rpm Release with the latest version and list it on your github-Release page? For example create a .rmp, a .deb and .tar.gz package that can be installed without having to be compiled?

joachimmetz commented 1 year ago

Why do you create a rpm Release with the latest version and list it on your github-Release page?

(1) because I don't have the time to maintain binaries (2) because this assumes an rpm or deb for every distro/architecture is the same (which is not the case).

As I indicated before, you don't seem to fully understand the ecosystem/problem space

simsong commented 1 year ago

I meant which version of bulk extractor….


Sent from my phone.

On Mar 16, 2023, at 9:36 AM, Zeno R.R. Davatz @.***> wrote:



On Kali Linux bulk_extractor works out of the box, no problem. Which version do you get?

lsb_release -r No LSB modules are available. Release: 2022.4

— Reply to this email directly, view it on GitHubhttps://github.com/simsong/bulk_extractor/issues/391#issuecomment-1471966951, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAMFHLGBRYB4KI4AYNV2PBLW4MJM3ANCNFSM6AAAAAAVZOGAJM. You are receiving this because you commented.Message ID: @.***>

zdavatz commented 1 year ago

Why do you create a rpm Release with the latest version and list it on your github-Release page?

(1) because I don't have the time to maintain binaries (2) because this assumes an rpm or deb for every distro/architecture is the same (which is not the case).

As I indicated before, you don't seem to fully understand the ecosystem/problem space

this version libewf-experimental-20230212.tar.gz seems to

./configure make -j3 sudo make install

without a problem. Trying it now on Oracle Linux.

zdavatz commented 1 year ago

I meant which version of bulk extractor….

bulk_extractor 2.0.0

zdavatz commented 1 year ago

Ok, I think I am past the libewf problem now @simsong

make -j3 is now hanging here on my Oracle box:

/usr/include/c++/8/bits/fs_ops.h:147: undefined reference to `std::filesystem::status(std::filesystem::__cxx11::path const&)'
image_process.o: In function `process_raw::add_file(std::filesystem::__cxx11::path)':
/home/opc/software/bulk_extractor/src/image_process.cpp:471: undefined reference to `std::filesystem::file_size(std::filesystem::__cxx11::path const&)'
image_process.o: In function `process_raw::open()':
/usr/include/c++/8/bits/fs_path.h:185: undefined reference to `std::filesystem::__cxx11::path::_M_split_cmpts()'
/usr/include/c++/8/bits/fs_path.h:185: undefined reference to `std::filesystem::__cxx11::path::_M_split_cmpts()'
image_process.o: In function `process_raw::open()':
/usr/include/c++/8/bits/fs_ops.h:121: undefined reference to `std::filesystem::status(std::filesystem::__cxx11::path const&)'
image_process.o: In function `process_raw::open()':
/usr/include/c++/8/bits/fs_path.h:185: undefined reference to `std::filesystem::__cxx11::path::_M_split_cmpts()'
image_process.o: In function `process_dir::process_dir(std::filesystem::__cxx11::path)':
/home/opc/software/bulk_extractor/src/image_process.cpp:716: undefined reference to `std::filesystem::__cxx11::recursive_directory_iterator::~recursive_directory_iterator()'
/home/opc/software/bulk_extractor/src/image_process.cpp:716: undefined reference to `std::filesystem::__cxx11::recursive_directory_iterator::~recursive_directory_iterator()'
/home/opc/software/bulk_extractor/src/image_process.cpp:716: undefined reference to `std::filesystem::__cxx11::recursive_directory_iterator::~recursive_directory_iterator()'
pcap_writer.o: In function `pcap_writer::pcap_writer(scanner_params const&)':
/usr/include/c++/8/bits/fs_path.h:185: undefined reference to `std::filesystem::__cxx11::path::_M_split_cmpts()'
/usr/include/c++/8/bits/fs_path.h:381: undefined reference to `std::filesystem::__cxx11::path::has_root_directory() const'
/usr/include/c++/8/bits/fs_path.h:258: undefined reference to `std::filesystem::__cxx11::path::has_filename() const'
/usr/include/c++/8/bits/fs_path.h:261: undefined reference to `std::filesystem::__cxx11::path::_M_split_cmpts()'
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:1230: bulk_extractor] Error 1
make[2]: *** Waiting for unfinished jobs....

moving this over to: https://github.com/simsong/bulk_extractor/issues/392