Compile fails when exiv2 is not installed

[D3vil0p3r]

I'm trying to compile bulk_extractor on NixOS but I'm getting the following error during make:

g++ -std=c++17 -DHAVE_CONFIG_H -I. -I..  -I../src/be20_api -I../src/be20_api/utfcpp/source -DUTC_OFFSET=+0000  -g  -pthread -Wall -MD -Wpointer-arith -Wshadow -Wwrite-strings -Wcast-align -Wredundant-decls -Wdisabled-optimization -Wfloat-equal -Wmultichar -Wmissing-noreturn -Woverloaded-virtual -Wsign-promo -Weffc++ -fPIC -O3 -MT bulk_extractor.o -MD -MP -MF $depbase.Tpo -c -o bulk_extractor.o bulk_extractor.cpp &&\
mv -f $depbase.Tpo $depbase.Po
In file included from bulk_extractor.cpp:49:
bulk_extractor_restarter.h: In member function ‘void bulk_extractor_restarter::restart(Phase1::Config&, scanner_config&)’:
bulk_extractor_restarter.h:91:55: warning: declaration of ‘sc’ shadows a member of ‘bulk_extractor_restarter’ [-Wshadow]
   91 |     void restart(Phase1::Config &cfg, scanner_config &sc) {
      |                                       ~~~~~~~~~~~~~~~~^~
bulk_extractor_restarter.h:24:30: note: shadowed declaration is here
   24 |     scanner_config          ≻
      |                              ^~
bulk_extractor_restarter.h:91:34: warning: declaration of ‘cfg’ shadows a member of ‘bulk_extractor_restarter’ [-Wshadow]
   91 |     void restart(Phase1::Config &cfg, scanner_config &sc) {
      |                  ~~~~~~~~~~~~~~~~^~~
bulk_extractor_restarter.h:25:30: note: shadowed declaration is here
   25 |     Phase1::Config          &cfg;
      |                              ^~~
bulk_extractor.cpp: In function ‘int bulk_extractor_main(std::ostream&, std::ostream&, int, char* const*)’:
bulk_extractor.cpp:485:22: error: no matching function for call to ‘bulk_extractor_restarter::restart()’
  485 |             r.restart();                    // load the restart file and rename report.xml
      |             ~~~~~~~~~^~
bulk_extractor_restarter.h:91:10: note: candidate: ‘void bulk_extractor_restarter::restart(Phase1::Config&, scanner_config&)’
   91 |     void restart(Phase1::Config &cfg, scanner_config &sc) {
      |          ^~~~~~~
bulk_extractor_restarter.h:91:10: note:   candidate expects 2 arguments, 0 provided
make[2]: *** [Makefile:1377: bulk_extractor.o] Error 1
make[2]: Leaving directory '/home/athena/pkgs/bulk-extractor/bulk_extractor/src'
make[1]: *** [Makefile:411: all-recursive] Error 1
make[1]: Leaving directory '/home/athena/pkgs/bulk-extractor/bulk_extractor'
make: *** [Makefile:352: all] Error 2

[D3vil0p3r]

@simsong

[D3vil0p3r]

Solved by adding afflib and exiv2 as dependencies. It could be useful to have them in README or BUILD instructions.

[simsong]

Afflib is no longer supported and should not be used. Where are the dependencies listed that you did not find them? Exiv should automatically detect and if it is not present the configure script shouldn't have included it.

Are you compiling from sources? It would be nice to fix this so that it doesn't cause other people problems.

[D3vil0p3r]

I took the list of dependencies from this BlackArch PKGBUILD. I'm compiling from source by downloading the .tar.gz from https://digitalcorpora.s3.amazonaws.com/downloads/bulk_extractor/bulk_extractor-2.0.3.tar.gz

I did a further test. I tried to remove afflib dependency and the compilation still works. If I remove exiv2, I get the error reported above. So I think that exiv2, at least in NixOS, should be necessary.

Furthermore, for BEViewer, is there a specific java version to install? For example jdk11, jdk17 or jdk19?

[simsong]

exiv2 is not required, but if it is not there the jpeg module won’t decide them. So the question is why did autoconf think exiv was present even when it wasn’t.

Where should these requirements be documented?

---

On Sun, Dec 31, 2023 at 9:02 AM D3vil0p3r @.***> wrote:

I took the list of dependencies from this BlackArch PKGBUILD https://github.com/BlackArch/blackarch/blob/master/packages/bulk-extractor/PKGBUILD. I'm compiling from source by downloading the .tar.gz from https://digitalcorpora.s3.amazonaws.com/downloads/bulk_extractor/bulk_extractor-2.0.3.tar.gz

I did a further test. I tried to remove afflib dependency and the compilation still works. If I remove exiv2, I get the error reported above. So I think that exiv2, at least in NixOS, should be necessary.

— Reply to this email directly, view it on GitHub https://github.com/simsong/bulk_extractor/issues/433#issuecomment-1872955931, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMFHLAF6W4RD6ZLDO7YWRLYMFV6BAVCNFSM6AAAAABBIIN6AWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSHE2TKOJTGE . You are receiving this because you were mentioned.Message ID: @.***>

[D3vil0p3r]

My only references where that PKGBUILD I linked you before and this .nix file of an old version of bulk_extractor: https://github.com/NixOS/nixpkgs/pull/82265 . On your repository, I referred to https://github.com/simsong/bulk_extractor/blob/main/etc/CONFIGURE_FEDORA36.bash to know what are the dependencies to use.

[simsong]

But those files aren’t part of the bulk_extractor release.  I don’t have any control over those files. I don’t even know what NixOS is. I’ve never used it. You complained to me, and I am happy to fix this, but I don’t know how. Can you please provide me with a file that I can include in the release that would resolve your problem? If not, can you please file your bug report elsewhere?On Dec 31, 2023, at 9:11 AM, D3vil0p3r @.***> wrote: My only references where that PKGBUILD I linked you before and this .nix file of an old version of bulk_extractor: NixOS/nixpkgs#82265

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[jonstewart]

For [hopefully helpful] context, NixOS is a distro of Linux that uses the nix package manager for controlling the configuration of the system. The nix package manager constructs a Merkle tree of the software packages installed, where both content of source files and configuration options feed into the overall hash of the package. nix then uses an elaborate system of symlinks to set one's PATH and other environment variables so that the environment is somewhat normal. I've played with it a bit a couple years ago. It's neat but (then) was ultimately too frustrating for me to use. The benefit is that everything about the system is well-specified and easily replicated from a very small set of Haskell-based configuration scripts; builds are fully reproducible.

Given the degree to which nix buggers the normal Linux directory structure and environment, the inherent complexity therein, the number of dependencies of bulk_extractor, and the paleolithic technology that is autoconf, it's not surprising that there's a build problem with bulk_extractor on NixOS. Simson, you are correct that the focus should be on configure.ac and what its behavior is when exiv2 isn't found.

[simsong]

Yes, I'm concerned that it didn't work when exiv2 wasn't installed.

[D3vil0p3r]

Another small non-impacting note is that, at the end of building, also a test_be file is generated inside bin directory (accessible by PATH). Would be cleaner if test_be won't be automatically stored in bin directory?

[simsong]

Well, it's a binary. It just shouldn't be installed. Why is it getting installed on your system? It doesn't get installed on mine, the last time I checked.

[D3vil0p3r]

Well, it's a binary. It just shouldn't be installed. Why is it getting installed on your system? It doesn't get installed on mine, the last time I checked.

I'm investigating, I think it could be related how nix works. Is there a make config to prevent the creation of this test_be?

[simsong]

Looks like test_be is specified as a program and there is no install target. If you want to send a PR, I'm happy to take it. It may be that test_be doesn't need to be listed as a bin program if it is listed as a check program.

https://github.com/simsong/bulk_extractor/blob/22da4f4f36040e854cdf8a4900146574f25e35f2/src/Makefile.am#L1

On Sun, Dec 31, 2023 at 10:25 AM D3vil0p3r @.***> wrote:

Well, it's a binary. It just shouldn't be installed. Why is it getting installed on your system? It doesn't get installed on mine, the last time I checked.

I'm investigating, I think it could be related how nix works. Is there a make config to prevent the creation of this test_be?

— Reply to this email directly, view it on GitHub https://github.com/simsong/bulk_extractor/issues/433#issuecomment-1872975046, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMFHLBNEPEE3KPIXXPUIS3YMF7XDAVCNFSM6AAAAABBIIN6AWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSHE3TKMBUGY . You are receiving this because you were mentioned.Message ID: @.***>

--

---

[D3vil0p3r]

What I see in bulk_extractor/src folder after make command in the root repo folder is that, inside generated Makefile, we have:

EXEEXT = 
...
bin_PROGRAMS = bulk_extractor$(EXEEXT) test_be$(EXEEXT)
...
install-binPROGRAMS: $(bin_PROGRAMS)
        @$(NORMAL_INSTALL)
        @list='$(bin_PROGRAMS)'; test -n "$(bindir)" || list=; \
        if test -n "$$list"; then \
          echo " $(MKDIR_P) '$(DESTDIR)$(bindir)'"; \
          $(MKDIR_P) "$(DESTDIR)$(bindir)" || exit 1; \
        fi; \
        for p in $$list; do echo "$$p $$p"; done | \
        sed 's/$(EXEEXT)$$//' | \
        while read p p1; do if test -f $$p \
          ; then echo "$$p"; echo "$$p"; else :; fi; \
        done | \
        sed -e 'p;s,.*/,,;n;h' \
            -e 's|.*|.|' \
            -e 'p;x;s,.*/,,;s/$(EXEEXT)$$//;$(transform);s/$$/$(EXEEXT)/' | \
        sed 'N;N;N;s,\n, ,g' | \
        $(AWK) 'BEGIN { files["."] = ""; dirs["."] = 1 } \
          { d=$$3; if (dirs[d] != 1) { print "d", d; dirs[d] = 1 } \
            if ($$2 == $$4) files[d] = files[d] " " $$1; \
            else { print "f", $$3 "/" $$4, $$1; } } \
          END { for (d in files) print "f", d, files[d] }' | \
        while read type dir files; do \
            if test "$$dir" = .; then dir=; else dir=/$$dir; fi; \
            test -z "$$files" || { \
              echo " $(INSTALL_PROGRAM_ENV) $(INSTALL_PROGRAM) $$files '$(DESTDIR)$(bindir)$$dir'"; \
              $(INSTALL_PROGRAM_ENV) $(INSTALL_PROGRAM) $$files "$(DESTDIR)$(bindir)$$dir" || exit $$?; \
            } \
        ; done
...

so I guess that when sudo make install is run, the function install-binPROGRAMS is run and I think it stores test_be in dest bin dir.

[D3vil0p3r]

Since in Makefile, test_be is already specified as check_PROGRAMS, the only change should be to remove test_be from bin_PROGRAMS, right? If so, I can open a PR for it.

[simsong]

I don’t know. If you want to give it a try and, if that works, send me a PR, that would be great.On Dec 31, 2023, at 10:37 AM, D3vil0p3r @.***> wrote: Since in Makefile, test_be is already specified as check_PROGRAMS, the only change should be to remove test_be from bin_PROGRAMS, right?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[D3vil0p3r]

I tested and the change does not erroneously generate test_be in bin anymore. Here the PR: https://github.com/simsong/bulk_extractor/pull/434

Just for info, for this small PR will be a small update to a stable version like 2.0.4?

[simsong]

bulk_extractor_restarter.h:24:30: note: shadowed declaration is here

This will be fixed in an upcoming PR