make clean does not delete neon_simd.o

lordadamson commented 3 years ago

Trying to cross-compile to iOS on a macOS

State the operating system and version (Ubutnu 17 x86_64, Windows 7 Professional x64, etc)

macOS 11.1 Big Sur

State the version of the Crypto++ library (Crypto++ 7.0, Master, etc)

Master

State how you built the library (Makefile, Cmake, distro, etc)

export IOS_SDK=iPhoneOS
export IOS_CPU=arm64

source TestScripts/setenv-ios.sh

export CXXFLAGS="-DNDEBUG -g2 -O3 -fPIC -pipe -fembed-bitcode"
make -f GNUmakefile-cross

Show a typical command line (the output of the compiler for cryptlib.cpp)

clang++ -DNDEBUG -g2 -O3 -fPIC -pipe -fembed-bitcode -Wall -arch arm64 -miphoneos-version-min=6 -stdlib=libc++ --sysroot "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.2.sdk" -c cryptlib.cpp

Show the link command (the output of the linker for libcryptopp.so or cryptest.exe)

This is where I discovered the issue because neon_simd.o was an old version from a previous build

libtool -static -o libcryptopp.a cryptlib.o cpu.o integer.o 3way.o adler32.o algebra.o algparam.o allocate.o arc4.o aria.o aria_simd.o ariatab.o asn.o authenc.o base32.o base64.o basecode.o bfinit.o blake2.o blake2b_simd.o blake2s_simd.o blowfish.o blumshub.o camellia.o cast.o casts.o cbcmac.o ccm.o chacha.o chacha_avx.o chacha_simd.o chachapoly.o cham.o cham_simd.o channels.o cmac.o crc.o crc_simd.o darn.o default.o des.o dessp.o dh.o dh2.o dll.o donna_32.o donna_64.o donna_sse.o dsa.o eax.o ec2n.o eccrypto.o ecp.o elgamal.o emsa2.o eprecomp.o esign.o files.o filters.o fips140.o fipstest.o gcm.o gcm_simd.o gf256.o gf2_32.o gf2n.o gf2n_simd.o gfpcrypt.o gost.o gzip.o hc128.o hc256.o hex.o hight.o hmac.o hrtimer.o ida.o idea.o iterhash.o kalyna.o kalynatab.o keccak.o keccak_core.o keccak_simd.o lea.o lea_simd.o luc.o mars.o marss.o md2.o md4.o md5.o misc.o modes.o mqueue.o mqv.o nbtheory.o neon_simd.o oaep.o osrng.o padlkrng.o panama.o pkcspad.o poly1305.o polynomi.o pssr.o pubkey.o queue.o rabbit.o rabin.o randpool.o rc2.o rc5.o rc6.o rdrand.o rdtables.o rijndael.o rijndael_simd.o ripemd.o rng.o rsa.o rw.o safer.o salsa.o scrypt.o seal.o seed.o serpent.o sha.o sha3.o sha_simd.o shacal2.o shacal2_simd.o shake.o shark.o sharkbox.o simeck.o simon.o simon128_simd.o skipjack.o sm3.o sm4.o sm4_simd.o sosemanuk.o speck.o speck128_simd.o square.o squaretb.o strciphr.o tea.o tftables.o threefish.o tiger.o tigertab.o ttmac.o tweetnacl.o twofish.o vmac.o wake.o whrlpool.o xed25519.o xtr.o xtrcrypt.o xts.o zdeflate.o zinflate.o zlib.o

Show the exact error message you are receiving (copy and paste it); or

error: libtool: file: neon_simd.o is not an object file (not allowed in a library)
make: *** [libcryptopp.a] Error 1

Clearly state the undesired behavior (and state the expected behavior)

If I do a make clean like this:

make -f GNUmakefile-cross clean
make clean

it won't delete neon_simd.o

Which leads to all sorts of confusing errors while linking because neon_simd.o is from a previous build.

I understand basic makefiles but I actually don't know how to fix this one because the make file is too advanced and complicated for me lol

If you direct me to where the issue might be I could open a PR with a fix.

For anyone looking for an immediate fix: just delete neon_simd.o and build again.

noloader commented 3 years ago

If I do a make clean like this:
make -f GNUmakefile-cross clean
make clean
it won't delete neon_simd.o

That's weird. What are the permissions on neon_simd.o? ls -Al neon_simd.* should show them.

This is where I discovered the issue because neon_simd.o was an old version from a previous build

So to reproduce, build iPhone/armv7, then make distclean, and then build iPhone/armv8?

noloader commented 3 years ago

This is a related check-in for GNUmakefile-cross: commit a783d2bf0742.

It fixes a missing CPPFLAGS that some folks may use. It does not fix your issue, however.

lordadamson commented 3 years ago

➜  cryptopp git:(master) ✗ ls -Al neon_simd.*
-rw-r--r--  1 adam  staff   5521 Dec 31 14:47 neon_simd.cpp
-rw-r--r--  1 adam  staff  54132 Jan  1 17:56 neon_simd.o

But it's not different from any other object file. For example:

➜  cryptopp git:(master) ✗ ls -Al integer.*
-rw-r--r--  1 adam  staff   133752 Dec 31 14:47 integer.cpp
-rw-r--r--  1 adam  staff    38699 Dec 31 14:47 integer.h
-rw-r--r--  1 adam  staff  3762732 Jan  1 17:54 integer.o

Here's the output of the clean commands:

➜  cryptopp git:(master) ✗ make -f GNUmakefile-cross clean
Here's what we found... IS_X86: 0, IS_X64: 0, IS_ARM32: 0, IS_ARMV8: 0

rm -f adhoc.cpp.o adhoc.cpp.proto.o cryptlib.o cpu.o integer.o 3way.o adler32.o algebra.o algparam.o allocate.o arc4.o aria.o aria_simd.o ariatab.o asn.o authenc.o base32.o base64.o basecode.o bfinit.o blake2.o blake2b_simd.o blake2s_simd.o blowfish.o blumshub.o camellia.o cast.o casts.o cbcmac.o ccm.o chacha.o chacha_avx.o chacha_simd.o chachapoly.o cham.o cham_simd.o channels.o cmac.o crc.o crc_simd.o darn.o default.o des.o dessp.o dh.o dh2.o dll.o donna_32.o donna_64.o donna_sse.o dsa.o eax.o ec2n.o eccrypto.o ecp.o elgamal.o emsa2.o eprecomp.o esign.o files.o filters.o fips140.o fipstest.o gcm.o gcm_simd.o gf256.o gf2_32.o gf2n.o gf2n_simd.o gfpcrypt.o gost.o gzip.o hc128.o hc256.o hex.o hight.o hmac.o hrtimer.o ida.o idea.o iterhash.o kalyna.o kalynatab.o keccak.o keccak_core.o keccak_simd.o lea.o lea_simd.o luc.o mars.o marss.o md2.o md4.o md5.o misc.o modes.o mqueue.o mqv.o nbtheory.o oaep.o osrng.o padlkrng.o panama.o pkcspad.o poly1305.o polynomi.o pssr.o pubkey.o queue.o rabbit.o rabin.o randpool.o rc2.o rc5.o rc6.o rdrand.o rdtables.o rijndael.o rijndael_simd.o ripemd.o rng.o rsa.o rw.o safer.o salsa.o scrypt.o seal.o seed.o serpent.o sha.o sha3.o sha_simd.o shacal2.o shacal2_simd.o shake.o shark.o sharkbox.o simeck.o simon.o simon128_simd.o skipjack.o sm3.o sm4.o sm4_simd.o sosemanuk.o speck.o speck128_simd.o square.o squaretb.o strciphr.o tea.o tftables.o threefish.o tiger.o tigertab.o ttmac.o tweetnacl.o twofish.o vmac.o wake.o whrlpool.o xed25519.o xtr.o xtrcrypt.o xts.o zdeflate.o zinflate.o zlib.o  rdrand-*.o adhoc.o test.o bench1.o bench2.o bench3.o datatest.o dlltest.o fipsalgt.o validat0.o validat1.o validat2.o validat3.o validat4.o validat5.o validat6.o validat7.o validat8.o validat9.o validat10.o regtest1.o regtest2.o regtest3.o regtest4.o

➜  cryptopp git:(master) ✗ make clean
rm -f adhoc.cpp.o adhoc.cpp.proto.o cryptlib.o cpu.o integer.o 3way.o adler32.o algebra.o algparam.o allocate.o arc4.o aria.o aria_simd.o ariatab.o asn.o authenc.o base32.o base64.o basecode.o bfinit.o blake2.o blake2b_simd.o blake2s_simd.o blowfish.o blumshub.o camellia.o cast.o casts.o cbcmac.o ccm.o chacha.o chacha_avx.o chacha_simd.o chachapoly.o cham.o cham_simd.o channels.o cmac.o crc.o crc_simd.o darn.o default.o des.o dessp.o dh.o dh2.o dll.o donna_32.o donna_64.o donna_sse.o dsa.o eax.o ec2n.o eccrypto.o ecp.o elgamal.o emsa2.o eprecomp.o esign.o files.o filters.o fips140.o fipstest.o gcm.o gcm_simd.o gf256.o gf2_32.o gf2n.o gf2n_simd.o gfpcrypt.o gost.o gzip.o hc128.o hc256.o hex.o hight.o hmac.o hrtimer.o ida.o idea.o iterhash.o kalyna.o kalynatab.o keccak.o keccak_core.o keccak_simd.o lea.o lea_simd.o luc.o mars.o marss.o md2.o md4.o md5.o misc.o modes.o mqueue.o mqv.o nbtheory.o oaep.o osrng.o padlkrng.o panama.o pkcspad.o poly1305.o polynomi.o pssr.o pubkey.o queue.o rabbit.o rabin.o randpool.o rc2.o rc5.o rc6.o rdrand.o rdtables.o rijndael.o rijndael_simd.o ripemd.o rng.o rsa.o rw.o safer.o salsa.o scrypt.o seal.o seed.o serpent.o sha.o sha3.o sha_simd.o shacal2.o shacal2_simd.o shake.o shark.o sharkbox.o simeck.o simon.o simon128_simd.o skipjack.o sm3.o sm4.o sm4_simd.o sosemanuk.o speck.o speck128_simd.o square.o squaretb.o sse_simd.o strciphr.o tea.o tftables.o threefish.o tiger.o tigertab.o ttmac.o tweetnacl.o twofish.o vmac.o wake.o whrlpool.o xed25519.o xtr.o xtrcrypt.o xts.o zdeflate.o zinflate.o zlib.o rdrand-*.o adhoc.o test.o bench1.o bench2.o bench3.o datatest.o dlltest.o fipsalgt.o validat0.o validat1.o validat2.o validat3.o validat4.o validat5.o validat6.o validat7.o validat8.o validat9.o validat10.o regtest1.o regtest2.o regtest3.o regtest4.o cryptlib.export.o cpu.export.o integer.export.o 3way.export.o adhoc.export.o adler32.export.o algebra.export.o algparam.export.o allocate.export.o arc4.export.o aria.export.o aria_simd.export.o ariatab.export.o asn.export.o authenc.export.o base32.export.o base64.export.o basecode.export.o bench1.export.o bench2.export.o bench3.export.o bfinit.export.o blake2.export.o blake2b_simd.export.o blake2s_simd.export.o blowfish.export.o blumshub.export.o camellia.export.o cast.export.o casts.export.o cbcmac.export.o ccm.export.o chacha.export.o chacha_avx.export.o chacha_simd.export.o chachapoly.export.o cham.export.o cham_simd.export.o channels.export.o cmac.export.o crc.export.o crc_simd.export.o darn.export.o datatest.export.o default.export.o des.export.o dessp.export.o dh.export.o dh2.export.o dll.export.o dlltest.export.o donna_32.export.o donna_64.export.o donna_sse.export.o dsa.export.o eax.export.o ec2n.export.o eccrypto.export.o ecp.export.o elgamal.export.o emsa2.export.o eprecomp.export.o esign.export.o files.export.o filters.export.o fips140.export.o fipsalgt.export.o fipstest.export.o gcm.export.o gcm_simd.export.o gf256.export.o gf2_32.export.o gf2n.export.o gf2n_simd.export.o gfpcrypt.export.o gost.export.o gzip.export.o hc128.export.o hc256.export.o hex.export.o hight.export.o hmac.export.o hrtimer.export.o ida.export.o idea.export.o iterhash.export.o kalyna.export.o kalynatab.export.o keccak.export.o keccak_core.export.o keccak_simd.export.o lea.export.o lea_simd.export.o luc.export.o mars.export.o marss.export.o md2.export.o md4.export.o md5.export.o misc.export.o modes.export.o mqueue.export.o mqv.export.o nbtheory.export.o oaep.export.o osrng.export.o padlkrng.export.o panama.export.o pkcspad.export.o poly1305.export.o polynomi.export.o pssr.export.o pubkey.export.o queue.export.o rabbit.export.o rabin.export.o randpool.export.o rc2.export.o rc5.export.o rc6.export.o rdrand.export.o rdtables.export.o regtest1.export.o regtest2.export.o regtest3.export.o regtest4.export.o rijndael.export.o rijndael_simd.export.o ripemd.export.o rng.export.o rsa.export.o rw.export.o safer.export.o salsa.export.o scrypt.export.o seal.export.o seed.export.o serpent.export.o sha.export.o sha3.export.o sha_simd.export.o shacal2.export.o shacal2_simd.export.o shake.export.o shark.export.o sharkbox.export.o simeck.export.o simon.export.o simon128_simd.export.o skipjack.export.o sm3.export.o sm4.export.o sm4_simd.export.o sosemanuk.export.o speck.export.o speck128_simd.export.o square.export.o squaretb.export.o sse_simd.export.o strciphr.export.o tea.export.o test.export.o tftables.export.o threefish.export.o tiger.export.o tigertab.export.o ttmac.export.o tweetnacl.export.o twofish.export.o validat0.export.o validat1.export.o validat10.export.o validat2.export.o validat3.export.o validat4.export.o validat5.export.o validat6.export.o validat7.export.o validat8.export.o validat9.export.o vmac.export.o wake.export.o whrlpool.export.o xed25519.export.o xtr.export.o xtrcrypt.export.o xts.export.o zdeflate.export.o zinflate.export.o zlib.export.o cryptlib.import.o cpu.import.o integer.import.o 3way.import.o adler32.import.o algebra.import.o algparam.import.o allocate.import.o arc4.import.o aria.import.o aria_simd.import.o ariatab.import.o asn.import.o authenc.import.o base32.import.o base64.import.o basecode.import.o bfinit.import.o blake2.import.o blake2b_simd.import.o blake2s_simd.import.o blowfish.import.o blumshub.import.o camellia.import.o cast.import.o casts.import.o cbcmac.import.o ccm.import.o chacha.import.o chacha_avx.import.o chacha_simd.import.o chachapoly.import.o cham.import.o cham_simd.import.o channels.import.o cmac.import.o crc.import.o crc_simd.import.o darn.import.o default.import.o des.import.o dessp.import.o dh.import.o dh2.import.o dll.import.o donna_32.import.o donna_64.import.o donna_sse.import.o dsa.import.o eax.import.o ec2n.import.o eccrypto.import.o ecp.import.o elgamal.import.o emsa2.import.o eprecomp.import.o esign.import.o files.import.o filters.import.o fips140.import.o fipstest.import.o gcm.import.o gcm_simd.import.o gf256.import.o gf2_32.import.o gf2n.import.o gf2n_simd.import.o gfpcrypt.import.o gost.import.o gzip.import.o hc128.import.o hc256.import.o hex.import.o hight.import.o hmac.import.o hrtimer.import.o ida.import.o idea.import.o iterhash.import.o kalyna.import.o kalynatab.import.o keccak.import.o keccak_core.import.o keccak_simd.import.o lea.import.o lea_simd.import.o luc.import.o mars.import.o marss.import.o md2.import.o md4.import.o md5.import.o misc.import.o modes.import.o mqueue.import.o mqv.import.o nbtheory.import.o oaep.import.o osrng.import.o padlkrng.import.o panama.import.o pkcspad.import.o poly1305.import.o polynomi.import.o pssr.import.o pubkey.import.o queue.import.o rabbit.import.o rabin.import.o randpool.import.o rc2.import.o rc5.import.o rc6.import.o rdrand.import.o rdtables.import.o rijndael.import.o rijndael_simd.import.o ripemd.import.o rng.import.o rsa.import.o rw.import.o safer.import.o salsa.import.o scrypt.import.o seal.import.o seed.import.o serpent.import.o sha.import.o sha3.import.o sha_simd.import.o shacal2.import.o shacal2_simd.import.o shake.import.o shark.import.o sharkbox.import.o simeck.import.o simon.import.o simon128_simd.import.o skipjack.import.o sm3.import.o sm4.import.o sm4_simd.import.o sosemanuk.import.o speck.import.o speck128_simd.import.o square.import.o squaretb.import.o sse_simd.import.o strciphr.import.o tea.import.o tftables.import.o threefish.import.o tiger.import.o tigertab.import.o ttmac.import.o tweetnacl.import.o twofish.import.o vmac.import.o wake.import.o whrlpool.import.o xed25519.import.o xtr.import.o xtrcrypt.import.o xts.import.o zdeflate.import.o zinflate.import.o zlib.import.o adhoc.import.o test.import.o bench1.import.o bench2.import.o bench3.import.o datatest.import.o dlltest.import.o fipsalgt.import.o validat0.import.o validat1.import.o validat2.import.o validat3.import.o validat4.import.o validat5.import.o validat6.import.o validat7.import.o validat8.import.o validat9.import.o validat10.import.o regtest1.import.o regtest2.import.o regtest3.import.o regtest4.import.o dlltest.dllonly.o

neon_simd.o is missing in both clean commands

noloader commented 3 years ago

OK, I think I see what is going on.

export IOS_SDK=iPhoneOS
export IOS_CPU=arm64
source TestScripts/setenv-ios.sh
export CXXFLAGS="-DNDEBUG -g2 -O3 -fPIC -pipe -fembed-bitcode"

Now, make distclean. Notice neon_simd.o is not included in the file list.

We actually have to make -f GNUmakefile-cross distclean. Notice neon_simd.o is included in the file list.

The reason the regular makefile running on x86 does not include neon_simd.o during clean is this piece of goodness in GNUmakefile : 1135:

# Remove unneeded arch specific files to speed build time.
ifeq ($(IS_PPC32)$(IS_PPC64),00)
  SRCS := $(filter-out ppc_%,$(SRCS))
endif
ifeq ($(IS_ARM32)$(IS_ARMV8),00)
  SRCS := $(filter-out arm_%,$(SRCS))
  SRCS := $(filter-out neon_%,$(SRCS))
endif
ifeq ($(IS_X86)$(IS_X32)$(IS_X64),000)
  SRCS := $(filter-out sse_%,$(SRCS))
endif

The source files are filtered from SRCS, and then the object files are missing from OBJS:

OBJS := $(SRCS:.cpp=.o)
OBJS := $(OBJS:.S=.o)

Later, make clean uses LIBOBJS, DLLOBJS, etc to clean the artifacts:

.PHONY: clean
clean:
    -$(RM) adhoc.cpp.o adhoc.cpp.proto.o $(LIBOBJS) rdrand-*.o $(TESTOBJS) $(DLLOBJS) $(LIBIMPORTOBJS) $(TESTIMPORTOBJS) $(DLLTESTOBJS)
    ...

lordadamson commented 3 years ago

aah good catch.

Feels like the kind of bug that is caused by caching. If we're not saving a significant build time with this trick then I'd recommend we get rid of it completely.

It's always nice to have things as simple and straight forward as possible.

noloader commented 3 years ago

I'm wondering if we should just rm *.o to sidestep it.

In the past I did not want to do it in case there were non-cryptopp object files there.

Ping @mouse07410. Mouse07410, are we allowed to whack all object files in our directory during a make clean or make distclean? Or are we limited to just the object files produced by the library?

I think the controlling document is GNU Coding Standards | Standard Targets. The clean and distclean recipes only authorize deleting object files "... that are created by configuring or building the program...".

lordadamson commented 3 years ago

Honestly rm *.o is the kind of low-tech solution I'd go for :D :D

noloader commented 3 years ago

OK, this should handle the problem: commit a184ce1aba1a. The commit builds a list of all potential object files and cleans them. It does not filter-out object files. It does not matter which makefile you use.

noloader commented 3 years ago

@lordadamson,

~~You'll need this one too. I missed it on the original commit. Commit 7315a6f907a1.~~

Hold on, that broke things. Reverting that one.

noloader commented 3 years ago

@lordadamson,

The difference between rm *.o and:

CLEAN_SRCS := $(wildcard *.cpp)
CLEAN_OBJS := $(CLEAN_SRCS:.cpp=.o)

is that *.o whacks all object files. CLEAN_OBJS only cleans object files produced by the library.

If you compile your own my_test.cxx, then its object file will be skipped because it is not included in CLEAN_SRCS. In fact, I use the *.cxx trick for test programs. Each *.cxx can have a main that would otherwise result in duplicate symbols main if they were included in CLEAN_SRCS.

noloader commented 3 years ago

This is the commit that handles Cryptogams object files: Commit ae20711372b1.

mouse07410 commented 3 years ago

My personal take: make clean is allowed to (and should?) remove all the object files in our directory. make distcleanshould also remove all the source files that the build (or configuration) process generated.

noloader commented 3 years ago

Thanks @mouse07410,

If this ever surfaces again we'll just whack all of the object files.

weidai11 / cryptopp

make clean does not delete neon_simd.o #998