search

facebookresearch / ocean

Ocean is the in-house framework for Computer Vision (CV) and Augmented Reality (AR) applications at Meta. It is platform independent and is mainly implemented in C/C++.
https://facebookresearch.github.io/ocean/
MIT License
639 stars 58 forks source link

Third party build script fails on protozero. #39

Closed robo-todd closed 1 month ago

robo-todd commented 1 month ago

Instructions to reproduce the problem:

Environment: Ubuntu 22.04 system on X86_64, Ocean cloned, checkout v1.0.1

Command or list of commands needed to trigger the problem:

cd ocean
mkdir temp
cd temp
../build/cmake/build_thirdparty_linuxunix.sh -c release -l shared -b build -i install

Full logs or other relevant observations:

Building: 9. protozero

-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Third-party dependencies
--   protozero
--     Found Git: /usr/bin/git (found version "2.34.1") 
--     CPM: Adding package protozero@1.7.1 (v1.7.1 to /home/todd/Projects/ocean/temp/build/third-party_cache/protozero/f1779d6ad7991645cf6607a0a29edc19a69d5e8d)
--     Building in C++20 mode
--     Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so (found version "3.12.4") 
--     Looking for clang-tidy
--     Looking for clang-tidy - found /usr/bin/clang-tidy
--     Looking for cppcheck
--     Looking for cppcheck - found
--     Looking for iwyu
--     Looking for iwyu - not found
--       Build target 'iwyu' will not be available.
--     Configuring documentation
--     Looking for doxygen
--     Found Doxygen: /usr/bin/doxygen (found version "1.9.1") found components: doxygen dot 
--     Looking for doxygen - found
--     Configuring documentation - done
--   protozero - completed
-- Third-party dependencies - completed
-- Configuring done (1.0s)
CMake Error at /home/todd/Projects/ocean/temp/build/third-party_cache/protozero/f1779d6ad7991645cf6607a0a29edc19a69d5e8d/CMakeLists.txt:79 (add_dependencies):
  The dependency target "writer_tests" of target "clang-tidy" does not exist.

-- Generating done (0.0s)
CMake Generate step failed.  Build files cannot be regenerated correctly.

Expected behavior:

End up with the necessary dependency libraries built.

thorntondr commented 1 month ago

We're working on reproducing this.

ASchneiderMeta commented 1 month ago

There's a commit on the way to address this. It basically duplicates a fix done in the protozero repo in the next commit after the release we use in third-party.

robo-todd commented 1 month ago

Update

The build now gets past protozero, but fails building libtiff. Seemingly unable to find its own "built" zlib? Looking at the filesystem there is no "install" directory down inside of the build/third-party_cache/libtiff// directory. There is an installed zlib.h in the install/linux_static_Release/include directory.

It is like the build is set up to look in the wrong place?

Commands

../build/cmake/build_thirdparty_linuxunix.sh -c release -l shared -b build -i install

Also fails in same way when run with -l static.

Build log output:

Building: 12. libtiff

-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Third-party dependencies
--   libtiff
--     Found Git: /usr/bin/git (found version "2.34.1") 
--     CPM: Adding package libtiff@4.6.0 (v4.6.0 to /home/todd/Projects/ocean/temp/build/third-party_cache/libtiff/0519a59270d2b5bb0de3b3cc937ad7567982101a)
--     Building tiff version 4.6.0
--     Performing Test C_FLAG_Wall
--     Performing Test C_FLAG_Wall - Success
--     Performing Test C_FLAG_Winline
--     Performing Test C_FLAG_Winline - Success
--     Performing Test C_FLAG_Wformat_security
--     Performing Test C_FLAG_Wformat_security - Success
--     Performing Test C_FLAG_Wpointer_arith
--     Performing Test C_FLAG_Wpointer_arith - Success
--     Performing Test C_FLAG_Wdisabled_optimization
--     Performing Test C_FLAG_Wdisabled_optimization - Success
--     Performing Test C_FLAG_Wno_unknown_pragmas
--     Performing Test C_FLAG_Wno_unknown_pragmas - Success
--     Performing Test C_FLAG_fstrict_aliasing
--     Performing Test C_FLAG_fstrict_aliasing - Success
--     Looking for assert.h
--     Looking for assert.h - found
--     Looking for fcntl.h
--     Looking for fcntl.h - found
--     Looking for io.h
--     Looking for io.h - not found
--     Looking for strings.h
--     Looking for strings.h - found
--     Looking for sys/time.h
--     Looking for sys/time.h - found
--     Looking for sys/types.h
--     Looking for sys/types.h - found
--     Looking for unistd.h
--     Looking for unistd.h - found
--     Looking for getopt
--     Looking for getopt - found
--     Looking for optarg
--     Looking for optarg - found
--     Looking for mmap
--     Looking for mmap - found
--     Looking for setmode
--     Looking for setmode - not found
--     Looking for stdint.h
--     Looking for stdint.h - found
--     Looking for stddef.h
--     Looking for stddef.h - found
--     Check size of size_t
--     Check size of size_t - done
CMake Error at /usr/share/cmake-3.28/Modules/FindZLIB.cmake:178 (file):
  file STRINGS file
  "/home/todd/Projects/ocean/temp/build/third-party_cache/libtiff/0519a59270d2b5bb0de3b3cc937ad7567982101a/install/linux_static_Release/include/zlib.h"
  cannot be read.
Call Stack (most recent call first):
  /home/todd/Projects/ocean/temp/build/third-party_cache/libtiff/0519a59270d2b5bb0de3b3cc937ad7567982101a/cmake/DeflateCodec.cmake:29 (find_package)
  /home/todd/Projects/ocean/temp/build/third-party_cache/libtiff/0519a59270d2b5bb0de3b3cc937ad7567982101a/CMakeLists.txt:102 (include)

--     Found ZLIB: install/linux_static_Release/lib/libz.a (found version "")  
--     Found Deflate: /usr/lib/x86_64-linux-gnu/libdeflate.so (found version "1.10") 
CMake Error at /home/todd/Projects/ocean/temp/build/third-party_cache/libtiff/0519a59270d2b5bb0de3b3cc937ad7567982101a/cmake/FindJPEG.cmake:87 (file):
  file STRINGS file
  "/home/todd/Projects/ocean/temp/build/third-party_cache/libtiff/0519a59270d2b5bb0de3b3cc937ad7567982101a/install/linux_static_Release/include/jpeglib.h"
  cannot be read.
Call Stack (most recent call first):
  /home/todd/Projects/ocean/temp/build/third-party_cache/libtiff/0519a59270d2b5bb0de3b3cc937ad7567982101a/cmake/JPEGCodec.cmake:29 (find_package)
  /home/todd/Projects/ocean/temp/build/third-party_cache/libtiff/0519a59270d2b5bb0de3b3cc937ad7567982101a/CMakeLists.txt:108 (include)

--     Found JPEG: install/linux_static_Release/lib/libjpeg.a  
--     Performing Test HAVE_JPEGTURBO_DUAL_MODE_8_12
--     Performing Test HAVE_JPEGTURBO_DUAL_MODE_8_12 - Failed
--     Found JBIG: /usr/lib/x86_64-linux-gnu/libjbig.so (found version "2.1") 
--     Looking for jbg_newlen
--     Looking for jbg_newlen - found
--     Could NOT find LERC (missing: LERC_LIBRARY LERC_INCLUDE_DIR) 
--     Looking for lzma_auto_decoder in /usr/lib/x86_64-linux-gnu/liblzma.so
--     Looking for lzma_auto_decoder in /usr/lib/x86_64-linux-gnu/liblzma.so - found
--     Looking for lzma_easy_encoder in /usr/lib/x86_64-linux-gnu/liblzma.so
--     Looking for lzma_easy_encoder in /usr/lib/x86_64-linux-gnu/liblzma.so - found
--     Looking for lzma_lzma_preset in /usr/lib/x86_64-linux-gnu/liblzma.so
--     Looking for lzma_lzma_preset in /usr/lib/x86_64-linux-gnu/liblzma.so - found
--     Found liblzma: /usr/lib/x86_64-linux-gnu/liblzma.so (found version "5.2.5") 
--     Found ZSTD: /usr/lib/x86_64-linux-gnu/libzstd.so (found version "1.4.8") 
--     Looking for ZSTD_decompressStream
--     Looking for ZSTD_decompressStream - found
--     Found WebP: /usr/lib/x86_64-linux-gnu/libwebp.so  
--     Looking for pow
--     Looking for pow - not found
--     Looking for pow
--     Looking for pow - found
--     Found CMath: TRUE  
CMake Deprecation Warning at /home/todd/Projects/ocean/temp/build/third-party_cache/libtiff/0519a59270d2b5bb0de3b3cc937ad7567982101a/doc/CMakeLists.txt:27 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

--     Found Python3: /usr/bin/python3.10 (found version "3.10.12") found components: Interpreter 
--     Looking for sphinx-build - /usr/bin/sphinx-build
--     Checking manual page dependencies
--     Sphinx manual pages: /home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/libtiff.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/tiffcp.1;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/tiffdump.1;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/tiffinfo.1;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/tiffset.1;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/tiffsplit.1;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/_TIFFauxiliary.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/_TIFFRewriteField.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFAccessTagMethods.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFbuffer.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFClientInfo.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFClose.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFcodec.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFcolor.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFCreateDirectory.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFCustomDirectory.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFCustomTagList.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFDataWidth.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFDeferStrileArrayWriting.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFError.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFFieldDataType.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFFieldName.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFFieldPassCount.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFFieldQuery.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFFieldReadCount.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFFieldTag.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFFieldWriteCount.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFFlush.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFGetField.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFmemory.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFMergeFieldInfo.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFOpen.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFPrintDirectory.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFProcFunctions.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFquery.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFReadDirectory.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFReadEncodedStrip.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFReadEncodedTile.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFReadFromUserBuffer.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFReadRawStrip.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFReadRawTile.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFReadRGBAImage.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFReadRGBAStrip.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFReadRGBATile.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFReadScanline.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFReadTile.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFRGBAImage.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFSetDirectory.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFSetField.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFSetTagExtender.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFsize.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFStrileQuery.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFstrip.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFswab.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFtile.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFWarning.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFWriteDirectory.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFWriteEncodedStrip.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFWriteEncodedTile.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFWriteRawStrip.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFWriteRawTile.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFWriteScanline.3tiff;/home/todd/Projects/ocean/temp/build/linux_static_Release/libtiff/_deps/libtiff-build/doc/man/TIFFWriteTile.3tiff;
--     
--     Libtiff is now configured for Linux-6.8.0-45-generic
--     
--       Libtiff build version:       4.6.0
--       Libtiff library version:     6.0.2
--       Libtiff release date:        20230908
--     
--       Installation directory:             /home/todd/Projects/ocean/temp/install/linux_static_Release
--       Documentation directory:            share/doc/tiff
--       C compiler:                         /usr/bin/cc
--       C++ compiler:                       /usr/bin/c++
--       Build shared libraries:             OFF
--       Build tools:                        OFF
--       Build tools-unsupported:            OFF
--       Build tests:                        OFF
--       Build contrib:                      OFF
--       Build docs:                         ON
--       Build deprecated features:          OFF
--       Enable linker symbol versioning:    FALSE
--       Support Microsoft Document Imaging: ON
--       Use win32 IO:                       OFF
--     
--      Support for internal codecs:
--       CCITT Group 3 & 4 algorithms:       ON
--       Macintosh PackBits algorithm:       ON
--       LZW algorithm:                      ON
--       ThunderScan 4-bit RLE algorithm:    ON
--       NeXT 2-bit RLE algorithm:           ON
--       LogLuv high dynamic range encoding: ON
--     
--      Support for external codecs:
--       ZLIB support:                       Requested:ON Availability:TRUE Support:TRUE
--       libdeflate support:                 Requested:OFF Availability:TRUE Support:FALSE
--       Pixar log-format algorithm:         Requested:ON Availability:TRUE Support:TRUE
--       JPEG support:                       Requested:ON Availability:TRUE Support:TRUE
--       Old JPEG support:                   Requested:ON Availability:TRUE Support:TRUE
--       JPEG 8/12 bit dual mode:            Requested:OFF Availability:FALSE Support:FALSE
--       ISO JBIG support:                   Requested:OFF Availability:TRUE Support:FALSE
--       LERC support:                       Requested:OFF Availability:FALSE Support:FALSE
--       LZMA2 support:                      Requested:OFF Availability:TRUE Support:FALSE
--       ZSTD support:                       Requested:OFF Availability:TRUE Support:FALSE
--       WEBP support:                       Requested:OFF Availability:TRUE Support:FALSE
--     
--       C++ support:                        ON (requested) TRUE (availability)
--     
--   libtiff - completed
-- Third-party dependencies - completed
-- Configuring incomplete, errors occurred!

Some builds have failed.
- static + Release
ASchneiderMeta commented 1 month ago

In successful builds, there is no libtiff/.../install directory. I don't understand why FindZLIB.cmake or FindJPEG.cmake would be looking for it. The latter is even supplied by libtiff itself, which makes it all the more puzzling. I haven't been able to reproduce your problem yet, but we're working on it.

robo-todd commented 1 month ago

So perhaps some insight can be gleaned from this:

I tried the same thing but gave full absolute pathnames for the -i and -b arguments and it worked. I can use that as a workaround for now and move on. But just using the simple paths or somehow using a sub-directory as a build location is getting something confused about paths.

ASchneiderMeta commented 1 month ago

That's a great clue! We'll look for a proper fix.

Committed https://github.com/facebookresearch/ocean/commit/a6d97550494f5fc5d67fe93d4710b31ba0b1eed2

ASchneiderMeta commented 1 month ago

With the original protozero failure fixed by 0e981b46952855db6ba491a76fc7d0b57cd42e80 and the subsequent relative path problem fixed by a6d97550494f5fc5d67fe93d4710b31ba0b1eed2, I'm going to close this issue. Feel free to reopen as you see fit.