A guide for building Meshroom and all the Dependenicies

Each required and optional component has good documentation for it's requirements and how to use it in other projects. (Thanks!)

However, if your goal is to create a working development version of meshroom, there are some potentially duplicate steps, or steps that in one document that might affect choices made when installing one of the other packages. And there seem to be some specific steps needed for meshroom that may not be obvious.

It would be really nice to have a top-bottom document that eg listed all the pre-reqs of the entire AliceVision/meshroom environment, and the build options they need.

Here's an example/potential start for Windows: https://gist.github.com/julianrendell/da6372201234d943d3e4f77e43674fd2

(Missing are instructions for taking the build and making a distributable version.)

Yeah, I've been struggling with this on linux. I still haven't been able to get opencv working with alicevision, and the second time I built everything, it wouldn't load the GUI because of a qml problem... and most frustratingly to me, it errors out when trying to solve LSCM texture unwrap, even though it seems to be using eigen which should be built properly.

LCSM rely on geogram, so you are probably missing a geogram build option.

@julianrendell I agree on the need to improve build explanations. Contributions in this direction would be much appreciated. But I would prefer to describe AliceVision build on the AliceVision repo as it's the only way to keep it up-to-date.

@fabiencastan - I understand that seems like the right place BUT I found it quite hard to get all the pre-requisites right across the 7 direct and indirect dependencies I discovered... and it's not easy for a beginner to know if they've traversed the whole tree!

Especially as some aren't technically required to run, but practically make Meshroom so much better to use (like qmlAlembic!)

On top of that some of the sub-components have different requirements on shared dependencies, like Qt and OpenCV... I tried vcpkg for the first time. It was very helpful, but it doesn't download pre-built libraries; I think I rebuilt Qt and OpenCV ~4 times each to get the right modules enabled to build all the AliceVision parts. Definitely user naivety, but I hope the docs could make this clearer.

<...insert writing and deleting many words...>

I think the crux for a new person trying to use trunk builds is availability and setup hurdles.

It's not that any of the component install instructions are wrong, just incomplete if the goal is to run Meshroom with all options available.

I just realized that I think this would fix this and a few other issues:

Tidy up the appveyor and docker scripts. Including the scripts that setup a clean CI environment for a given OS!

I know CI in principle, but the I have no idea how the build dependencies get into the appveyor OS image(s)! Same for the docker- references to MESHROOM_BUNDLE, but no idea what that is or how it's generated.

If the CI dependency setup was scripted, it's a tested document that a person could also run on their workstation, or execute line by line.

This might also allow for a more regular CI build for Windows, Mac, Docker that could be made available for users that want to try new features, but don't have the skill to build it all.

@EwoutH I think I saw your name on some appveyor related issues; would this help those too?

@fabiencastan would that work for you? The current component build instructions are technically correct as they are; full CI setup scripts for Meshroom would be the missing documentation needed for 'integrators' .

@EwoutH I think I saw your name on some appveyor related issues; would this help those too?

Yes this will help a lot! As soon as there are proper build guides I will start setting op CI

@EwoutH - let's make it happen. I'm not familiar with AppVeyor, but in a past life set up Hudson and Jenkins servers and builds from scratch. But I'm a bit of a "fundamentalist": in that I prefer to start with getting a "bootstrap from basics" build working first (as an original "golden master generator") and then build up caching/faster builds after that.

There is an existing appveyor.yml in the Meshroom repo, but it's setup as a python based app; doesn't build the pre-req binaries, so what it generates would run... but not be able to do anything ;-). If I'm understanding it correctly.

I'd like to see something that builds all the binary pre-reqs and Meshroom and produces a useable package.

This could then be tweaked per branch as needed to use specific versions of dependencies (if needed.)

I had a quick look at AppVeyor, and it looks like there is a VS2019 image with QT installed, and that it will work with VCPKG (https://www.appveyor.com/docs/lang/cpp/). There is a 60 minute build time limit that we might hit. There is build caching, but it's limited to 1Gb (https://www.appveyor.com/docs/build-cache/) - which could be an issue. My naive VCPKG setup is occupying ~40Gb... but a big chunk is QT (which shouldn't be needed via VCPKG), and it looks like it can be configured to only build X64; might be able to get it under 1Gb. Also looks like CMake is installed in their default windows config.

My suggestion is that we create a fork that we both have access to, create an appveyor directory (to keep things tidy), set up appveyor account(s) (not sure how teams work for appveyor) and set up most of the work to be done with basic batch scripts that are simple enough that a person could execute or follow them to build their own working dev environment. This would solve my wish of having a "how to build a dev environment/trunk build" guide.

My focus/interest right now is seeing this work for Windows so I can point customers/colleagues/students too it. But it should be relatively straight forward (and probably a lot easier/less annoying) to do similar for a docker image - which seems a very reasonable initial way to make it deployable on Linux. And if it's easier to deploy up-to-date nodes, maybe we can figure out how to set up kubernetes/rancher/etc (vaguely know what they are, not used any...) FireWorks based render pool configuration :-)

I don't have a lot of time to work on this, but I think it could be a good fit: tweak a config, let it run, adjust the next day. What do you think?

A bit busy until Thursday, I will start/help development then

I tried to build with readme from @julianrendell , but i get stuck at building magma now on win 10. Can anyone please give me some help with that?

1) tried with nmake generator . Command is cmake .. -DCMAKE_TOOLCHAIN_FILE=%TCF% -DCMAKE_PREFIX_PATH=%PREFIX% -DVCPKG_TARGET_TRIPLET=x64-windows -G "NMake Makefiles" -DCMAKE_INSTALL_PREFIX=%INSTALL% -DCMAKE_BUILD_TYPE=Release -DUSE_FORTRAN=OFF -DCMAKE_WINDOWS_EXPORT_ALL_SYMBOLS=TRUE -DBUILD_SHARED_LIBS=TRUE but when i run "nmake install" i got the "command line is too long" error 2) tried with VS 2019. Command is: cmake .. -DCMAKE_TOOLCHAIN_FILE=%TCF% -DCMAKE_PREFIX_PATH=%PREFIX% -DVCPKG_TARGET_TRIPLET=x64-windows -G "Visual Studio 16 2019" -A x64 -DCMAKE_INSTALL_PREFIX=%INSTALL% -DCMAKE_BUILD_TYPE=Release -DUSE_FORTRAN=OFF -DCMAKE_WINDOWS_EXPORT_ALL_SYMBOLS=TRUE -DBUILD_SHARED_LIBS=TRUE but when i run "build solution" i got the "error MSB6006: "cmd.exe" exited with code 9020." 3) also tried adding

SET(CMAKE_C_USE_RESPONSE_FILE_FOR_OBJECTS 1)
SET(CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS 1)

SET(CMAKE_C_RESPONSE_FILE_LINK_FLAG "@")
SET(CMAKE_CXX_RESPONSE_FILE_LINK_FLAG "@")

to CMakeLists.txt and vcpkg toolchain file - still useless.

CMake version: 3.18.1 VS version: 16.7.3

Ok i've found something: if i open D:\VS_projects\meshroom_with_dependencies\magma_build\CMakeFiles\magma.dir\build.make an replace that very long command line with echo >nul && "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin\nvcc.exe" -Xcompiler -fPIC -DHAVE_CUBLAS -gencode arch=compute_75,code=sm_75 -gencode arch=compute_75,code=compute_75 -DADD_ -m64 -ccbin "C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/cl.exe" -dlink --options-file "D:/VS_projects/meshroom_with_dependencies/magma/_build/link_files.txt" -o D:/VS_projects/meshroom_with_dependencies/magma/_build/CMakeFiles/magma.dir/./magma_intermediate_link.obj

where D:/VS_projects/meshroom_with_dependencies/magma/_build/link_files.txt is file, that contains every *.obj file from very long command line written in one line, then i'm able to go further.

It builds some objects and stops at one with following error:

[  0%] Building CXX object CMakeFiles/magma.dir/interface_cuda/interface.cpp.obj
interface.cpp
D:\VS_projects\meshroom_with_dependencies\magma\include\magma_operators.h(54): warning C4100: x: неиспользованный формальный параметр
D:\VS_projects\meshroom_with_dependencies\magma\include\magma_operators.h(55): warning C4100: x: неиспользованный формальный параметр
D:\VS_projects\meshroom_with_dependencies\magma\interface_cuda\interface.cpp(424): warning C4996: 'ctime': This function or variable may be unsafe. Consider using ctime_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online
 help for details.
D:\VS_projects\meshroom_with_dependencies\magma\interface_cuda\interface.cpp(461): error C2039: "memoryType": не является членом "cudaPointerAttributes".
C:\PROGRA~1\NVIDIA~2\CUDA\v11.0\include\driver_types.h(1370): note:  см. объявление "cudaPointerAttributes"
NMAKE : fatal error U1077: C:\PROGRA~2\MICROS~3\2019\COMMUN~1\VC\Tools\MSVC\1427~1.291\bin\Hostx64\x64\cl.exe : возвращенный код "0x2"
Stop.
NMAKE : fatal error U1077: "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\bin\HostX64\x64\nmake.exe" : возвращенный код "0x2"
Stop.
NMAKE : fatal error U1077: "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\bin\HostX64\x64\nmake.exe" : возвращенный код "0x2"
Stop.

EDIT: the reason for this error was CUDA version (mine is 11.0). They replaced "memoryType" with "type". So i opened D:\VS_projects\meshroom_with_dependencies\magma\interface_cuda\interface.cpp and changed return (attr.memoryType == cudaMemoryTypeDevice); with return (attr.type == cudaMemoryTypeDevice);

It's building other objects now. Waiting for...

UPD: now i'm get stuck with following, any thoughts? [ 28%] Linking CXX shared library magma.dll CMAKE_NM-NOTFOUND LINK: command "C:\PROGRA~2\MICROS~3\2019\COMMUN~1\VC\Tools\MSVC\1427~1.291\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\magma.dir\objects1.rsp /out:magma.dll /implib:lib\magma.lib /pdb:D:\VS_projects\meshroom_with_dependencies\magma\_build\mag ma.pdb /dll /version:0.0 /DEF:CMakeFiles\magma.dir\exports.def C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\lib\x64\cudart_static.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\lib\x64\cudadevrt.lib D:\VS_projects\vcp kg\installed\x64-windows\lib\lapack.lib D:\VS_projects\vcpkg\installed\x64-windows\lib\libf2c.lib D:\VS_projects\vcpkg\installed\x64-windows\lib\openblas.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\lib\x64\cudart.lib C:\Progra m Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\lib\x64\cublas.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\lib\x64\cusparse.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg3 2.lib advapi32.lib /MANIFEST /MANIFESTFILE:magma.dll.manifest" failed (exit code 1120) with the following output: Создается библиотека lib\magma.lib и объект lib\magma.exp LINK : warning LNK4098: библиотека по умолчанию "MSVCRT" конфликтует с использованием других библиотек; используйте /NODEFAULTLIB:library LINK : warning LNK4217: Символ "fflush", определенный в "libucrt.lib(fflush.obj)", импортирован "libf2c.lib(wsfe.c.obj)" в функции "xw_end". LINK : warning LNK4286: Символ "fflush", определенный в "libucrt.lib(fflush.obj)", импортирован "libf2c.lib(sig_die.c.obj)". LINK : warning LNK4217: Символ "fflush", определенный в "libucrt.lib(fflush.obj)", импортирован "libf2c.lib(close.c.obj)" в функции "f_clos". LINK : warning LNK4286: Символ "_errno", определенный в "libucrt.lib(errno.obj)", импортирован "libf2c.lib(err.c.obj)". LINK : warning LNK4286: Символ "_errno", определенный в "libucrt.lib(errno.obj)", импортирован "libf2c.lib(wrtfmt.c.obj)". LINK : warning LNK4286: Символ "_errno", определенный в "libucrt.lib(errno.obj)", импортирован "libf2c.lib(endfile.c.obj)". LINK : warning LNK4217: Символ "_errno", определенный в "libucrt.lib(errno.obj)", импортирован "libf2c.lib(wsfe.c.obj)" в функции "s_wsfe". LINK : warning LNK4286: Символ "_errno", определенный в "libucrt.lib(errno.obj)", импортирован "libf2c.lib(fmt.c.obj)". LINK : warning LNK4286: Символ "_errno", определенный в "libucrt.lib(errno.obj)", импортирован "libf2c.lib(sfe.c.obj)". LINK : warning LNK4286: Символ "_errno", определенный в "libucrt.lib(errno.obj)", импортирован "libf2c.lib(open.c.obj)". LINK : warning LNK4286: Символ "__acrt_iob_func", определенный в "libucrt.lib(_file.obj)", импортирован "libf2c.lib(f77_aloc.c.obj)". LINK : warning LNK4217: Символ "__acrt_iob_func", определенный в "libucrt.lib(_file.obj)", импортирован "libf2c.lib(fmt.c.obj)" в функции "do_fio". LINK : warning LNK4286: Символ "__acrt_iob_func", определенный в "libucrt.lib(_file.obj)", импортирован "libf2c.lib(err.c.obj)". LINK : warning LNK4286: Символ "__acrt_iob_func", определенный в "libucrt.lib(_file.obj)", импортирован "libf2c.lib(wrtfmt.c.obj)". LINK : warning LNK4286: Символ "__acrt_iob_func", определенный в "libucrt.lib(_file.obj)", импортирован "libf2c.lib(sig_die.c.obj)". LINK : warning LNK4286: Символ "__stdio_common_vfprintf", определенный в "libucrt.lib(output.obj)", импортирован "libf2c.lib(f77_aloc.c.obj)". LINK : warning LNK4217: Символ "__stdio_common_vfprintf", определенный в "libucrt.lib(output.obj)", импортирован "libf2c.lib(fmt.c.obj)" в функции "_vfprintf_l". LINK : warning LNK4286: Символ "__stdio_common_vfprintf", определенный в "libucrt.lib(output.obj)", импортирован "libf2c.lib(err.c.obj)". LINK : warning LNK4286: Символ "__stdio_common_vfprintf", определенный в "libucrt.lib(output.obj)", импортирован "libf2c.lib(wrtfmt.c.obj)". LINK : warning LNK4286: Символ "__stdio_common_vfprintf", определенный в "libucrt.lib(output.obj)", импортирован "libf2c.lib(sig_die.c.obj)". LINK : warning LNK4217: Символ "free", определенный в "libucrt.lib(free.obj)", импортирован "libf2c.lib(s_cat.c.obj)" в функции "s_cat". LINK : warning LNK4286: Символ "free", определенный в "libucrt.lib(free.obj)", импортирован "libf2c.lib(open.c.obj)". LINK : warning LNK4286: Символ "free", определенный в "libucrt.lib(free.obj)", импортирован "libf2c.lib(close.c.obj)". LINK : warning LNK4217: Символ "fclose", определенный в "libucrt.lib(fclose.obj)", импортирован "libf2c.lib(open.c.obj)" в функции "f_open". LINK : warning LNK4286: Символ "fclose", определенный в "libucrt.lib(fclose.obj)", импортирован "libf2c.lib(close.c.obj)". LINK : warning LNK4286: Символ "fclose", определенный в "libucrt.lib(fclose.obj)", импортирован "libf2c.lib(endfile.c.obj)". LINK : warning LNK4217: Символ "fopen", определенный в "libucrt.lib(fopen.obj)", импортирован "libf2c.lib(open.c.obj)" в функции "f_open". LINK : warning LNK4286: Символ "fopen", определенный в "libucrt.lib(fopen.obj)", импортирован "libf2c.lib(endfile.c.obj)". LINK : warning LNK4217: Символ "fseek", определенный в "libucrt.lib(fseek.obj)", импортирован "libf2c.lib(open.c.obj)" в функции "f_open". LINK : warning LNK4286: Символ "fseek", определенный в "libucrt.lib(fseek.obj)", импортирован "libf2c.lib(err.c.obj)". LINK : warning LNK4286: Символ "fseek", определенный в "libucrt.lib(fseek.obj)", импортирован "libf2c.lib(endfile.c.obj)". LINK : warning LNK4217: Символ "putc", определенный в "libucrt.lib(fputc.obj)", импортирован "libf2c.lib(open.c.obj)" в функции "f__putbuf". LINK : warning LNK4217: Символ "__stdio_common_vsprintf", определенный в "libucrt.lib(output.obj)", импортирован "libf2c.lib(open.c.obj)" в функции "_vsnprintf_l". LINK : warning LNK4286: Символ "__stdio_common_vsprintf", определенный в "libucrt.lib(output.obj)", импортирован "libf2c.lib(wref.c.obj)". LINK : warning LNK4286: Символ "__stdio_common_vsprintf", определенный в "libucrt.lib(output.obj)", импортирован "libf2c.lib(endfile.c.obj)". LINK : warning LNK4217: Символ "strncmp", определенный в "libucrt.lib(strncmp.obj)", импортирован "libf2c.lib(open.c.obj)" в функции "f_open". LINK : warning LNK4217: Символ "malloc", определенный в "libucrt.lib(malloc.obj)", импортирован "libf2c.lib(open.c.obj)" в функции "f__bufadj". LINK : warning LNK4286: Символ "malloc", определенный в "libucrt.lib(malloc.obj)", импортирован "libf2c.lib(f77_aloc.c.obj)". LINK : warning LNK4217: Символ "ftell", определенный в "libucrt.lib(ftell.obj)", импортирован "libf2c.lib(err.c.obj)" в функции "f__nowreading". LINK : warning LNK4286: Символ "ftell", определенный в "libucrt.lib(ftell.obj)", импортирован "libf2c.lib(endfile.c.obj)". LINK : warning LNK4217: Символ "signal", определенный в "libucrt.lib(signal.obj)", импортирован "libf2c.lib(sig_die.c.obj)" в функции "sig_die". LINK : warning LNK4217: Символ "exit", определенный в "libucrt.lib(exit.obj)", импортирован "libf2c.lib(sig_die.c.obj)" в функции "sig_die". LINK : warning LNK4286: Символ "exit", определенный в "libucrt.lib(exit.obj)", импортирован "libf2c.lib(exit_.c.obj)". LINK : warning LNK4217: Символ "abort", определенный в "libucrt.lib(abort.obj)", импортирован "libf2c.lib(sig_die.c.obj)" в функции "sig_die". LINK : warning LNK4217: Символ "atoi", определенный в "libucrt.lib(atox.obj)", импортирован "libf2c.lib(wref.c.obj)" в функции "wrt_E". LINK : warning LNK4217: Символ "fread", определенный в "libucrt.lib(fread.obj)", импортирован "libf2c.lib(endfile.c.obj)" в функции "copy". LINK : warning LNK4217: Символ "fwrite", определенный в "libucrt.lib(fwrite.obj)", импортирован "libf2c.lib(endfile.c.obj)" в функции "copy". libf2c.lib(open.c.obj) : error LNK2019: ссылка на неразрешенный внешний символ __imp_fputs в функции f__putbuf. libf2c.lib(open.c.obj) : error LNK2019: ссылка на неразрешенный внешний символ __imp_rewind в функции f_open. libf2c.lib(endfile.c.obj) : error LNK2001: неразрешенный внешний символ __imp_rewind. libf2c.lib(open.c.obj) : error LNK2019: ссылка на неразрешенный внешний символ __imp_tmpfile в функции f_open. libf2c.lib(endfile.c.obj) : error LNK2001: неразрешенный внешний символ __imp_tmpfile. libf2c.lib(open.c.obj) : error LNK2019: ссылка на неразрешенный внешний символ __imp_access в функции f_open. OLDNAMES.lib(access.obi) : error LNK2001: неразрешенный внешний символ __imp_access. libf2c.lib(err.c.obj) : error LNK2019: ссылка на неразрешенный внешний символ __imp_perror в функции err__fl. libf2c.lib(err.c.obj) : error LNK2019: ссылка на неразрешенный внешний символ __imp_freopen в функции f__nowreading. libf2c.lib(endfile.c.obj) : error LNK2001: неразрешенный внешний символ __imp_freopen. libf2c.lib(close.c.obj) : error LNK2019: ссылка на неразрешенный внешний символ __imp_remove в функции f_clos. OLDNAMES.lib(access.obi) : error LNK2001: неразрешенный внешний символ __imp__access. magma.dll : fatal error LNK1120: неразрешенных внешних элементов: 8 NMAKE : fatal error U1077: D:\Programs\CMake\bin\cmake.exe : возвращенный код "0xffffffff" Stop. NMAKE : fatal error U1077: "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\bin\HostX64\x64\nmake.exe" : возвращенный код "0x2" Stop. NMAKE : fatal error U1077: "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\bin\HostX64\x64\nmake.exe" : возвращенный код "0x2" Stop.

UPD: Found in build.make the command and changed the code:

removed D:\VS_projects\meshroom_with_dependencies\magma_build\CMakeFiles\magma.dir\objects1.rsp
added /MACHINE:X64

So now my command is: D:\Programs\CMake\bin\cmake.exe -E vs_link_dll --intdir=D:\VS_projects\meshroom_with_dependencies\magma\_build\CMakeFiles\magma.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100183~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100183~1.0\x64\mt.exe --manifests -- C:\PROGRA~2\MICROS~3\2019\COMMUN~1\VC\Tools\MSVC\1427~1.291\bin\Hostx64\x64\link.exe /nologo /out:magma.dll /implib:lib\magma.lib /pdb:D:\VS_projects\meshroom_with_dependencies\magma\_build\magma.pdb /dll /version:0.0 /DEF:D:\VS_projects\meshroom_with_dependencies\magma\_build\CMakeFiles\magma.dir\exports.def "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\lib\x64\cudart_static.lib" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\lib\x64\cudadevrt.lib" D:\VS_projects\vcpkg\installed\x64-windows\lib\lapack.lib D:\VS_projects\vcpkg\installed\x64-windows\lib\libf2c.lib D:\VS_projects\vcpkg\installed\x64-windows\lib\openblas.lib "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\lib\x64\cudart.lib" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\lib\x64\cublas.lib" "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\lib\x64\cusparse.lib" kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MACHINE:X64

So the result is: ...failed (exit code 1120) with the following output: LINK : warning LNK4001: не указаны объектные файлы; использованы библиотеки Создается библиотека lib\magma.lib и объект lib\magma.exp LINK : error LNK2001: неразрешенный внешний символ _DllMainCRTStartup. magma.dll : fatal error LNK1120: неразрешенных внешних элементов: 1

alicevision / Meshroom

A guide for building Meshroom and all the Dependenicies #910