search

codex-storage / nim-codex

Decentralized Durability Engine
https://codex.storage
Apache License 2.0
69 stars 25 forks source link

Add Nix Flake with derivation, shell, service #940

Open jakubgs opened 1 month ago

jakubgs commented 1 month ago

It would be helpful to have a Nix flake for this package. The most basic version would just include a build shell, but a a more advanced version would also include a full derivation and also a NixOS service module.

If possible we should also have CI job for our Nix builds.

This was requested by Corey, but Jarrad would also appreciate it.

jakubgs commented 1 month ago

Here is an attempt by Corey to get something working, some parts of the derivation might be useful: codex-flake.nix.txt

corpetty commented 1 month ago

This shell script worked, but it didn't include the CMakeLists.txt patch I had to do: shell.nix.txt

Here's how I updated the CMakeLists.txt file (from Claude.ai):

  1. Open the CMakeLists.txt file in a text editor: nano /home/petty/codex/repos/nim-codex/vendor/nim-leopard/vendor/leopard/CMakeLists.txt

  2. replace the else block (just before the add_library command), add the following:

    else()
check_cxx_compiler_flag("-march=native" CXX_FLAG_MARCH_NATIVE)
check_cxx_compiler_flag("-mtune=native" CXX_FLAG_MTUNE_NATIVE)
check_cxx_compiler_flag("-msse" CXX_FLAG_SSE)
check_cxx_compiler_flag("-msse2" CXX_FLAG_SSE2)
check_cxx_compiler_flag("-msse3" CXX_FLAG_SSE3)
check_cxx_compiler_flag("-mssse3" CXX_FLAG_SSSE3)
check_cxx_compiler_flag("-msse4.1" CXX_FLAG_SSE41)
check_cxx_compiler_flag("-msse4.2" CXX_FLAG_SSE42)
check_cxx_compiler_flag("-mavx" CXX_FLAG_AVX)
check_cxx_compiler_flag("-mavx2" CXX_FLAG_AVX2)

if(CXX_FLAG_MARCH_NATIVE)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=native")
endif()
if(CXX_FLAG_MTUNE_NATIVE)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mtune=native")
endif()
if(CXX_FLAG_SSE)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -msse")
endif()
if(CXX_FLAG_SSE2)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -msse2")
endif()
if(CXX_FLAG_SSE3)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -msse3")
endif()
if(CXX_FLAG_SSSE3)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mssse3")
endif()
if(CXX_FLAG_SSE41)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -msse4.1")
endif()
if(CXX_FLAG_SSE42)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -msse4.2")
endif()
if(CXX_FLAG_AVX)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mavx")
endif()
if(CXX_FLAG_AVX2)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mavx2")
endif()

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -Werror")
endif()
veaceslavdoina commented 1 month ago

We are shipping two binaries right now codex and crdl (Circuit download tool)

https://github.com/codex-storage/nim-codex/blob/17f0988fc754b85d9f9cd7f6b7f947a1e36ec39e/.github/workflows/release.yml#L84-L87

veaceslavdoina commented 2 weeks ago

@markoburcul, the following lines in Makefile

+ CXXFLAGS ?= -std=c++17 -mssse3
+ export CXXFLAGS

broke the builds on Linux arm64 with the error - c++: error: unrecognized command-line option ‘-mssse3’

more details ![Image](https://github.com/user-attachments/assets/c21e6b28-5c03-4c07-94c3-83ac5931b245) ```shell CMake Error at /usr/share/cmake-3.22/Modules/CMakeTestCXXCompiler.cmake:62 (message): The C++ compiler "/usr/bin/c++" is not able to compile a simple test program. It fails with the following output: Change Dir: /opt/nim-codex/vendor/nim-leveldbstatic/build/CMakeFiles/CMakeTmp Run Build Command(s):/usr/bin/gmake -f Makefile cmTC_dc503/fast && gmake[1]: Entering directory '/opt/nim-codex/vendor/nim-leveldbstatic/build/CMakeFiles/CMakeTmp' /usr/bin/gmake -f CMakeFiles/cmTC_dc503.dir/build.make CMakeFiles/cmTC_dc503.dir/build gmake[2]: Entering directory '/opt/nim-codex/vendor/nim-leveldbstatic/build/CMakeFiles/CMakeTmp' Building CXX object CMakeFiles/cmTC_dc503.dir/testCXXCompiler.cxx.o /usr/bin/c++ -std=c++17 -mssse3 -o CMakeFiles/cmTC_dc503.dir/testCXXCompiler.cxx.o -c /opt/nim-codex/vendor/nim-leveldbstatic/build/CMakeFiles/CMakeTmp/testCXXCompiler.cxx c++: error: unrecognized command-line option ‘-mssse3’ gmake[2]: *** [CMakeFiles/cmTC_dc503.dir/build.make:78: CMakeFiles/cmTC_dc503.dir/testCXXCompiler.cxx.o] Error 1 gmake[2]: Leaving directory '/opt/nim-codex/vendor/nim-leveldbstatic/build/CMakeFiles/CMakeTmp' gmake[1]: *** [Makefile:127: cmTC_dc503/fast] Error 2 gmake[1]: Leaving directory '/opt/nim-codex/vendor/nim-leveldbstatic/build/CMakeFiles/CMakeTmp' CMake will not be able to correctly generate this project. Call Stack (most recent call first): CMakeLists.txt:7 (project) ```
veaceslavdoina commented 2 weeks ago

mssse3 are x86 specific and we should not use them for arm