Request for Help - Githubissues

Loki-Astari commented 3 months ago

I posted on Reddit a couple of days ago about a comparison of C++ JSON libraries.

There were several comments that I missed your library. Any chance you want to help and add your library?

I have simplified the project in a branch to make it easy for you to add the appropriate code:

> git clone git@github.com:Loki-Astari/JsonBenchmark.git
> cd JsonBenchmark
> git checkout JsonPerformance
> ./configure
> make
> ./runOneTest all

RealTimeChris commented 2 months ago

Alright it should be up to date/working as good as it's going to for now. Let me know what you think. At this branch: https://github.com/RealTimeChris/JsonBenchmark/tree/AddJsonifier

Loki-Astari commented 2 months ago

Build Errors now:

Building Dir Memory Finish
Building Dir ThirdParty Start
Makefile:6: ../../thirdparty/Jsonifier/avx_flags.txt: No such file or directory
make[2]: *** No rule to make target `../../thirdparty/Jsonifier/avx_flags.txt'.  Stop.
make[1]: *** [ThirdParty.dir] Error 2

Loki-Astari commented 2 months ago

My mistake.

You added an init script. Now it works.

Loki-Astari commented 2 months ago

Just noticed your project is on this branch:

commit e444ef5d9a4d79e284ae65d4bb73abd71012e525 (HEAD, origin/dev)

Do you want me to leave it here. Or do you want to merge that into main first?

RealTimeChris commented 2 months ago

Can you please use that/the dev branch?

Loki-Astari commented 2 months ago

No problem.

Loki-Astari commented 2 months ago

You can delete hashes.

The git submodule system works on hashs. If you delete a hash the submodule will not update correctly.

This currently breaking the build as you keep doing updates.

Please tag the current head of development so that github does not delete it. Then you can continue to modify the dev branch as you see fit.

Loki-Astari commented 2 months ago

Currently dev is b317b993a220fc3cde0c7211181ebc0c16ba563f

RealTimeChris commented 2 months ago

Wdym "tag current head" of development?

Loki-Astari commented 2 months ago

Your build is currently breaking:

s -Wno-missing-braces -Wno-error -Wno-class-memaccess -I../../thirdparty/Jsonifier/Include/ JsonifierTest.cppERROR
g++ -c JsonifierTest.cpp -o release/JsonifierTest.o -I /opt/homebrew/include -I /opt/homebrew/include -fPIC -Wall -Wextra -Werror -Wstrict-aliasing -pedantic -Wunreachable-code -Wno-long-long -Wdeprecated -Wdeprecated-declarations -Wmissing-braces -Wmissing-field-initializers -Wunused-variable -Winconsistent-missing-override -Wdelete-non-abstract-non-virtual-dtor -Wdelete-non-virtual-dtor -Wno-deprecated-register -Wno-literal-range -Wno-unknown-pragmas -I/Users/martinyork/Repo/JsonBenchmark/build/include -isystem /Users/martinyork/Repo/JsonBenchmark/build/include3rd -O3 -Wno-unreachable-code -DTHOR_USE_CPLUSPLUS23 -std=c++2b -DMOCK_FUNC(x)=::x -DMOCK_TFUNC(x)=::x -Wno-logical-op-parentheses -Wno-unknown-warning-option -Wno-ignored-attribute -Wno-unused-parameter -Wno-unused-local-typedefs -Wno-deprecated -Wno-unused-variable -Wno-pedantic -Wno-ignored-qualifiers -Wno-missing-braces -Wno-error -Wno-class-memaccess -I../../thirdparty/Jsonifier/Include/
========================================
In file included from JsonifierTest.cpp:4:
src/ThirdParty/./JsonifierTypes.h:7:35: error: use of undeclared identifier 'geometry_data'
template<> struct jsonifier::core<geometry_data> {
                                  ^

Loki-Astari commented 2 months ago

In guthub if you rebase and or mess around with the branch. Any hashs that are orphaned (ie head not explicitly taged will be purged).

You can prevent this by adding either a tag.

git tag XXXX

Or by marking it as a branch

git branch XXXX

Does not matter what the name is.

Loki-Astari commented 2 months ago

Note the hash e444ef5d9a4d79e284ae65d4bb73abd71012e525 no longer exists in github this was the working version.

RealTimeChris commented 2 months ago

chris@DESKTOP-SUABLEI:~$ git clone https://github.com/RealTimeChris/JsonBenchmark --branch=AddJsonifier --depth=1 ./Test23
Cloning into './Test23'...
remote: Enumerating objects: 234, done.
remote: Counting objects: 100% (234/234), done.
remote: Compressing objects: 100% (161/161), done.
remote: Total 234 (delta 15), reused 184 (delta 5), pack-reused 0 (from 0)
Receiving objects: 100% (234/234), 925.02 KiB | 4.51 MiB/s, done.
Resolving deltas: 100% (15/15), done.
chris@DESKTOP-SUABLEI:~$ cd Test23
chris@DESKTOP-SUABLEI:~/Test23$ ./configure
Submodule 'build' (https://github.com/Loki-Astari/ThorMaker.git) registered for path 'build'
Cloning into '/home/chris/Test23/build'...
Submodule path 'build': checked out '02c427c6b85519a106dcf6a561c9d92d7ddb826b'
Submodule 'googletest' (https://github.com/Loki-Astari/googletest.git) registered for path 'build/googletest'
Submodule 'vera-plusplus' (https://github.com/Loki-Astari/vera-plusplus.git) registered for path 'build/vera-plusplus'
Cloning into '/home/chris/Test23/build/googletest'...
Cloning into '/home/chris/Test23/build/vera-plusplus'...
Submodule path 'build/googletest': checked out 'cc366710bbf40a9816d47c35802d06dbaccb8792'
Submodule path 'build/vera-plusplus': checked out 'de33164885c90951be29779aea7c236b817deb13'
Checking Windows Symbolic Links: Linux
    Not Windows
checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C++... yes
checking whether g++ accepts -g... yes
checking for g++ option to enable C++11 features... none needed
STARTING COVERAGE CHECK
13.2.0
13.2.0
Submodule 'thirdparty/Jsonifier' (https://github.com/RealTimeChris/Jsonifier) registered for path 'thirdparty/Jsonifier'
Cloning into '/home/chris/Test23/thirdparty/Jsonifier'...
remote: Enumerating objects: 16, done.
remote: Counting objects: 100% (16/16), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 16 (delta 9), reused 16 (delta 9), pack-reused 0 (from 0)
Unpacking objects: 100% (16/16), 3.80 MiB | 5.21 MiB/s, done.
From https://github.com/RealTimeChris/Jsonifier
 * branch            5a83b5ec8db8e9a380d03692ad659053e4ba6d1a -> FETCH_HEAD
Submodule path 'thirdparty/Jsonifier': checked out '5a83b5ec8db8e9a380d03692ad659053e4ba6d1a'
checking whether C++ compiler accepts -Wno-unused-private-field... yes
checking whether C++ compiler accepts -Wno-deprecated-register... yes
checking whether C++ compiler accepts -Winconsistent-missing-override... no
./configure: line 3973: -Werror: command not found
checking whether C++ compiler accepts -Wdelete-non-abstract-non-virtual-dtor... no
./configure: line 4017: -Werror: command not found
checking whether C++ compiler accepts -Wdelete-non-virtual-dtor... yes
checking whether C++ compiler accepts -Wno-literal-suffix... yes
checking whether C++ compiler accepts -Wno-literal-range... yes
checking whether C++ compiler accepts -std=c++11... yes
checking whether C++ compiler accepts -std=c++14... yes
checking whether C++ compiler accepts -std=c++17... yes
checking whether C++ compiler accepts -std=c++20... yes
checking whether C++ compiler accepts -std=c++23... yes
checking whether C++ compiler accepts -std=c++1x... no
checking whether C++ compiler accepts -std=c++1y... yes
checking whether C++ compiler accepts -std=c++1z... yes
checking whether C++ compiler accepts -std=c++2a... yes
checking whether C++ compiler accepts -std=c++2b... yes
checking for bzip2... bzip2
~/Test23/build/third ~/Test23
~/Test23/build/googletest/googletest ~/Test23/build/third
ar: creating libgtest.a
a - gtest-all.o
a - gtest_main.o
~/Test23/build/third
Installing google test
pushd /home/chris/Test23/build
~/Test23/build ~/Test23/build/third
mkdir -p bin include include3rd lib share/man
rm -f lib/libgtest.*
cp /home/chris/Test23/build/googletest/googletest/libgtest.a lib/
rm -f include3rd/gtest
ln -s /home/chris/Test23/build/googletest/googletest/include/gtest include3rd/gtest
~/Test23/build/third
~/Test23
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking how to print strings... printf
checking for gcc... gcc
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking whether gcc understands -c and -o together... yes
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for file... file
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking for gawk... gawk
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /usr/bin/dd
checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1
checking for mt... no
checking if : is a manifest tool... no
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking how to run the C++ preprocessor... g++ -E
checking for ld used by g++... /usr/bin/ld -m elf_x86_64
checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC -DPIC
checking if g++ PIC flag -fPIC -DPIC works... yes
checking if g++ static flag -static works... yes
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... (cached) GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a race-free mkdir -p... /usr/bin/mkdir -p
checking whether make sets $(MAKE)... yes
checking whether make supports the include directive... yes (GNU style)
checking whether make supports nested variables... yes
Checking Windows Symbolic Links: Linux
    Not Windows
./configure: line 17785: pushd: third: No such file or directory
checking dependency style of gcc... none
checking dependency style of g++... none
checking that generated files are newer than configure... done
: creating ./config.status
config.status: creating Makefile.extra
config.status: creating Makefile.config
config.status: WARNING:  'build/autotools/build/Makefile.config.in' 'Makefile.config.in' seems to ignore the --datarootdir setting
config.status: creating config.h
config.status: creating src/benchmark/benchmarkConfig.h
config.status: creating src/Memory/MemoryConfig.h
config.status: executing libtool commands
config.status: executing depfiles commands
chris@DESKTOP-SUABLEI:~/Test23$ make
Building Dir src Start
Building Dir Memory Start
Building release
g++ -c -O3  Memory.cpp                                                                             OK
g++ -shared -o release/libMemory.so -O3                                                            ^Cmake[3]: *** [/home/chris/Test23/build/tools/main.Makefile:552: release/libMemory.so] Interrupt
make[2]: *** [/home/chris/Test23/build/tools/main.Makefile:391: release] Interrupt
make[1]: *** [/home/chris/Test23/build/tools/Project.Makefile:81: Memory.dir] Interrupt
make: *** [/home/chris/Test23/build/tools/Project.Makefile:81: src.dir] Interrupt

chris@DESKTOP-SUABLEI:~/Test23$ ./init3rdParty
Initializing: Jsonifier
-- The CXX compiler identification is GNU 13.2.0
^[[A^[[A-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The C compiler identification is GNU 13.2.0
-- The CXX compiler identification is GNU 13.2.0

-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (1.3s)
-- Generating done (0.0s)
-- Build files have been written to: /home/chris/Test23/thirdparty/Jsonifier/CMake/Build
[ 50%] Building CXX object CMakeFiles/feature_detector.dir/main.cpp.o
[100%] Linking CXX executable feature_detector
[100%] Built target feature_detector
CPU Brand: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
-- Instruction Set Found: LzCnt
-- Instruction Set Found: PopCnt
-- Instruction Set Found: Bmi
-- Instruction Set Found: Bmi2
-- Instruction Set Found: Avx2
-- Configuring done (2.7s)
-- Generating done (0.0s)
-- Build files have been written to: /home/chris/Test23/thirdparty/Jsonifier/Build
AVX_FLAGS written to avx_flags.txt
DONE

chris@DESKTOP-SUABLEI:~/Test23$ make
Building Dir src Start
Building Dir Memory Start
Building release
g++ -shared -o release/libMemory.so -O3                                                            OK
 Done Building Memory release/libMemory.so
Start Installing Headers
 Install Header Memory.h
 Install Header MemoryConfig.h
End Installing Headers
Start Installing Libraries
 Install - release - libMemory23.so
End Installing Libraries
Building Dir Memory Finish
Building Dir ThirdParty Start
Building release
g++ -c -O3 -mlzcnt -mpopcnt -mbmi -mbmi2 -mavx2 -mavx -mlzcnt -mpopcnt -mbmi -mbmi2 -O3 -Wno-unused-parameter -Wno-unused-local-typedefs -Wno-deprecated -Wno-unused-variable -Wno-pedantic -Wno-ignored-qualifiers -Wno-missing-braces -Wno-error -Wno-class-memaccess -I../../thirdparty/Jsonifier/Include/ JsonifierTest.cppOK
g++ -c -O3  TestManager.cpp                                                                        OK
g++ -c -O3  test.cpp                                                                               OK
g++ -shared -o release/libThirdParty.so -O3                                                        OK
 Done Building ThirdParty release/libThirdParty.so
Start Installing Headers
 Install Header JsonifierTypes.h
 Install Header TestManager.h
 Install Header TypeSafe.h
 Install Header TypeSafeCatalog.h
 Install Header TypeSafeCountry.h
 Install Header TypeSafePass01.h
 Install Header TypeSafePass02.h
 Install Header TypeSafePass03.h
 Install Header TypeSafeRound09.h
 Install Header TypeSafeRound10.h
 Install Header TypeSafeTwitter.h
 Install Header test.h
End Installing Headers
Start Installing Libraries
 Install - release - libThirdParty23.so
End Installing Libraries
Building Dir ThirdParty Finish
Building Dir benchmark Start
Building release
g++ -c -O3  PerformanceChecker.cpp                                                                 OK
g++ -c -O3  TestSuite.cpp                                                                          OK
g++ -c -O3  benchmark.cpp                                                                          OK
g++ -o release/benchmark.prog -O3                                                                  OK
 Done Building release/benchmark
g++ -c -O3  memory.cpp                                                                             OK
g++ -o release/memory.prog -O3                                                                     OK
 Done Building release/memory
Start Installing Applications
 Install - release - benchmark23
 Install - release - memory23
End Installing Applications
Building Dir benchmark Finish
Building Dir src Finish
chris@DESKTOP-SUABLEI:~/Test23$

RealTimeChris commented 2 months ago

https://github.com/RealTimeChris/Jsonifier/releases/tag/Testing Here is the tag.

Loki-Astari commented 2 months ago

I must be doing something wrong. Its late I will try again in the morning after I get some sleep.

RealTimeChris commented 2 months ago

Note that I added some new types in the TypeSafeCountry.h etc files. I looked into your actions history and this seems to be the reason it's failing to build.

Loki-Astari commented 2 months ago

OK. Got it. I have your specific types into their own files which I included from your JsonifierTest.cpp file.

Loki-Astari commented 2 months ago

OK. Not sure what is going wrong. It builds but segfaults when running performance/canada

I tried your original branch and it works there. I was wondering if you can have a look at my branch and see what I messed up.

> Project: git@github.com:Loki-Astari/JsonBenchmark.git
> Branch: JsonifierAttempt2
> Test with: ./runOneTest Jsonifier performance/canada

I was carefull and the only change I made as moving your types out of the TypeSafe files into their own files.

https://github.com/Loki-Astari/JsonBenchmark/actions/runs/10744216808/job/29800648631

Loki-Astari commented 2 months ago

Results are published but your code still crashes on the Canada data.

RealTimeChris commented 2 months ago

When I try to run make.

RealTimeChris commented 2 months ago

Alright so I may have fixed it by changing the JsonifierTypes.h file to this:

template<typename T>
class GetValue: public TestAction
{

  public:
      virtual bool Parse(const char* json, size_t length , std::unique_ptr<ParseResultBase>& reply) const
    {
          std::unique_ptr<GetValueResult<T>>    parsedData = std::make_unique<GetValueResult<T>>();
          auto error = parser.parseJson(parsedData->data, jsonifier::string_view{ json, length });
          if (error) [[likely]] {
              reply = std::move(parsedData);
              return true;
          }
        return true;
    }
    virtual bool Stringify(const ParseResultBase& parsedData, std::unique_ptr<StringResultBase>& reply)  const
    {
        return Prettify(parsedData, reply);
    }
    virtual bool Prettify(const ParseResultBase& parsedData, std::unique_ptr<StringResultBase>& reply) const
    {
        GetValueResult<T>const& parsedDataInput = dynamic_cast<GetValueResult<T> const&>(parsedData);
        std::unique_ptr<StringResultUsingString>    output = std::make_unique<StringResultUsingString>();
        auto error = parser.serializeJson < jsonifier::serialize_options{ .prettify = true } > (parsedDataInput.data, output->result);
        if (error) [[likely]] {
            reply = std::move(output);
        }
        return true;
    }
};

Loki-Astari commented 2 months ago

I have a fix in place.

For parse you are equal 6th place. For Stringify you are 3rd. For Prettify you are 2nd.

Those are solid numbers. Good job. Your code solidly beats mine in all categories and compared to all others you use significantly less memory.

RealTimeChris / Json-Performance

Request for Help #14