Cyan4973 / FiniteStateEntropy

New generation entropy codecs : Finite State Entropy and Huff0
BSD 2-Clause "Simplified" License
1.33k stars 143 forks source link

Sync with zstd units 'debug', 'hist', 'cpu' and 'xxhash'. #116

Closed TheVice closed 7 months ago

TheVice commented 7 months ago

Hello.

Some of units (debug, hist, cpu, xxhash) from zstd repository can be safe renew same one's at this repository. Only changes for the hist - paths at the includes still should be own in this project.

Also not know for sure about copyright header - should I keep original in this project or should be provided as is from the zstd repository.

Thank you.

Cyan4973 commented 7 months ago

What are the benefits from updating these units ?

Also, note that the xxhash library can be taken directly from Cyan4973/xxhash, this is probably proper as it would avoid transporting zstd-specific settings.

TheVice commented 7 months ago

Historical I see that process was in the past - so I suggest after the CI was added - that can be provided again, with checking how effect it on this project.

I assume that zstd more covered on testing and benchmarking, so expect that such update made security and performance improves for this project.

And that also add sure - that while reading Zstandard specification in part where FSE describe and when read this project code - we have most sync state as possible.

If that is not a goal - that not necessary.

Ye, I know that xxhash have own repository but assume that it also periodically syncing with zstd, where actually made recent change in that unit.

P.S. By the way - xxhash project - is the place where we was meet first time ;) And I learned about the project from the Zstandard specification.

TheVice commented 7 months ago

OK, so let's close it for now.

By the way, it currently not checking, but proposed change, at the xxhash unit, will break compatibility with build tools that present at the Ubuntu 14.04 for example. I see that code in this repository support very old standard of C, so probably that moment with breaking is not acceptable.

For my understanding, if continue to sync with that repository, relationship with first one probably next, with special notes to the source from the programs folder. zstd FiniteStateEntropy
N/A FiniteStateEntropy/lib/Archives/hufx6.h
N/A FiniteStateEntropy/lib/Archives/hufx6_decompress.c
zstd/lib/common/bitstream.h FiniteStateEntropy/lib/bitstream.h
zstd/lib/common/compiler.h FiniteStateEntropy/lib/compiler.h
zstd/lib/common/debug.c FiniteStateEntropy/lib/debug.c
zstd/lib/common/debug.h FiniteStateEntropy/lib/debug.h
zstd/lib/common/entropy_common.c FiniteStateEntropy/lib/entropy_common.c
zstd/lib/common/error_private.h FiniteStateEntropy/lib/error_private.h
N/A FiniteStateEntropy/lib/error_public.h
zstd/lib/common/fse.h FiniteStateEntropy/lib/fse.h
N/A FiniteStateEntropy/lib/fseU16.c
N/A FiniteStateEntropy/lib/fseU16.h
zstd/lib/compress/fse_compress.c FiniteStateEntropy/lib/fse_compress.c
zstd/lib/common/fse_decompress.c FiniteStateEntropy/lib/fse_decompress.c
zstd/lib/compress/hist.c FiniteStateEntropy/lib/hist.c
zstd/lib/compress/hist.h FiniteStateEntropy/lib/hist.h
zstd/lib/common/huf.h FiniteStateEntropy/lib/huf.h
zstd/lib/compress/huf_compress.c FiniteStateEntropy/lib/huf_compress.c
zstd/lib/decompress/huf_decompress.c FiniteStateEntropy/lib/huf_decompress.c
zstd/lib/common/mem.h FiniteStateEntropy/lib/mem.h
N/A FiniteStateEntropy/programs/bench.c
N/A FiniteStateEntropy/programs/bench.h
N/A FiniteStateEntropy/programs/commandline.c
zstd/lib/common/cpu.h FiniteStateEntropy/programs/cpu.h
zstd/programs/fileio.c FiniteStateEntropy/programs/fileio.c
zstd/programs/fileio.h FiniteStateEntropy/programs/fileio.h
N/A FiniteStateEntropy/programs/fseDist.c
N/A FiniteStateEntropy/programs/fseDist.h
zstd/tests/fullbench.c FiniteStateEntropy/programs/fullbench.c
zstd/tests/fuzzer.c FiniteStateEntropy/programs/fuzzer.c
N/A FiniteStateEntropy/programs/fuzzerHuff0.c
N/A FiniteStateEntropy/programs/fuzzerU16.c
N/A FiniteStateEntropy/programs/probaGenerator.c
zstd/lib/common/xxhash.c FiniteStateEntropy/programs/xxhash.c
zstd/lib/common/xxhash.h FiniteStateEntropy/programs/xxhash.h
N/A FiniteStateEntropy/programs/zlibh.c
N/A FiniteStateEntropy/programs/zlibh.h

Notes for programs - units from that folder have only same names, but functionally in most cases very different.

Thank you for your time.