Possible to provide standalone source code files?

[MatthiasPetermann]

Hello, I recently came across the PAQ file format and the family of archiving tools associated with it. I've been quite satisfied with zpaq715 so far, and now I've become aware of zpaqfranz. What I particularly like is that some features have been added here that I found missing in zpaq, such as the ability to quickly get information about all saved versions in the form of a table. Great work, and it's nice to see active development happening here.

It's only been a few days, but I already think that zpaq might be my personal discovery of the year. I can hardly believe I didn't come across it sooner. At this point, I'd like to ask a question: the fact that the source code on GitHub has been merged into a single CPP file makes it very easy to compile it on various platforms. It was also straightforward to get a current version running on my FreeBSD. One small drawback is that navigating in the large file is quite cumbersome if you want to understand how the program works. It seems to be a generated compilation of a series of standalone source code files. Is there a possibility that these source code files or the script to create the compiled file could also be uploaded to GitHub?

Thank you and best regards, Matthias

[fcorbelli]

Hello, ...and it's nice to see active development happening here.

I am almost ready to release a specific function to detect (already in zpaqfranz) and RECOVER (first time ever) from SHA-1 collisions

(...) One small drawback is that navigating in the large file is quite cumbersome if you want to understand how the program works. It seems to be a generated compilation of a series of standalone source code files. Is there a possibility that these source code files or the script to create the compiled file could also be uploaded to GitHub?

Actually no, it is not created from several different sources through some script, because I had to modify them as I went along to make them compatible, even in very subtle situations (e.g., macros called off and variables always called off). It's "handwork"

At the beginning of the source there are the individual portions of the software, and where I got them from For example

14 [BSD 2-Clause license]         xxHash Copyright (C) 2012-2020 Yann Collet          https://github.com/memcached/memcached/blob/master/xxhash.h
 /// LICENSE_START.14
 /// LICENSE_END.14

If you search for LICENSE_START.14 and LICENSE_END.14 you'll find this example

zpaqfranz is made by zpaq.715 UNION (other software, like hashers, fixed by hand) UNION my own code (it is easy to spot, by style and sometimes even by italian variables too 😄 )

Therefore no BUT If you're curious, all you have to do is ask, and I'll answer to the best of my ability

[ghost]

Even if it was a "handwork" it could be beneficial to potential users of the source code (not to the end users of binaries, of course, but to the people who want to read the code, understand it or modify it) to have the original sources from different origins in separate files, and then for each release to, like the author of the question imagined, "produce a generated compilation of a series of standalone source code files."

If I understood correctly, as the precedence, that the original zpaq already embedded libdivsufsort (with the MIT license) in the libzpaq.cpp at some point in time of its development, but the resulting libzpaq.cpp is still some order of magnitude smaller (I think around 200K or less then) , and the original author could have had additional reasons for his solution as he, as far as I understand, planned to change the license of his own work and publish something like a "minimal but sufficient" version of libzpaq.cpp and of zpaq.cpp using the library, intentionally removing already implemented features or simplifying the code to achieve similar effect but in less "fixed" parts including less interface options, and then stop his own contributions, leaving a version "good enough and minimal enough to be left alone." It was consistent with author's vision and approach, and the resulting library is still a masterpiece (even having its own JIT (!) compiling engine for x86 and x64 in that 200K library, allowing it to, if I understand, even today define additional compression algorithms externally of the library, still to be executed as the native machine x86 or x64 code. And I don't know what the state of all that is for the cases of ARM-based processors. But if one would want to develop in that direction, one would probably be motivated to split even the libzpaq to smaller files, at least before a "very stable" version is reached again.

I understand that there's never a single answer and single perspective to some organizing decisions, but my vote would be for split files for the projects that intend to not be "frozen in time" or where the author would be interested in the contributions from others.

[fcorbelli]

Even if it was a "handwork" (...) to have the original sources from different origins in separate files (...)

It is practically impossible to understand how it works, and how it was developed, "top down" The way I think is best is to start from the development forum, and from the various threads "bottom up". Then study the older versions to get an idea of ​​the direction of development

If I understood correctly (...) And I don't know what the state of all that is for the cases of ARM-based processors.

I can answer: use -DNOJIT for arm CPU-like, -DBIG for PowerPCs, -DANCIENT etc Evertyhing is in the source code (starting comments)

But if one would want to develop in that direction, one would probably be motivated to split even the libzpaq to smaller files, at least before a "very stable" version is reached again.

I admit I'm not very interested. It's a burdensome job, and of little importance (for me)

I understand that there's never a single answer and single perspective to some organizing decisions, but my vote would be for split files for the projects that intend to not be "frozen in time" or where the author would be interested in the contributions from others.

Historically zpaq has had very little feedback, zpaqfranz even less, in the order of a few dozen

The time it would require of me could be used for additional and useful functions

If anyone is really interested, all they need to do is read Mahoney's documentation and the source for version 7.15, zpaqd and unzpaq 206

[ghost]

Thanks @fcorbelli . @MatthiasPetermann what is your use case where you would prefer to use more smaller files? Do you need some specific new feature?

[ghost]

(Not directly connected with this topic, but slightly tangential as it's hard for me to figure out locally. I also don't know if it's "expected" as I think I've read somewhere that it's less tested on Intel: on my machine "hw" version with the -hw parameter seems to always crash. The technologies reported by cpuz: Instruction sets MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, EM64T, VT-x, AES, AVX )

[fcorbelli]

The hw use asm code for sha1 Works only on CPU that support SHA1 istructions Like AMD or newer Intel

[ghost]

So does -hw ignore what the cpuid returns? I've tried to call the piece of your code ihavehw() in an independent program and it seems it detect 0 for SHA?

1838: new ecx 2142954431
1843: new ebx 641
SSSE3 :OK
SSE41 :OK
SHA   :NO

[fcorbelli]

No cpuid test for zpaqfranzhw It is up to you Your cpu does not support sha1 Therefore zpaqfranzhw -hw will crash

Standard zpaqfranz autodetect hw support, even on Linux, running walton's code (No zpaqfranzhw on Linux) Zpaqfranzhw use 7z code by Pavlov authorization

Everything is in the source

[ghost]

Thanks. I could find the relevant statement in the code now:

 For performances reason, no run-time CPU compatibility checks,

Do you know the measurable difference of cpuid being executed even if the cpuid is to be run only once for the whole time of the program (just before the first call to the functions which depend on the new instruction set)? I believed everything else (as an example, actually calculating even one checksum) would have to take much more time than e.g. two cpuid instructions.

[fcorbelli]

Of course I know Measured too But the problem is not here, but in checking a boolean flag every time, disrupting cache loading, even with a preferred register (not honored by all compiler) You can try yourself And you'll find that cpuid is faster than boolean

Btw I use a specific AMD build. Maybe I'll relase

[fcorbelli]

Rather OT but... zpaqfranzhw (aka: -DHWSHA1) has a very small performance increase, compared with "normal" zpaqfranz. I report some numerical data on my test set

zpaqfranzhw 58.11 (AMD native only) 37.046
zpaqfranzhw 58.11 (AMD native)      37.157/37.437 (12)
zpaqfranz   58.11 (generic)         37.844
zpaqfranz   54.10 (old release)     43.344

The AMD-only code is about 3% faster than the "generic" one
Or, the "generic" is just as fast as specific asm code (on the same HW accelerated hardware) 😄 , running (about) 13% faster than the older releases (aka: zpaq 7.15 implementation) Therefore some die-hard user can "play" with the .asm, but without a big gain

=> short version: use zpaqfranz.exe 🥇