[info] OpenBSD support

[dertuxmalwieder]

FYI, re:supported systems,

unsurprisingly, zpaqfranz 54.15 works just fine on OpenBSD 7.1. :-)

(I built a port for it, but it's not published yet.)

[fcorbelli]

I'm almost ready to release the new version, 55.1, which introduces a function that didn't exist before (change from 54 to 55 is motivated for this reason) Using some sort of ramdisk (it's actually fake, nothing more than memory allocated with malloc ()). There is a specific function to take the amount of "free" RAM. I'm not quite sure if it works with OpenBSD. Please try. Just run b (benchmark) and it will show you (maybe) the amount of RAM (or an error). 55_1f.zip Thank you

EDIT: a quick try on OpenBSD 6.6 fail, the main problem is

sysctlbyname

and others (gcc 4.2.1) I normally use FreeBSD, rarely OpenBSD Can you please explain how you compile 54.15?

[dertuxmalwieder]

Can you please explain how you compile 54.15?

Just using make works rather well if you use the system’s own clang++. (I had to patch your Makefile to replace g++. Makes no sense on OpenBSD.) No errors appeared during compilation and running.

[fcorbelli]

In fact no (I am talking about 55.1)

root:/tmp/zp# clang++ -Dunix -O3 -march=native zpaqfranz.cpp -o zpaqfranz -pthread
zpaqfranz.cpp:36360:2: error: use of undeclared identifier 'sysctlbyname'
        sysctlbyname("vm.stats.vm.v_free_count", &free, &size, NULL, 0);
        ^
zpaqfranz.cpp:36361:2: error: use of undeclared identifier 'sysctlbyname'
        sysctlbyname("vm.stats.vm.v_inactive_count", &inactive, &size, NULL, 0);
        ^
zpaqfranz.cpp:36363:2: error: use of undeclared identifier 'sysctlbyname'
        sysctlbyname("vm.stats.vm.v_page_size", &pagesize, &size, NULL, 0);

Some kind of OpenBSD function needed

[dertuxmalwieder]

I haven’t tried 55.1 yet. I’ll investigate after the meeting that is set to happen in a few minutes. :-)

[fcorbelli]

Thank you There is a new function the get the "reasonably" free RAM, I'll need an OpenBSD version

#ifdef BSD  // BSD or Mac OS/X
    int pagesize    =0;
    int inactive    =0;
    int free        =0;
    size_t size     = sizeof free;
    sysctlbyname("vm.stats.vm.v_free_count", &free, &size, NULL, 0);
    sysctlbyname("vm.stats.vm.v_inactive_count", &inactive, &size, NULL, 0);
    size = sizeof pagesize;
    sysctlbyname("vm.stats.vm.v_page_size", &pagesize, &size, NULL, 0);
    return (free+inactive)*pagesize;/// / 1024 ;
#else /// Linux or something similar

Happy meeeting!

[fcorbelli]

Please test the 55.1g 55_1g.zip

You can see the free ram in b (benchmark)

zpaqfranz b

Thanks

[dertuxmalwieder]

Looks almost good to me:

$ uname -smr
OpenBSD 7.1 amd64
$ clang++ -Dunix -O3 -march=native zpaqfranz.cpp -o zpaqfranz -pthread
$ ./zpaqfranz b
zpaqfranz v55.1g-experimental archiver,  compiled Jul  7 2022
RAM seems 18.446.744.072.472.489.984       ###<<< (Or does it? ;-))
Benchmarks: XXHASH64 XXH3 SHA-1 SHA-256 BLAKE3 CRC-32 CRC-32C WYHASH WHIRLPOOL MD5 SHA-3 NILSIMSA
Time limit          5 s  (-n X)
Chunks of     390.62 KB  (-minsize Y)

00000005 s     XXHASH64: speed (    1.86 GB/s)
00000005 s         XXH3: speed (    2.41 GB/s)
00000005 s        SHA-1: speed (  285.34 MB/s)
00000005 s      SHA-256: speed (   80.92 MB/s)
00000005 s       BLAKE3: speed (  183.94 MB/s)
00000005 s       CRC-32: speed (    2.71 GB/s)
00000005 s      CRC-32C: speed (  960.08 MB/s)
00000005 s       WYHASH: speed (    2.50 GB/s)
00000005 s    WHIRLPOOL: speed (   60.73 MB/s)
00000005 s          MD5: speed (  302.35 MB/s)
00000005 s        SHA-3: speed (  184.94 MB/s)
00000005 s     NILSIMSA: speed (    2.70 GB/s)

Results:

   WHIRLPOOL:    60.73 MB/s (done   303.65 MB)
     SHA-256:    80.92 MB/s (done   404.74 MB)
      BLAKE3:   183.94 MB/s (done   919.73 MB)
       SHA-3:   184.94 MB/s (done   924.69 MB)
       SHA-1:   285.34 MB/s (done     1.39 GB)
         MD5:   302.35 MB/s (done     1.48 GB)
     CRC-32C:   960.08 MB/s (done     4.69 GB)
    XXHASH64:     1.86 GB/s (done     9.28 GB)
        XXH3:     2.41 GB/s (done    12.06 GB)
      WYHASH:     2.50 GB/s (done    12.52 GB)
    NILSIMSA:     2.70 GB/s (done    13.50 GB)
      CRC-32:     2.71 GB/s (done    13.53 GB)

franzomips single thread index 1.367 (quick CPU check, raw 1.367)
Atom N2800      (phy)  4     336.05 %
Xeon E3 1245 V2 (vir)  4      56.63 %
i5-6200U        (phy)  2      71.87 %
Xeon E5 2620 V4 (phy)  8      74.01 %
Xeon E5 2630 V4 (phy) 10      88.13 %
Xeon D-1541     (vir)  8      67.34 %
i5-3570         (phy)  4      46.27 %
i7-4790K        (phy)  4      41.89 %
i7-8700K        (phy)  6      40.73 %
i9-10900        (phy) 10      36.92 %
AMD-5950X       (phy) 16      28.51 %

60.005 seconds (000:01:00)  (all OK)

Thank you for the quick response. Looking forward to the release! (I found ZPAQ just yesterday and I'm already impressed.)

[fcorbelli]

MMmhhh... not good at all

RAM seems 18.446.744.072.472.489.984

Some other digging needed (I am writing the Solaris version now)

[dertuxmalwieder]

I suspect uvmexp could help: https://gist.github.com/leiless/b342982da2d7050f866cb6735288e71c#file-openbsd_get_free_usable_memsize-c-L32=

[fcorbelli]

It is already inside, maybe the problem is in the pagesize / overflow Please report attached file's output 55_1h.zip

#ifdef __OpenBSD__  // OpenBSD is not == FreeBSD
int64_t internal_getramdisksize()
{
    printf("Hello, OpenBSD here\n");

    static int uvmexp_mib[] = {CTL_VM, VM_UVMEXP};
    struct uvmexp uvmexp;
    size_t size;
    size = sizeof(uvmexp);

    printf("Size is %d\n",size);

    if (sysctl(uvmexp_mib, 2, &uvmexp, &size, NULL, 0) == -1) 
        error("36445 sysctl failed");
    else
        printf("OK, sysctl done\n");
    int pagesize = getpagesize();
    printf("The pagesize is %s\n",migliaia(pagesize));
    printf("The free    is %s\n",migliaia(uvmexp.free));

    return uvmexp.free * pagesize;
}
#endif

On a 6.6

root:/tmp/zp# ./zpaqfranz b
zpaqfranz v55.1h-experimental archiver,  compiled Jul  7 2022
Hello, OpenBSD here
Size is 344
OK, sysctl done
The pagesize is 4.096
The free    is 166.240
RAM seems 680.919.040
Benchmarks: XXHASH64 XXH3 SHA-1 SHA-256 BLAKE3 CRC-32 CRC-32C WYHASH WHIRLPOOL MD5 SHA-3 NILSIMSA
Time limit          5 s  (-n X)
Chunks of     390.62 KB  (-minsize Y)

[dertuxmalwieder]

7.1:

$ clang++ -Dunix -O3 -march=native zpaqfranz.cpp -o zpaqfranz -pthread
zpaqfranz.cpp:36381:24: warning: format specifies type 'int' but the argument has type 'size_t' (aka 'unsigned long') [-Wformat]
        printf("Size is %d\n",size);
                        ~~    ^~~~
                        %zu

(You should fix that.)

Results:

$ ./zpaqfranz b
zpaqfranz v55.1h-experimental archiver,  compiled Jul  7 2022
Hello, OpenBSD here
Size is 344
OK, sysctl done
The pagesize is 4.096
The free    is 1.795.240
RAM seems 18.446.744.072.472.920.064
Benchmarks: XXHASH64 XXH3 SHA-1 SHA-256 BLAKE3 CRC-32 CRC-32C WYHASH WHIRLPOOL MD5 SHA-3 NILSIMSA
Time limit          5 s  (-n X)
Chunks of     390.62 KB  (-minsize Y)
...

(edit: fixed output..)

[fcorbelli]

Just a debug pre-release How much free ram do you have (from top)? About 7GB? Seems overflow issue

[dertuxmalwieder]

7016 MB as of now.

[fcorbelli]

Good. Overflow Please try the attached 55_1i.zip

[dertuxmalwieder]

"Good, overflow!" is something I wouldn't want to hear a C/C++ developer say, honestly. 🤣

55.1i:

$ clang++ -Dunix -O3 -march=native zpaqfranz.cpp -o zpaqfranz -pthread
$ ./zpaqfranz b
zpaqfranz v55.1i-experimental archiver,  compiled Jul  7 2022
Hello, OpenBSD here
Size (without warning) is 344
OK, sysctl done
The pagesize is 4.096
The free    is 1.795.451
Big pagesize  4.096
Big free      1.795.451
Big ram       7.354.167.296
RAM seems 7.354.167.296
...

[fcorbelli]

The c/c ++ compilers have now become lasagna code, not even spaghetti code, due to a monstrous complexity and many subtle incompatibilities Can I put you in the acknowledgment?

After "hunting" top source code (for MacOS and Solaris free RAM) I will release the new 55.1 That, of course, won't work at all, like any major release change But it's a feature, afterall :)

[dertuxmalwieder]

The c/c ++ compilers have now become lasagna code, not even spaghetti code, due to a monstrous complexity and many subtle incompatibilities

There are reasons why I stopped any work on my own C++ projects. C is still a good thing. ;-)

Can I put you in the acknowledgment?

Sure! :-)

After "hunting" top source code (for MacOS and Solaris free RAM) I will release the new 55.1

I have an OmniOS (illumos) server under my control (yes, it's mine ;-)), if I can help you there as well...?

[fcorbelli]

I downloaded right now the Solaris 11.4 top command source code, to see how it report free memory

Yes, I typically do this way :)

Work in progress :-)

[fcorbelli]

hahaha getting something on Solaris is VERY hard!

[dertuxmalwieder]

I guess kstat2 could do?

[fcorbelli]

It is rather weird Anyway, if you can, try the attached 55_1j.zip

with

g++ -O3 -march=native -DSOLARIS zpaqfranz.cpp -o zpaqfranz  -pthread -static-libgcc -lkstat

please note the -lkstat

Thanks

#ifdef SOLARIS 
//this is a reworked m_sunos5.c machine-specific top source code

// need a -lkstat to build

#include <kvm.h>
#include <kstat.h>

int64_t internal_getramdisksize()
{
    ///printf("Hello, solaris here\n");
    long freemem;
    static kstat_t      *ks = NULL;
    kvm_t               *kd;
    static kstat_ctl_t  *kc = NULL;
    kstat_named_t       *kn;

    /* total comes from sysconf */
    ///*freemem = sysconf(_SC_PHYS_PAGES);

    if ((kc=kstat_open()) == NULL)
        error("36416: unable to open kstat");

// Solaris compilers does not like some "direct assign", firing warnings
// So this seems ugly, but for a reason
    char    p_unix          [100];
    char    p_system_pages  [100];
    char    p_freemem       [100];
    sprintf(p_unix,"unix");
    sprintf(p_system_pages,"system_pages");
    sprintf(p_freemem,"freemem");

    ks = kstat_lookup(kc, p_unix, 0, p_system_pages);

    if (ks==NULL)
        error("36340: ks is null, quit");

    if (kstat_read(kc,ks,0)!=-1)
    {
        kn=(kstat_named_t*)kstat_data_lookup(ks, p_freemem);
        if (kn!=NULL)
            freemem=(long)(kn->value.ui32);
        else
            error("36440: guru on kstat lookup");
    }
    else
        error("36443: guru on kstat_read");
    int64_t big_ram =freemem;
    return big_ram;
}
#endif

[dertuxmalwieder]

Heads up: kstat is deprecated for kstat2 (but admittedly, illumos does not move fast).

Anyway:

> uname -smor
SunOS 5.11 i86pc illumos

> g++ -O3 -march=native -DSOLARIS zpaqfranz.cpp -o zpaqfranz  -pthread -static-libgcc -lkstat
> ./zpaqfranz b
zpaqfranz v55.1j-experimental archiver,  compiled Jul  7 2022
RAM seems 212.858
Benchmarks: XXHASH64 XXH3 SHA-1 SHA-256 BLAKE3 CRC-32 CRC-32C WYHASH WHIRLPOOL MD5 SHA-3 NILSIMSA
Time limit          5 s  (-n X)
Chunks of     390.62 KB  (-minsize Y)
...

212.858 seems weird: vmstat prints 2861916 under "free".

[fcorbelli]

I am missing something This is a omnios zpaqfranz works fine, BUT those seems to be "pages", not bytes How big is a page?

root@omnios:/zp# kstat -p|grep system_pages:freemem
unix:0:system_pages:freemem     1654599
root@omnios:/zp# ./zpaqfranz b
zpaqfranz v55.1j-experimental archiver,  compiled Jul  8 2022
Hello, solaris here
RAM seems 1.654.379

[dertuxmalwieder]

4096 usually (see the pagesize command). Use getpagesize() to fetch that value.

[fcorbelli]

8K on SPARC, so a getpagesize is really needed :)

[dertuxmalwieder]

Ha! Been a while since I touched a SPARC. (I wish they were affordable for desktops.)

[dertuxmalwieder]

Is there a technical reason why you do this ...

#ifdef SOLARIS
int64_t internal_getramdisksize()
...
#endif

#ifdef __OpenBSD__
int64_t internal_getramdisksize()
...
#endif

#ifdef BSD
#ifndef __OpenBSD__
int64_t internal_getramdisksize()
...
#endif
#endif

... instead of this?

int64_t internal_getramdisksize()
{
#ifdef BSD
# ifdef __OpenBSD__
/* OpenBSD code */
# else
/* Free/Net/DragonFly BSD code */
# endif
#elif defined(SOLARIS)
/* Solaris code */
#endif
}

[fcorbelli]

Aesthetic preference. If the function is not defined, at compile time, an error message appears (with my comment)
If the function is defined in any case it may happen that it does not work as expected

[dertuxmalwieder]

I see. Just wondering as you complained about noodle code above. ;-)

[fcorbelli]

    return internal_getramdisksize(); /// if you get an error HERE substitute with return 0;

will "popup" on compiling

[fcorbelli]

This include MacOS 10+ and - maybe - BSD/Solaris support 55_1k.zip

[dertuxmalwieder]

I happen to own a MacBook as well (of course), but I won't be near it until tomorrow, so I'll trust you with macOS support for now. ;-)

OmniOS:

> ./zpaqfranz b
zpaqfranz v55.1k-experimental archiver,  compiled Jul  8 2022
RAM seems 820.944.896

Seems wrong:

> vmstat
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr bk ro s0 --   in   sy   cs us sy id
 0 0 0 6562776 2803220 73 245 0 0  0  0  8 12 10 -0  0 2452 1994  497  2  1 97

Or am I misinterpreting the output? because it may well be that there's only 820M of RAM free on that server... ;-)

[fcorbelli]

Try top I take the code from the machine-specific source of top top.zip

On Mac: I am not sure for older OS, I rarely use MacOS

On the other hand, the Apple APIs change at a frightening speed, and at the moment I have 1 (one) user of zpaqfranz on Mac :)

[dertuxmalwieder]

illumos doesn’t have top. I usually stick with the base tools unless they don’t work well enough. I’m out of town until tomorrow, but if top works for you, I believe you. ;-)

I use macOS 12 or whatever the newest is. Once my OpenBSD port (waiting for 55.1 before I put it on their mailing list) is released, I’ll see what is needed to make a pkgsrc port as well. NetBSD’s pkgsrc is fairly widely used on macOS, so I’ll only have to maintain two packages.

[fcorbelli]

I am writing the changelog, before making a new release https://github.com/fcorbelli/zpaqfranz/blob/main/CHANGELOG.md

Using platform specific functions I expect the -ramdisk function ... not to work at all, except on Windows Its purpose, together with -longpath, is to perform restorability tests from different systems (Windows) of archives made on FreeBSD machines In this way you avoid the risk that some kind of file is not recoverable due to filesystem problems, or compiler errors (I happened, I also reported them to GNU) If you are "curious" I could explain why it is not easy to test the zpaq archives, or you can read it directly on the reference forum Given a file within an archive there is no effective way to calculate its overall hash. zpaq, in fact, doesn't quite do this. But I'm pretty paranoid, and so I actually want to make sure that the files are not only uncorrupted, without undetected SHA-1 collision (currently only two files are known, but hey...), but actually extractable, that is, writable and readable back. This is much, much, much less simple than it might seem. Just imagine that you have two different files pippo.txt and PIPPO.txt and a lpt1.txt on a Linux machine. How do you extract them on Windows?

[dertuxmalwieder]

LPT1.txt could work (I’d need to test that), case-sensitivity is (in theory) supported by NTFS, but most likely disabled.

[fcorbelli]

Those are Windows' limitations, not NTFS 13.zip Try to extract this "thing" on Windows :)

Of course zpaqfranz 55... can do :)))

[dertuxmalwieder]

So, the README says

New Release: 55.1 (July 2022)

but the release is not there yet. Confusing! 😂

[dertuxmalwieder]

I posted the OpenBSD port to the ports mailing list. I hope they'll accept it.

Closing this ticket for now as it's "resolved". :-)

[dertuxmalwieder]

FYI, the port was imported to OpenBSD. Assuming the user runs the latest version (or has the ports infrastructure installed), zpaqfranz can either be built locally...

cd /usr/ports/archivers/zpaqfranz
make install

... or installed as a package once the servers have it:

pkg_add zpaqfranz

I'll try to keep the port maintained for the foreseeable future.

[fcorbelli]

Thank you very much!