feat: add musl prebuild

SilentBot1 commented 3 months ago

In this PR, we add musl prebuilds of libdatachannel to node-datachannel to allow users on musl-based systems (Alpine, Kiss Linux) to use node-datachannel without needing to build libdatachannel locally.

This is done by generating sysroots locally for glibc (ubuntu) and musl (alpine) and using these to prebuild libdatachannel.

An example of these builds can be seen here: https://github.com/SilentBot1/node-datachannel/releases/tag/v0.9.1 And this can be tested locally by doing npm i https://github.com/SilentBot1/node-datachannel.git

SilentBot1 commented 3 months ago

There appears to be an issue with glibc builds still which we're working through, I will mark this ready for review once we've got these changes ready.

SilentBot1 commented 3 months ago

This should now be fixed and ready for review!

murat-dogan commented 3 months ago

Thank you very much for the PR. This is a big change :)

Could you please explain me archid? There is 4.

murat-dogan commented 3 months ago

And if I compare the binary sizes they are different. Do you have an idea about that?

https://github.com/SilentBot1/node-datachannel/releases/tag/v0.9.1 https://github.com/murat-dogan/node-datachannel/releases/tag/v0.9.1

funniray commented 3 months ago

Could you please explain me archid? There is 4.

Different programs can't agree on what to call each architecture.

archid -> name of architecture for the target triple
arch -> name of architecture according to docker/podman
nodearch -> name of architecture according to nodejs

Alpine and Debian disagree on the architecture for the triple for armv7. Debian calls it arm, while Alpine calls it armv7. This is why there's an override on debian arm[v7] builds to set the triple's target to just arm instead of armv7. I can rename archid to something like triple_arch if that's clearer.

And if I compare the binary sizes they are different. Do you have an idea about that?

libssl.a on alpine is 6.3MB libssl.a on debian is 1.2MB

as libssl is linked statically, I'd assume that's why the filesize is larger on musl vs debian

murat-dogan commented 3 months ago

Thank you for the explanation.

I think it will be good first to merge with another branch. So we can test it better, and then merge it with the master. What do you think?

I created a new branch here https://github.com/murat-dogan/node-datachannel/tree/new-github-actions

SilentBot1 commented 3 months ago

Thank you for the explanation.

I think it will be good first to merge with another branch. So we can test it better, and then merge it with the master. What do you think?

I created a new branch here https://github.com/murat-dogan/node-datachannel/tree/new-github-actions

Sure, I've repointed this PR to the new branch you've mentioned. Please note, that we probably will want to bump the version for this branch once this is merged (or give it a postfix like -dev / -testing) so it doesn't interfere with the currently published release and its prebuilds.

Feel free to merge whenever you are ready.

murat-dogan commented 3 months ago

I have created a release here. Could you please also check?

https://github.com/murat-dogan/node-datachannel/releases/tag/v0.10.0-dev

murat-dogan commented 3 months ago

For Linux x64 ; v0.9.2 binary size is 10.2 MB v0.10.0-dev binary size is 8.4 MB

Less is better of course, but I wonder why. GCC versions are same and OpenSSL versions are 1.1.1.f and 1.1.1w This should not make much difference. Any idea?

funniray commented 3 months ago

I checked and found out

openssl 1.1.1w on debian bullseye is only like 4kb smaller than openssl 1.1.1f on ubuntu 20.04. (so shouldn't cause a difference)
they both link to the same exact dynamic libraries
gcc as it's currently configured doesn't even use the sysroot's glibc, so it should be linking against the exact same build of glibc as the previous builds

I'm honestly unsure what could be causing the smaller builds. I don't have enough experience with C/C++ to know how to debug why the files are larger any further. As far as I'm aware however, the only difference the compilers would see is the openssl version

murat-dogan commented 3 months ago

The difference is pthread linking I am checking it

Current

murat@murat-ThinkBook:~/js/node-datachannel/build/Release$ ldd node_datachannel.node_org 
    linux-vdso.so.1 (0x00007ffef929a000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ca84bc00000)
    /lib64/ld-linux-x86-64.so.2 (0x00007ca84cb4f000)

New

murat@murat-ThinkBook:~/js/node-datachannel/build/Release$ ldd node_datachannel.node
    linux-vdso.so.1 (0x00007ffe481af000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f6a60e04000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6a60dff000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6a60200000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f6a60e24000)

funniray commented 3 months ago

Hey there, I don't seem to see the same results as you. Are you sure you're checking the right file?

funniray@DESKTOP:~/test-test$ ldd new-linux.node
        linux-vdso.so.1 (0x00007ffe3f3bd000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa93e873000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa93e86e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa93e65c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa93ef54000)
funniray@DESKTOP:~/test-test$ ldd old-linux.node
        linux-vdso.so.1 (0x00007fff3795b000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f9701272000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f970126d000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f970105b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f9701abf000)

old-linux.node was from 0.9.2-x64 new-linux.node was from our fork's build as it's currently not present here

murat-dogan commented 3 months ago

Yep, you are right. I was comparing with local built version.

Then, I am again at the beginning :)

murat-dogan commented 3 months ago

Found a diff by running command readelf -s

Current

ELF Header:
  Magic:   7f 45 4c 46 02 01 01 03 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - GNU
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x1ec000
  Start of program headers:          64 (bytes into file)
  Start of section headers:          10247840 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         10
  Size of section headers:           64 (bytes)
  Number of section headers:         33
  Section header string table index: 32

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .note.gnu.bu[...] NOTE             0000000000000270  00000270
       0000000000000024  0000000000000000   A       0     0     4
  [ 2] .gnu.hash         GNU_HASH         0000000000000298  00000298
       000000000001a878  0000000000000000   A       3     0     8
  [ 3] .dynsym           DYNSYM           000000000001ab10  0001ab10
       0000000000058ea8  0000000000000018   A       4     1     8
  [ 4] .dynstr           STRTAB           00000000000739b8  000739b8
       00000000000c5cac  0000000000000000   A       0     0     1
  [ 5] .gnu.version      VERSYM           0000000000139664  00139664
       000000000000768e  0000000000000002   A       3     0     2
  [ 6] .gnu.version_r    VERNEED          0000000000140cf8  00140cf8
       0000000000000130  0000000000000000   A       4     4     8
  [ 7] .rela.dyn         RELA             0000000000140e28  00140e28
       0000000000070968  0000000000000018   A       3     0     8
  [ 8] .rela.plt         RELA             00000000001b1790  001b1790
       0000000000022548  0000000000000018  AI       3    25     8
  [ 9] .init             PROGBITS         00000000001d4000  001d4000
       0000000000000020  0000000000000000  AX       0     0     4
  [10] .plt              PROGBITS         00000000001d4020  001d4020
       0000000000016e40  0000000000000010  AX       0     0     16
  [11] .plt.got          PROGBITS         00000000001eae60  001eae60
       00000000000003f0  0000000000000008  AX       0     0     8
  [12] .text             PROGBITS         00000000001ec000  001ec000
       0000000000495f00  0000000000000000  AX       0     0     4096
  [13] .fini             PROGBITS         0000000000681f00  00681f00
       000000000000000d  0000000000000000  AX       0     0     4
  [14] .rodata           PROGBITS         0000000000682000  00682000
       0000000000084000  0000000000000000   A       0     0     64
  [15] .stapsdt.base     PROGBITS         0000000000706000  00706000
       0000000000000001  0000000000000000   A       0     0     1
  [16] .eh_frame_hdr     PROGBITS         0000000000706004  00706004
       000000000001e194  0000000000000000   A       0     0     4
  [17] .eh_frame         PROGBITS         0000000000724198  00724198
       000000000009fdb8  0000000000000000   A       0     0     8
  [18] .gcc_except_table PROGBITS         00000000007c3f50  007c3f50
       000000000001b44d  0000000000000000   A       0     0     4
  [19] .tbss             NOBITS           00000000007e0568  007df568
       0000000000002780  0000000000000000 WAT       0     0     8
  [20] .init_array       INIT_ARRAY       00000000007e0568  007df568
       0000000000000190  0000000000000008  WA       0     0     8
  [21] .fini_array       FINI_ARRAY       00000000007e06f8  007df6f8
       0000000000000008  0000000000000008  WA       0     0     8
  [22] .data.rel.ro      PROGBITS         00000000007e0700  007df700
       000000000003dc00  0000000000000000  WA       0     0     32
  [23] .dynamic          DYNAMIC          000000000081e300  0081d300
       0000000000000200  0000000000000010  WA       4     0     8
  [24] .got              PROGBITS         000000000081e500  0081d500
       0000000000003b00  0000000000000008  WA       0     0     8
  [25] .got.plt          PROGBITS         0000000000822000  00821000
       000000000000b730  0000000000000008  WA       0     0     8
  [26] .data             PROGBITS         000000000082d740  0082c740
       0000000000006780  0000000000000000  WA       0     0     32
  [27] .bss              NOBITS           0000000000833ec0  00832ec0
       0000000000009f60  0000000000000000  WA       0     0     32
  [28] .comment          PROGBITS         0000000000000000  00832ec0
       000000000000002b  0000000000000001  MS       0     0     1
  [29] .note.stapsdt     NOTE             0000000000000000  00832eec
       00000000000000e8  0000000000000000           0     0     4
  [30] .symtab           SYMTAB           0000000000000000  00832fd8
       000000000007d160  0000000000000018          31   6174     8
  [31] .strtab           STRTAB           0000000000000000  008b0138
       0000000000115c34  0000000000000000           0     0     1
  [32] .shstrtab         STRTAB           0000000000000000  009c5d6c
       0000000000000132  0000000000000000           0     0     1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  R (retain), D (mbind), l (large), p (processor specific)

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x00000000001d3cd8 0x00000000001d3cd8  R      0x1000
  LOAD           0x00000000001d4000 0x00000000001d4000 0x00000000001d4000
                 0x00000000004adf0d 0x00000000004adf0d  R E    0x1000
  LOAD           0x0000000000682000 0x0000000000682000 0x0000000000682000
                 0x000000000015d39d 0x000000000015d39d  R      0x1000
  LOAD           0x00000000007df568 0x00000000007e0568 0x00000000007e0568
                 0x0000000000053958 0x000000000005d8b8  RW     0x1000
  DYNAMIC        0x000000000081d300 0x000000000081e300 0x000000000081e300
                 0x0000000000000200 0x0000000000000200  RW     0x8
  NOTE           0x0000000000000270 0x0000000000000270 0x0000000000000270
                 0x0000000000000024 0x0000000000000024  R      0x4
  TLS            0x00000000007df568 0x00000000007e0568 0x00000000007e0568
                 0x0000000000000000 0x0000000000002780  R      0x8
  GNU_EH_FRAME   0x0000000000706004 0x0000000000706004 0x0000000000706004
                 0x000000000001e194 0x000000000001e194  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  GNU_RELRO      0x00000000007df568 0x00000000007e0568 0x00000000007e0568
                 0x0000000000041a98 0x0000000000041a98  R      0x1

 Section to Segment mapping:
  Segment Sections...
   00     .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt 
   01     .init .plt .plt.got .text .fini 
   02     .rodata .stapsdt.base .eh_frame_hdr .eh_frame .gcc_except_table 
   03     .init_array .fini_array .data.rel.ro .dynamic .got .got.plt .data .bss 
   04     .dynamic 
   05     .note.gnu.build-id 
   06     .tbss 
   07     .eh_frame_hdr 
   08     
   09     .init_array .fini_array .data.rel.ro .dynamic .got

new

ELF Header:
  Magic:   7f 45 4c 46 02 01 01 03 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - GNU
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x172000
  Start of program headers:          64 (bytes into file)
  Start of section headers:          8362336 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         10
  Size of section headers:           64 (bytes)
  Number of section headers:         31
  Section header string table index: 30

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .note.gnu.bu[...] NOTE             0000000000000270  00000270
       0000000000000024  0000000000000000   A       0     0     4
  [ 2] .gnu.hash         GNU_HASH         0000000000000298  00000298
       0000000000013e7c  0000000000000000   A       3     0     8
  [ 3] .dynsym           DYNSYM           0000000000014118  00014118
       000000000003de90  0000000000000018   A       4     1     8
  [ 4] .dynstr           STRTAB           0000000000051fa8  00051fa8
       000000000008b52b  0000000000000000   A       0     0     1
  [ 5] .gnu.version      VERSYM           00000000000dd4d4  000dd4d4
       000000000000528c  0000000000000002   A       3     0     2
  [ 6] .gnu.version_r    VERNEED          00000000000e2760  000e2760
       0000000000000120  0000000000000000   A       4     4     8
  [ 7] .rela.dyn         RELA             00000000000e2880  000e2880
       000000000005ba88  0000000000000018   A       3     0     8
  [ 8] .rela.plt         RELA             000000000013e308  0013e308
       000000000001e318  0000000000000018  AI       3    24     8
  [ 9] .init             PROGBITS         000000000015d000  0015d000
       0000000000000020  0000000000000000  AX       0     0     4
  [10] .plt              PROGBITS         000000000015d020  0015d020
       0000000000014220  0000000000000010  AX       0     0     16
  [11] .plt.got          PROGBITS         0000000000171240  00171240
       0000000000000328  0000000000000008  AX       0     0     8
  [12] .text             PROGBITS         0000000000172000  00172000
       00000000003e4fd0  0000000000000000  AX       0     0     4096
  [13] .fini             PROGBITS         0000000000556fd0  00556fd0
       000000000000000d  0000000000000000  AX       0     0     4
  [14] .rodata           PROGBITS         0000000000557000  00557000
       000000000007e438  0000000000000000   A       0     0     64
  [15] .eh_frame_hdr     PROGBITS         00000000005d5438  005d5438
       0000000000016df4  0000000000000000   A       0     0     4
  [16] .eh_frame         PROGBITS         00000000005ec230  005ec230
       000000000007e590  0000000000000000   A       0     0     8
  [17] .gcc_except_table PROGBITS         000000000066a7c0  0066a7c0
       000000000001531f  0000000000000000   A       0     0     4
  [18] .tbss             NOBITS           00000000006811b8  006801b8
       0000000000002760  0000000000000000 WAT       0     0     8
  [19] .init_array       INIT_ARRAY       00000000006811b8  006801b8
       0000000000000160  0000000000000008  WA       0     0     8
  [20] .fini_array       FINI_ARRAY       0000000000681318  00680318
       0000000000000008  0000000000000008  WA       0     0     8
  [21] .data.rel.ro      PROGBITS         0000000000681320  00680320
       0000000000036040  0000000000000000  WA       0     0     32
  [22] .dynamic          DYNAMIC          00000000006b7360  006b6360
       0000000000000200  0000000000000010  WA       4     0     8
  [23] .got              PROGBITS         00000000006b7560  006b6560
       0000000000002a98  0000000000000008  WA       0     0     8
  [24] .got.plt          PROGBITS         00000000006ba000  006b9000
       000000000000a120  0000000000000008  WA       0     0     8
  [25] .data             PROGBITS         00000000006c4120  006c3120
       0000000000006650  0000000000000000  WA       0     0     32
  [26] .bss              NOBITS           00000000006ca780  006c9770
       0000000000006ef0  0000000000000000  WA       0     0     32
  [27] .comment          PROGBITS         0000000000000000  006c9770
       000000000000002b  0000000000000001  MS       0     0     1
  [28] .symtab           SYMTAB           0000000000000000  006c97a0
       000000000005d330  0000000000000018          29   5341     8
  [29] .strtab           STRTAB           0000000000000000  00726ad0
       00000000000d2d7a  0000000000000000           0     0     1
  [30] .shstrtab         STRTAB           0000000000000000  007f984a
       0000000000000116  0000000000000000           0     0     1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  R (retain), D (mbind), l (large), p (processor specific)

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x000000000015c620 0x000000000015c620  R      0x1000
  LOAD           0x000000000015d000 0x000000000015d000 0x000000000015d000
                 0x00000000003f9fdd 0x00000000003f9fdd  R E    0x1000
  LOAD           0x0000000000557000 0x0000000000557000 0x0000000000557000
                 0x0000000000128adf 0x0000000000128adf  R      0x1000
  LOAD           0x00000000006801b8 0x00000000006811b8 0x00000000006811b8
                 0x00000000000495b8 0x00000000000504b8  RW     0x1000
  DYNAMIC        0x00000000006b6360 0x00000000006b7360 0x00000000006b7360
                 0x0000000000000200 0x0000000000000200  RW     0x8
  NOTE           0x0000000000000270 0x0000000000000270 0x0000000000000270
                 0x0000000000000024 0x0000000000000024  R      0x4
  TLS            0x00000000006801b8 0x00000000006811b8 0x00000000006811b8
                 0x0000000000000000 0x0000000000002760  R      0x8
  GNU_EH_FRAME   0x00000000005d5438 0x00000000005d5438 0x00000000005d5438
                 0x0000000000016df4 0x0000000000016df4  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  GNU_RELRO      0x00000000006801b8 0x00000000006811b8 0x00000000006811b8
                 0x0000000000038e48 0x0000000000038e48  R      0x1

 Section to Segment mapping:
  Segment Sections...
   00     .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt 
   01     .init .plt .plt.got .text .fini 
   02     .rodata .eh_frame_hdr .eh_frame .gcc_except_table 
   03     .init_array .fini_array .data.rel.ro .dynamic .got .got.plt .data .bss 
   04     .dynamic 
   05     .note.gnu.build-id 
   06     .tbss 
   07     .eh_frame_hdr 
   08     
   09     .init_array .fini_array .data.rel.ro .dynamic .got

Difference is on Section Headers: 15,29

I have never used this lib https://wiki.ubuntu.com/Kernel/Systemtap

funniray commented 3 months ago

Found a diff by running command readelf -s

Difference is on Section Headers: 15,29

I have never used this lib https://wiki.ubuntu.com/Kernel/Systemtap

Definitely a difference, but I'm unsure if it's the real difference that's causing the filesize differences. After extracting, the new one is almost 2MB smaller. The sections you mentioned only make up 233 bytes in total. It's possible that the tool is storing more data elsewhere in the file, but I'm unsure if it's the actual culprit.

murat-dogan commented 3 months ago

From readelf command I mostly see SSL Lib differences besides SystemTap

If all tests are passed we can merge I think. What do you think?

murat-dogan commented 3 months ago

I also compared the performance, no difference

funniray commented 3 months ago

From readelf command I mostly see SSL Lib differences besides SystemTap

If all tests are passed we can merge I think. What do you think?

Since the tests pass and the openssl 1.1 version is newer than the one in prior builds, I see no reason to not go through with merging it

SilentBot1 commented 3 months ago

Likewise, everything has worked as expected in all my testing, so this looks good to me. :shipit:

murat-dogan commented 3 months ago

I think also same. Thanks!

murat-dogan commented 3 months ago

I have just released v0.10.0 https://github.com/murat-dogan/node-datachannel/releases/tag/v0.10.0

SilentBot1 commented 3 months ago

Awesome, thanks :)

As a quick note @murat-dogan, it appears the macos-11 runner has possibly been deprecated, so the OSX pre-builds aren't running. The image may need to be bumped to macos-12 (and tested) to ensure OSX prebuilds remain.

murat-dogan commented 3 months ago

https://github.com/actions/runner-images/issues/9255

It seems, also in 2-3 months macos-12 will be removed

murat-dogan commented 3 months ago

I updated Mac-x64 CI file also to OpenSSL 1.1.1w and binary file sizes ~2MB decreased.

murat-dogan commented 3 months ago

And unfortunately, I don't have a Mac. I can not test it, but it should be OK

SilentBot1 commented 3 months ago

And unfortunately, I don't have a Mac. I can not test it, but it should be OK

I'll test this now and let you know, I have both an M2 and arm64 mac available to test on.

murat-dogan commented 3 months ago

For arm64 I am still checking

murat-dogan commented 3 months ago

Tried to update Mac M1 arm64 to also 1.1.1w but since there is no Mac M1 native machine on github actions, now I could not.

Anyway with 1.1.1t now ready to test.

SilentBot1 commented 3 months ago

The pre-builds worked on both devices (arm64, amd64), so they look good to me :)

murat-dogan / node-datachannel

feat: add musl prebuild #264