Refactor and simplify repositories, build systems, and packages.

gogins commented 3 years ago

Reinstalling Ubuntu on my laptop and then preparing a new build system for this repository has demonstrated once again that things are too complex for casual developers.

This will be an ongoing issue until it is reasonable to suppose that a casual developer can clone this repository and build all its targets using only system packages. Or as close to that as I can get without wasting time.

The same should be true of new repositories that contain projects moved out of csound-extended.

Possibly, implement continuous integration that builds for Ubuntu, Windows, and Mac OS.

First refactor the repositories:

[x] csound-android, leave as is.
[x] csound-examples.
[x] csound-extended, includes CsoundAC and the Python interface, remove nudruz and OM2Csound, remove Haskell stuff, remove csound.node, remove WebAssembly stuff.
[x] csound-extended-haskell, includes all Haskell stuff. Move this out, that's all.
[x] csound-extended-manual.
[x] csound-extended-node.
[x] csound-extended-nudruz.
[x] csound-extended-OM2Csound, just move this out.
[x] csound-extended-wasm, includes WebAssembly modules for both Csound and CsoundAC, has both Csound and CsoundAC as Git submodules.
[x] csound-vst3-opcodes, basically leave as is, just need to debug on Windows.

Don't use filter-branch! Use filter-repo instead, e.g.

git filter-repo --subdirectory-filter haskell
# Check that we are indeed no longer connected to the main csound-extended repo...
git remote -v
# Create an empty csound-extended-haskell repo in GitHub...
git remote add origin https://github.com/gogins/csound-extended-haskell.git
# Check that we are indeed now connected to the new csound-extended-haskell repo...
git remote -v
git branch -M main
git push -u origin main

Then ensure that the following repositories build and that their examples run, in the order given:

[x] csound-extended, includes CsoundAC and the Python interface. Keeping the Debian package, cleaning up Lintian errors.
[x] csound-extended-node, do over with npm for both nw.js and node.js (not doing that, only one .so needs to be downloaded and installed), depends on Csound.
[x] csound-extended-wasm, includes WebAssembly modules for both Csound and CsoundAC, has both Csound and CsoundAC as Git submodules, ensure that it builds and that the examples run from GitHub pages.
[x] csound-examples, download csound-extended-wasm for this.
[x] csound-extended-manual, download csound-extended-wasm for this.
[x] csound-extended-nudruz, ensure examples run.

gogins commented 3 years ago

Some hints: https://gist.github.com/youhide/121750fc4878801ea8e908080b535beb

gogins commented 3 years ago

Yikes:

mkg@xenakis:~/csound-extended/build-linux$ ldd _CsoundAC.so 
    linux-vdso.so.1 (0x00007ffd84be3000)
    libCsoundAC.so.6.0 => /home/mkg/csound-extended/build-linux/libCsoundAC.so.6.0 (0x00007f245031c000)
    libcsnd6.so.6.0 => /usr/local/lib/libcsnd6.so.6.0 (0x00007f24502c8000)
    libcsound64.so.6.0 => /usr/local/lib/libcsound64.so.6.0 (0x00007f244fef7000)
    libpython3.9.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0 (0x00007f244f945000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f244f764000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f244f615000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f244f5f8000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f244f406000)
    libfltk_images.so.1.3 => /usr/lib/x86_64-linux-gnu/libfltk_images.so.1.3 (0x00007f244f3f5000)
    libopencv_imgcodecs.so.3.4 => /usr/local/lib/libopencv_imgcodecs.so.3.4 (0x00007f244ea74000)
    libopencv_imgproc.so.3.4 => /usr/local/lib/libopencv_imgproc.so.3.4 (0x00007f244d121000)
    libopencv_core.so.3.4 => /usr/local/lib/libopencv_core.so.3.4 (0x00007f244bddd000)
    libsndfile.so.1 => /usr/lib/x86_64-linux-gnu/libsndfile.so.1 (0x00007f244bd5e000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f244bd3b000)
    libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f244bcac000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f244bca6000)
    libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f244bc78000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f244bc5c000)
    libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f244bc55000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f24507d6000)
    libpng16.so.16 => /usr/lib/x86_64-linux-gnu/libpng16.so.16 (0x00007f244bc1d000)
    libjpeg.so.8 => /usr/lib/x86_64-linux-gnu/libjpeg.so.8 (0x00007f244bb98000)
    libfltk.so.1.3 => /usr/lib/x86_64-linux-gnu/libfltk.so.1.3 (0x00007f244ba61000)
    libpng12.so.0 => /lib/x86_64-linux-gnu/libpng12.so.0 (0x00007f244b83c000)
    libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f244b7b9000)
    libjasper.so.1 => /usr/lib/x86_64-linux-gnu/libjasper.so.1 (0x00007f244b564000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f244b559000)
    libFLAC.so.8 => /usr/lib/x86_64-linux-gnu/libFLAC.so.8 (0x00007f244b51b000)
    libogg.so.0 => /usr/lib/x86_64-linux-gnu/libogg.so.0 (0x00007f244b50e000)
    libvorbis.so.0 => /usr/lib/x86_64-linux-gnu/libvorbis.so.0 (0x00007f244b4de000)
    libvorbisenc.so.2 => /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2 (0x00007f244b433000)
    libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f244b40a000)
    libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f244b3e9000)
    librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f244b3c9000)
    libssh.so.4 => /usr/lib/x86_64-linux-gnu/libssh.so.4 (0x00007f244b35b000)
    libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f244b346000)
    libnettle.so.7 => /usr/lib/x86_64-linux-gnu/libnettle.so.7 (0x00007f244b30c000)
    libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f244b136000)
    libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f244b0e9000)
    libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f244b093000)
    liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f244b082000)
    libbrotlidec.so.1 => /usr/lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007f244b072000)
    libXrender.so.1 => /usr/lib/x86_64-linux-gnu/libXrender.so.1 (0x00007f244ae68000)
    libXcursor.so.1 => /usr/lib/x86_64-linux-gnu/libXcursor.so.1 (0x00007f244ae5b000)
    libXfixes.so.3 => /usr/lib/x86_64-linux-gnu/libXfixes.so.3 (0x00007f244ae53000)
    libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f244ae3e000)
    libXft.so.2 => /usr/lib/x86_64-linux-gnu/libXft.so.2 (0x00007f244ae23000)
    libfontconfig.so.1 => /usr/lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007f244addc000)
    libXinerama.so.1 => /usr/lib/x86_64-linux-gnu/libXinerama.so.1 (0x00007f244add7000)
    libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f244ac9a000)
    libwebp.so.6 => /usr/lib/x86_64-linux-gnu/libwebp.so.6 (0x00007f244aa31000)
    libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f244a988000)
    liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f244a95d000)
    libjbig.so.0 => /usr/lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f244a74f000)
    libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f244a5cd000)
    libhogweed.so.5 => /usr/lib/x86_64-linux-gnu/libhogweed.so.5 (0x00007f244a595000)
    libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f244a511000)
    libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f244a239000)
    libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f244a103000)
    libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f244a0ed000)
    libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f244a010000)
    libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f2449fdf000)
    libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f2449fd8000)
    libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f2449fc7000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f2449fab000)
    libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f2449f8e000)
    libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f2449f49000)
    libbrotlicommon.so.1 => /usr/lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007f2449f26000)
    libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007f2449e65000)
    libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f2449e5c000)
    libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f2449e32000)
    libffi.so.7 => /usr/lib/x86_64-linux-gnu/libffi.so.7 (0x00007f2449e26000)
    libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f2449e1f000)
    libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f2449e11000)
    libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f2449d7e000)
    libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f2449cd7000)
    libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f2449c9f000)
    libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007f2449c86000)
    libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f2449c80000)
    libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f2449c76000)
    libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007f2449c4c000)
    libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f2449c3a000)
    libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f2449bec000)
    libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f2449ac3000)
    libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f2449a88000)
    libbsd.so.0 => /usr/lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f2449a6c000)
mkg@xenakis:~/csound-extended/build-linux$

gogins commented 3 years ago

Many of these are secondary and tertiary dependencies.

gogins commented 3 years ago

And for my Linux build of Csound with Python support I have the following. But current Csound omits the Python opcodes and should probably omit the csnd6 module.

mkg@xenakis:~/csound/build$ ldd _csnd6.so 
    linux-vdso.so.1 (0x00007ffe1ddf8000)
    libcsnd6.so.6.0 => /home/mkg/csound/build/libcsnd6.so.6.0 (0x00007efd58b0f000)
    libcsound64.so.6.0 => /home/mkg/csound/build/libcsound64.so.6.0 (0x00007efd5873e000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007efd58530000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007efd58515000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efd58323000)
    libsndfile.so.1 => /usr/lib/x86_64-linux-gnu/libsndfile.so.1 (0x00007efd582a6000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007efd58281000)
    libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007efd581f2000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007efd580a3000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007efd5809d000)
    /lib64/ld-linux-x86-64.so.2 (0x00007efd58cc7000)
    libFLAC.so.8 => /usr/lib/x86_64-linux-gnu/libFLAC.so.8 (0x00007efd5805f000)
    libogg.so.0 => /usr/lib/x86_64-linux-gnu/libogg.so.0 (0x00007efd58052000)
    libvorbis.so.0 => /usr/lib/x86_64-linux-gnu/libvorbis.so.0 (0x00007efd58022000)
    libvorbisenc.so.2 => /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2 (0x00007efd57f77000)
    libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007efd57f4e000)
    libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007efd57f2d000)
    librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007efd57f0d000)
    libssh.so.4 => /usr/lib/x86_64-linux-gnu/libssh.so.4 (0x00007efd57e9f000)
    libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007efd57e8a000)
    libnettle.so.7 => /usr/lib/x86_64-linux-gnu/libnettle.so.7 (0x00007efd57e50000)
    libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007efd57c7a000)
    libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007efd57c2d000)
    libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007efd57bd7000)
    liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007efd57bc6000)
    libbrotlidec.so.1 => /usr/lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007efd57bb6000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007efd57b9a000)
    libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007efd57a18000)
    libhogweed.so.5 => /usr/lib/x86_64-linux-gnu/libhogweed.so.5 (0x00007efd579e0000)
    libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007efd5795c000)
    libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007efd57684000)
    libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007efd5754e000)
    libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007efd57538000)
    libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007efd5745b000)
    libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007efd5742a000)
    libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007efd57423000)
    libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007efd57412000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007efd573f6000)
    libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007efd573d9000)
    libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007efd57394000)
    libbrotlicommon.so.1 => /usr/lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007efd57371000)
    libffi.so.7 => /usr/lib/x86_64-linux-gnu/libffi.so.7 (0x00007efd57363000)
    libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007efd5735c000)
    libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007efd57350000)
    libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007efd572bd000)
    libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007efd57216000)
    libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007efd571dc000)
    libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007efd571c3000)
    libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007efd57199000)
    libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007efd57187000)
    libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007efd57139000)
    libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007efd57010000)
    libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007efd56fd3000)
mkg@xenakis:~/csound/build$

gogins commented 3 years ago

Much of the difference obviously is due to my use of the OpenCV library which itself has many dependencies. I will see if I can back that out.

gogins commented 3 years ago

Does OpenCV install from sources or as a system package? Either way. Therefore, I will try to capture all the dependencies in some sort of script.

gogins commented 3 years ago

Note: Runtime dependencies do not include build time dependencies that are static libraries or tools.

gogins commented 3 years ago

Probably the best way to do this is to implement a Github Action build of the csound-extended package. That should identify all the build dependencies. But this would probably be pretty time-consuming.

I could also create Debian packages and run test builds in an isolated environment to identify build dependencies.

Just to be clear, the goal here is not provide finished packages a newbie can simply install from the system, but something the typical computer musician could download and install, or even build and install. The smaller granularity is to enable people to pick and choose what they really want or need without getting confused or bothered by unneeded dependencies.

tjingboem commented 3 years ago

would it be possible to be able to choose what the ueser wants to install or not, and then check for the dependencies only for the one he/she wants to install? Just asking, not demanding :-)

gogins commented 3 years ago

I haven't decided what to do about this mess, but I am thinking about it.

My main problem is that I am trying to be primarily a composer and only secondarily a developer. So the software is really done when it builds for me, and I haven't thought so much about making it buildable by other people. But I am thinking about it.

Michael Gogins Irreducible Productions http://michaelgogins.tumblr.com Michael dot Gogins at gmail dot com

On Mon, May 17, 2021 at 8:55 AM Menno Knevel @.***> wrote:

would it be possible to be able to choose what the ueser wants to install or not, and then check for the dependencies only for the one he/she wants to install? Just asking, not demanding :-)

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/gogins/csound-extended/issues/158#issuecomment-842300826, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQIGJNHHNRDOXUVSOWCJD3TOEG23ANCNFSM4YPD65EQ .

tjingboem commented 3 years ago

please stay a composer primarily! If you'd know how to do this, that would be great, but if it is a burden, please don't waste your time on this ;-)

gogins commented 3 years ago

Git submodules to remove from csound-extended:

[x] dependencies/cm-fomus.
[x] dependencies/cm2.
[x] dependencies/cope/david-cope-cmmc (first copy cope to csound-extended-nudruz).
[x] dependencies/csound.
[x] dependencies/fomus.
[x] dependencies/liblo.
[x] dependencies/libsndfile.
[x] dependencies/oboe.
[x] dependencies/portaudio.
[x] dependencies/weyl.

Other things to remove:

[x] examples_and_tests/aeolus.
[x] examples_and_tests/csound_html5.
[x] examples_and_tests/nudruz.

To remove a submodule:

# No trailing slash!
git submodule deinit -f -- a/submodule    
rm -rf .git/modules/a/submodule
git rm -f a/submodule

gogins commented 3 years ago

See https://gitlab.kitware.com/cmake/cmake/-/issues/21834 for Lintian errors and warnings.

gogins commented 3 years ago

Creating continuous integration builds, should I decide to do that, will be a new issue.

gogins / csound-extended

Refactor and simplify repositories, build systems, and packages. #158