More standardized, internetless build and better build instructions

[fpesari]

Hello,

I am a packager on openSUSE and wanted to make a package for stargate. Builds on OBS (Open Build Service, the build system used by openSUSE) happen in a virtual machine unconnected to the Internet so the fact that make tries to download some packages from PyPi is problematic.

I tried building stargate via the make command but it tried to run pip and install some packages from the Internet, which as I said is not allowed on OBS.

Nevertheless, I proceeded with the build on my local system. It compiled correctly and make install worked but the stargate executable failed twice - first when mido was missing, second (after having installed mido) when sglib was missing. I had manually run python3 setup.py install --user to install the Python package in the src directory but that apparently did not provide sglib.

My suggestions:

• Indicate all dependencies at both build-time and runtime (like portaudio, portmidi, pymarshal, wavefile, mido) in the build instructions
• Make sure that make does not connect to the Internet, especially if Python dependencies are found on the system
• Make sure that make install installs everything needed to run stargate, including the Python package containing sglib which should respect the distro's hierarchy (openSUSE also uses lib64, see #14). Also, make install tried to execute the (non-existing on my system) /mingw64/bin/jq program (if it's the jq program for querying JSON, I have it installed in /usr/bin/jq)

[ghost]

Hi @fpesari , I am happy to work with you to resolve these issues. I already made a first pass at fixing this here

The current build system was really only meant for the end user who prefers to compile their own software, since mido, pymarshal and python-wavefile are not universally available in LInux distros. If you are going to package those as well, I can create a way to skip that step for distro package builds, or just ignore those commands if they fail. If you do not want to package those dependencies, then I can vendor them (but I imagine that is frowned upon by distro maintainers)

As far as sglib (stargate-lib) missing, that is an internal dependency in this repo. If you were building the last release, and using a symlink to $(PREFIX)/bin/stargate to invoke Stargate, then that was a bug regarding the use of symlinks that I have since fixed, and will release soon. But if you would like to send me your logs in ~/stargate/log/*, I can confirm the exact cause.

Also, make install tried to execute the (non-existing on my system) /mingw64/bin/jq program (if it's the jq program for querying JSON, I have it installed in /usr/bin/jq)

Nothing is wrong, that's just noise required to make the same Makefile work on Windows and Linux, it invokes multiple jqs until it finds one that works.

lib/lib64

I'll move those files to $(PREFIX)/share/stargate. They should not be globally accessible in site-packages.

[fpesari]

Hello @stargateaudio and thanks for the fast and accurate response!

Yes, I am probably going to package those Python dependencies (unless they have a deep hierarchy of dependencies themselves, but knowing what they do I doubt it).

While vendoring is often frowned upon, that's mostly because of security issues AFAIK, which IMHO are a bit less serious when we are talking dependencies which do not connect to the Internet. Of course, I can see the possibility for exploits in PyMarshal, for example, so I would understand leaving vendoring as a last resort. But downloading packages from PyPi can be just as unsafe due to malware present on it.

I think the best of both worlds can be achieved by doing this: if there is an installed system-wide dependency, use that, if not, try to download it via pip, else (if there is no Internet) use the vendored one. This way everyone should be happy and would use the safest option by default.

I will wait for the next official release containing these fixes to make a package on OBS so I can send you a proper log.

[ghost]

Files in $(PREFIX)/lib were moved to $(PREFIX)/share. Tonight after work I will come up with a more comprehensive Python module vendoring/fallback strategy

[ghost]

I believe the latest main branch solves all of these problems. Please have a look and let me know if I missed anything.

• Additional Makefile targets and documentation were added for people who do not want any vendored dependencies.
• You can now use a standard build of SBSMS, it will be used in the absence of a vendored copy of stargate-sbsms
• No need to use git submodule anymore, libcds is vendored directly into the engine source code, and by extension, the source releases are now viable to build Stargate DAW

[fpesari]

Good job! I've already started building the stargate package for openSUSE.

Vendoring for Python packages, if done, should be done by you - in other words, this repo should contain the vendored Python dependencies. Right now, in my local build, it downloaded them from the Internet and created a src/sg_py_vendor directory with them but in an Internetless environment, this is not possible.

Also, sbsms fails to build both on my local machine and in the VM. Here is the log.

PS: git seems to be a (soft) build dependency, even if it is not currently listed.

[ghost]

@fpesari please check out the new packaging instructions

For this use-case of packaging without vendored dependencies, there is now:

make distro
# and
make install_distro

It also appears that you are missing jq, I will add that as a build time dependency.

I don't see that SBSMS actually failed, it looks like pip3 install failed.

[ghost]

Also, if you do not want to use vendored SBSMS, you can package the source: https://sourceforge.net/projects/sbsms/

[fpesari]

Hello,

actually, I still had to manually download the Python packages from my local machine before uploading the package on OBS :( but the build failed anyway because pip3 fails. pip3 should not be called if packages are present in the sg_py_vendor directory, otherwise the build will fail.

I made an openSUSE package for libsbsms and sbsms, so I wanted to use that. While compiling them I found out that it's better to use a vendored copy of sbsms by default. It's very lightweight, is not updated very frequently and it is unavailable in some distro repositories so I guess it's best to always vendor it.

[ghost]

actually, I still had to manually download the Python packages from my local machine before uploading the package on OBS :( but the build failed anyway because pip3 fails. pip3 should not be called if packages are present in the sg_py_vendor directory, otherwise the build will fail.

Are you using the make distro and make install_distro targets I added instead of make and make install? Those should not be calling pip, if they are, please send me the build logs so that I can understand why it happened.

[fpesari]

But above you said:

For this use-case of packaging without vendored dependencies, there is now: make distro

I want to use vendored dependencies, so I am using the make command.

[ghost]

I want to use vendored dependencies, so I am using the make command.

OK, got it. I will need to develop a solution for this, will post back to this issue when it's done.

[ghost]

@fpesari , the new Makefile should solve this:

• sg_py_vendor is now a permanent part of the repo
• pip is only called in a specific target, which is not called by any other target
• Every non-core package has it's own target now, so you can pick and choose exactly what you want to install

[ghost]

Although I am concerned about the long term viability of keeping sg_py_vendor in repo, in case I decide to add a binary module later, at least for now I do not plan to add more dependencies.

[fpesari]

Here is the openSUSE package for Stargate :tada: I have not tested it yet so I ask you to kindly wait a bit before closing this ticket.

In a way, I would say "mission accomplished" because this is a standardized build: make and make install behave as expected.

In the long run, I think the best solution would be:

• Use system libraries by default if they are found
• If they are not available, fall back to vendored libraries
• Give packagers the option to use vendored libraries even if there are system libraries, and eventually fine-tune which ones they want to be vendored

Most developers who do this use CMake (but meson would also be fine, even more considering you use Python), which makes this kind of behavior easier to implement and more standardized. Sure, you can use regular make and pkg-config but I think that investing in a build system now pays back in the long run.

[fpesari]

So, I tested it and it works. openSUSE users will be able to install it easily now! Thanks @stargateaudio for cooperating with us from the GeekosDAW project.

[ghost]

Excellent, it has been a pleasure working with you. I will test your packages myself sometime this weekend. I will also evaluate your suggestion of moving to something like Meson soon, need to work through some high priority bugs first.

[ghost]

A couple of things:

1. The new release is up, highly recommend building a package from that
2. Running into trouble trying to test it on OpenSuse Leap

When I try to install GeekOS DAW using the instructions on the website, I get the attached error message. I proceed to install without the package that needs mawk, but then stargate is not available in zypper. When I try to download the package manually (I think maybe it's for Tumbleweed only?), I get Dependency resolution failed when installing. But maybe this is user error, I have not used OpenSuse in a while.

[fpesari]

Hello:

1. Done. Could you please adopt a more standardized name for releases? {name}-{version} rather than {name}-release-{version}
2. It was Tumbleweed-only because of a missing line in the RPM spec, I fixed it, thanks.

About that error: it seems the mawk package is discontinued on Leap. I tried to fix, perhaps it will work now. Thank you for reporting it! Aside from that, those instructions are more for people who want some programs preinstalled. You can just add the GeekosDAW repo via YaST > Software management > Configuration > Repositories and you will have access to all our packages. These are the URLs (set a priority lower than 99):

Leap 15.3: https://download.opensuse.org/repositories/home:/geekositalia:/daw/openSUSE_Leap_15.3/ Tumbleweed: https://download.opensuse.org/repositories/home:/geekositalia:/daw/openSUSE_Tumbleweed/

[oldcastlehq]

@fpesari I tested on Leap 15.3 and it gave me this error:

Adding PYTHONPATH /usr Did not detect local development environment, trying system installation Adding PYTHONPATH /usr/share/stargate/stargate [2021-11-13 20:02:52,092] INFO /usr/bin/stargate : 72 - sys.argv == ['/usr/bin/stargate'] [2021-11-13 20:02:52,093] ERROR /usr/share/stargate/stargate/sglib/log.py: 90 - Traceback (most recent call last):

File "/usr/share/stargate/stargate/sgui/sgqt.py", line 7, in import PyQt6

ModuleNotFoundError: No module named 'PyQt6'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/bin/stargate", line 170, in main()

File "/usr/bin/stargate", line 167, in main start_stargate()

File "/usr/bin/stargate", line 95, in start_stargate app, scaler = _setup()

File "/usr/bin/stargate", line 73, in _setup from sgui.sgqt import QApplication, QGuiApplication, QtCore

File "/usr/share/stargate/stargate/sgui/sgqt.py", line 11, in import PyQt5

ModuleNotFoundError: No module named 'PyQt5'

I think it's not installing all the dependencies.

[fpesari]

Thank you for reporting the error @oldcastlehq, please try now

[oldcastlehq]

@fpesari thanks. It installs now, but it crashes after startup. I'll report this to @stargateaudio.

[oldcastlehq]

Hi @fpesari. From the error below, I think some dependencies are still missing. Is that right?

oldcastle@hp:~/Public/butikk> stargate Adding PYTHONPATH /usr Did not detect local development environment, trying system installation Adding PYTHONPATH /usr/share/stargate/stargate [2021-11-14 10:31:51,430] INFO /usr/bin/stargate : 72 - sys.argv == ['/usr/bin/stargate'] [2021-11-14 10:31:51,431] INFO /usr/share/stargate/stargate/sgui/sgqt.py: 19 - Using PyQt5 [2021-11-14 10:31:51,655] INFO /usr/share/stargate/stargate/sglib/lib/util.py: 82 - install prefix: /usr [2021-11-14 10:31:51,656] INFO /usr/share/stargate/stargate/sglib/lib/util.py: 199 - Using SBSMS: /usr/bin/stargate-sbsms [2021-11-14 10:31:51,656] INFO /usr/share/stargate/stargate/sglib/lib/util.py: 275 - BIN_PATH==/usr/bin/stargate-engine [2021-11-14 10:31:51,656] INFO /usr/share/stargate/stargate/sglib/lib/util.py: 637 - BIN_PATH == /usr/bin/stargate-engine [2021-11-14 10:31:51,694] WARNING /usr/bin/stargate : 82 - Unable to set QGuiApplication.setHighDpiScaleFactorRoundingPolicy type object 'QGuiApplication' has no attribute 'setHighDpiScaleFactorRoundingPolicy' [2021-11-14 10:31:51,727] WARNING /usr/share/stargate/stargate/sglib/models/theme.py: 706 - Theme file: 'None', does not exist, using default [2021-11-14 10:31:51,727] INFO /usr/share/stargate/stargate/sglib/models/theme.py: 711 - Using theme file /usr/share/stargate/themes/default/default.sgtheme [2021-11-14 10:31:51,737] INFO /usr/share/stargate/stargate/sglib/models/theme.py: 602 - Overriding {'highlight_border': '#ffffff', 'transport_button_size': 40, 'transport_clock_background': '#333390', 'transport_clock_color': '#cccccc', 'assets_subdir': 'dark', 'background_color': '#17181d', 'background_color_alt': '#1f1f27', 'background_color_disabled': '#181818', 'border_color': '#444444', 'button_pressed': '#2d2d2d', 'checkbox_check': '#cccccc', 'combobox': '#1f1f27', 'highlight': '#cccccc', 'hscrollbar': '#666666', 'hscrollbar_groove': '#111111', 'item_handle': '#cccccc', 'text_background_color': '#1f1f27', 'text_color': '#cccccc', 'text_color_alt': '#1e1e1e', 'text_color_disabled': '#d0d0d0', 'tab_selected': '#aaaaaa', 'track_panel_background': '#17181d', 'vscrollbar': '#666666', 'vscrollbar_groove': '#111111'} [2021-11-14 10:31:51,738] INFO /usr/share/stargate/stargate/sglib/models/theme.py: 603 - with {'combobox': 'qlineargradient(\n x1:0, y1:0, x2:0, y2:1,\n stop:0 #1f1f27, stop: 0.3 #23232b, stop: 0.6 #23232b, stop:1 #1f1f27\n)', 'hscrollbar': 'qlineargradient(\n x1:0, y1:0, x2:1, y2:0,\n stop:0 #555555, stop: 0.5 #777777, stop:1 #555555\n)', 'hscrollbar_groove': 'qlineargradient(\n x1:0, y1:0, x2:0, y2:1,\n stop:0 #444444, stop: 0.5 #222222, stop:1 #444444\n)', 'transport_clock_background': 'qlineargradient(\n x1: 0, y1: 0, x2: 0, y2: 1,\n stop: 0 #515190, stop: 0.3 #333390,\n stop: 0.7 #303090, stop: 1 #111190\n)', 'vscrollbar': 'qlineargradient(\n x1:0, y1:0, x2:0, y2:1,\n stop:0 #555555, stop: 0.5 #777777, stop:1 #555555\n)', 'vscrollbar_groove': 'qlineargradient(\n x1:0, y1:0, x2:1, y2:0,\n stop:0 #444444, stop: 0.5 #222222, stop:1 #444444\n)'} [2021-11-14 10:31:51,738] INFO /usr/share/stargate/stargate/sglib/models/theme.py: 605 - Result: {'highlight_border': '#ffffff', 'transport_button_size': 40, 'transport_clock_background': 'qlineargradient(\n x1: 0, y1: 0, x2: 0, y2: 1,\n stop: 0 #515190, stop: 0.3 #333390,\n stop: 0.7 #303090, stop: 1 #111190\n)', 'transport_clock_color': '#cccccc', 'assets_subdir': 'dark', 'background_color': '#17181d', 'background_color_alt': '#1f1f27', 'background_color_disabled': '#181818', 'border_color': '#444444', 'button_pressed': '#2d2d2d', 'checkbox_check': '#cccccc', 'combobox': 'qlineargradient(\n x1:0, y1:0, x2:0, y2:1,\n stop:0 #1f1f27, stop: 0.3 #23232b, stop: 0.6 #23232b, stop:1 #1f1f27\n)', 'highlight': '#cccccc', 'hscrollbar': 'qlineargradient(\n x1:0, y1:0, x2:1, y2:0,\n stop:0 #555555, stop: 0.5 #777777, stop:1 #555555\n)', 'hscrollbar_groove': 'qlineargradient(\n x1:0, y1:0, x2:0, y2:1,\n stop:0 #444444, stop: 0.5 #222222, stop:1 #444444\n)', 'item_handle': '#cccccc', 'text_background_color': '#1f1f27', 'text_color': '#cccccc', 'text_color_alt': '#1e1e1e', 'text_color_disabled': '#d0d0d0', 'tab_selected': '#aaaaaa', 'track_panel_background': '#17181d', 'vscrollbar': 'qlineargradient(\n x1:0, y1:0, x2:0, y2:1,\n stop:0 #555555, stop: 0.5 #777777, stop:1 #555555\n)', 'vscrollbar_groove': 'qlineargradient(\n x1:0, y1:0, x2:1, y2:0,\n stop:0 #444444, stop: 0.5 #222222, stop:1 #444444\n)'} [2021-11-14 10:31:51,775] INFO /usr/share/stargate/stargate/sgui/util.py: 258 - Setting icon to '/usr/share/pixmaps/stargate.ico' [2021-11-14 10:31:51,798] INFO /usr/share/stargate/stargate/sglib/lib/pidfile.py: 37 - Creating pidfile ~/stargate/ui.pid : 25213 /usr/share/stargate/stargate/share/stargate/stargate [2021-11-14 10:31:51,883] INFO /usr/share/stargate/stargate/sgui/welcome.py: 93 - Project history: [] [2021-11-14 10:31:56,259] WARNING /usr/bin/stargate : 59 - qt_message_handler: 1 None:0:None "QXcbConnection: XCB error: 3 (BadWindow), sequence: 684, resource id: 37751246, major code: 40 (TranslateCoords), minor code: 0" [2021-11-14 10:32:03,305] ERROR /usr/share/stargate/stargate/sglib/log.py: 90 - Traceback (most recent call last):

    File "/usr/bin/stargate", line 170, in <module>
      main()

    File "/usr/bin/stargate", line 167, in main
      start_stargate()

    File "/usr/bin/stargate", line 121, in start_stargate
      from sgui.main import main

    File "/usr/share/stargate/stargate/sgui/main.py", line 29, in <module>
      from sgui.daw import entrypoint as daw

    File "/usr/share/stargate/stargate/sgui/daw/entrypoint.py", line 23, in <module>
      from .hardware import HardwareWidget, MidiDevicesDialog

    File "/usr/share/stargate/stargate/sgui/daw/hardware.py", line 4, in <module>
      from sgui.daw.sequencer.audio_input import AudioInputWidget

    File "/usr/share/stargate/stargate/sgui/daw/sequencer/__init__.py", line 14, in <module>
      from .seq import ItemSequencer

    File "/usr/share/stargate/stargate/sgui/daw/sequencer/seq.py", line 29, in <module>
      from sgui.daw.lib.midi_file import DawMidiFile

    File "/usr/share/stargate/stargate/sgui/daw/lib/midi_file.py", line 4, in <module>
      from sglib.lib.midi import load_midi_file

    File "/usr/share/stargate/stargate/sglib/lib/midi.py", line 2, in <module>
      import mido

    ModuleNotFoundError: No module named 'mido'

[fpesari]

That's because mido was not vendored along with pymarshal and wavefile and there is no openSUSE package for it. I think I might make one because it fits the GeekosDAW repo.

You can get around it via pip3 install mido --user for now but I will try to package mido within a day or two. Meanwhile, if @stargateaudio wants to vendor it since it's not a super-common package, that'd also be good.

[fpesari]

I made an openSUSE package for mido, too. Please try re-installing stargate from GeekosDAW and let me know what happens.

[ghost]

I also added mido to the vendor package

[oldcastlehq]

All good now. Thanks!