Build and publish Debian packages for Red

laggron42 commented 4 years ago

Hello, after a discussion in AC, I started working on a Debian package (.deb) for Red-DiscordBot, with all the automated stuff.

Doing this will make the install process as easy as possible on all Debian-based Linux distros, such as Linux Mint or Ubuntu. Installing the bot would only require a single command:

sudo apt install red-discordbot

Python, git, dependencies, venvs, everything is included and the next command would simply be redbot-setup. As easy as this.

Current status

For now I'm messing inside an ubuntu-server virtualbox, see if I can successfully build a debian package and make it work. Then I will work on automated builds and publications.

I'm using dh-virtualenv to automatically build the Python package, and it works pretty well! This is the test and crash part, I'm far from opening a PR, which means it's the time to discuss about your concerns and suggestions about this.

With the setup below, I got a mostly working Debian package. The binary files are located over /opt/venvs/, I need to figure out how to add this to the path, or copy the files to a known directory included in path.

Once this works, I'm gonna have to write tests and automated builds with Github Actions, plus the publication to Debian/Ubuntu archives.

How to test

If you want to try what I've done so far, here are the steps (for Ubuntu 20.04):

Install the deps

sudo apt-get install build-essential debhelper devscripts equivs python3-all python3-setuptools python3.8-dev python3-pip dh-python git openjdk-11-jre-headless

Install dh-virtualenv. There's no source for Ubuntu 20.04 yet, so you must build from source.

Clone the repo, gud fresh install, with the name of the release

git clone https://github.com/Cog-Creators/Red-DiscordBot red-discordbot_3.4.0

Create a debian folder inside the cloned repo and insert the following files (subject to change. a lot):

control

Source: red-discordbot
Section: python
Priority: extra
Maintainer: "Cog-Creators" <cogcreators@gmail.com>
Standards-Version: 3.4.0
Build-Depends: debhelper (>= 11~), dh-virtualenv (>= 0.8), python3-all, python3-setuptools, python3.8-dev, python3-pip, dh-python, git, openjdk-11-jre-headless

Package: red-discordbot
Architecture: any
Depends: ${python3:Depends}, ${misc:Depends}
X-Python3-Version: >= 3.8.1
Homepage: https://github.com/Cog-Creators/Red-DiscordBot
Description: A highly customisable Discord bot

rules

#! /usr/bin/make -f

%:
    dh $@ --with python-virtualenv

changelog

red-discordbot (3.4.0) UNRELEASED; urgency=medium

  * Initial release.

 -- El Laggron <laggron42@gmail.com>  Mon, 17 Aug 2020 22:49:36 +0000

compat (it's important I swear, don't omit)
```
9
```

Check that you have 4 files with ls debian.

Build at the root of the repo

( deactivate ; dpkg-buildpackage -us -uc -b )

If the build is successful, you will have plenty of files in the parent (..) directory. Look for the .deb and try to run apt install ./file.deb.

If the install is successful, you should be able to see the result over /opt/venvs/red-discordbot/, that's where the binary files are.

Basic issues

[ ] The venv isn't writable. Needs to modify the permissions of the dir on install.
[ ] The bin files aren't in path. Need to either make the install process add /opt/venvs/red-discordbot/bin in PATH, or make a syslink inside /usr/bin, which is the option recommended by dh-python.

Issues that needs discussion

The installed venv is system-wide, which is a radical change from a normal redbot install

A problem raised by an install with apt is that the installation location (and the env) becomes system-wide, instead of user, or venv wide. This means that, if you create multiple instances, even with different users, they will share the same environment.

There is still a venv, it will not conflict with the other packages that can be installed. It's just an unique venv that, for now, can't be duplicated.

This problem isn't actually new, but apt makes it harder to fix, because it's theoretically not possible to create multiple venvs with the same apt install. If people want a different venv, they would have to roll back to the initial installation method.

What could we do to fix this? Not changing code on debian package's end, but directly in core Red, to allow for example one venv per instance?

synrg commented 4 years ago

I think multiple venvs is a developer problem, not so much a user problem. Wouldn't the typical user deployment be fine with just one venv? And wouldn't developers rather just install the normal way and not from a package? I don't see this as a huge limitation.

Drapersniper commented 4 years ago

Imo ... it a user wants multiple venvs they can use another install method. The idea of this one is simplicity, sometimes for simplify you need to give up on feature/customisability

Jackenmen commented 4 years ago

Each instance of Red can have different cogs installed with different dependencies (pip libraries). While currently Downloader installs dependencies using pip's --target option (which install into a folder), this option comes with its problems that we want to fix. Current idea (as in, before apt package was brought to the table) was to make Downloader install dependencies into venv Red is installed in (when Red is installed in venv), which would solve those issues. Now though, the issue would be not being able to have multiple instances on one server because the instances would have conflicting dependencies. Also, even installing the dependencies into apt package's venv doesn't seem like perfect option either - it would require Red to be able to write to apt package's venv, which I don't think is that good idea (and what about security of that?)?

Cog-Creators / Red-DiscordBot