search

daq-tools / kotori

A flexible data historian based on InfluxDB, Grafana, MQTT, and more. Free, open, simple.
https://getkotori.org/
GNU Affero General Public License v3.0
108 stars 17 forks source link

Optimize packaging #23

Closed amotl closed 4 years ago

amotl commented 4 years ago

We've learned from @RuiPinto96 and @Dewieinns through #7, #19 and #22 that the packaging might not be done appropriately.

Within this issue, we will try to walk through any issues observed. Thanks again for your feedback, we appreciate that very much!

amotl commented 4 years ago

[...] why Kotori pulls in such a wrath of dependencies through the Debian package installation path which might not be required at all (don't know why it actually should need Chief, Icecast, Jack Audio, Kerberos and Postfix)

-- https://github.com/daq-tools/kotori/issues/7#issuecomment-593163782

We just learned from [1] that

--install-suggests will be recursive, meaning that it will also calculate suggestions from all the packages.

Installing all dependencies recursively is not really intended.

So, we should remove all the --deb-suggests lines from the fpm builder script [2] and better put them within --deb-recommends.

[1] https://askubuntu.com/questions/117635/how-to-install-suggested-packages-in-apt-get [2] https://github.com/daq-tools/kotori/blob/dfa62db1cfc03f9ec716f0ec494d4479e44eba35/packaging/builder/fpm-package#L37-L39

amotl commented 4 years ago

Regarding installation on Ubuntu vs. Debian, we found that the libc6 versions are different between the systems, which will essentially stop Kotori from being invoked properly when using the package built on Debian 10 Buster within Ubuntu 18 Bionic Beaver.

/opt/kotori/bin/python: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /opt/kotori/bin/python)

As already found within #7, the Python version will also matter because of the ImportError: cannot import name _remove_dead_weakref issue, so we can't use the package built on Debian 9 Stretch either.

amotl commented 4 years ago

We've just released Kotori 0.24.5 and tried to streamline the installation experience. We tested it on a Debian "buster" 10 virtual machine and it worked flawlessly. Packages for Debian "stretch" 9 have also been updated.

We also published a Docker image to Docker Hub [1], which can be used on all other machines. A minimal documentation about that is available at [2]. The image is based on Debian "buster" 10.

We will be happy to get feedback from you about the updates we made.

[1] https://hub.docker.com/repository/docker/daqzilla/kotori [2] https://getkotori.org/docs/setup/docker.html

Dewieinns commented 4 years ago

I realize this is not your preferred OS but I upgraded to Kotori 0.24.5 this morning on my Ubuntu server. Again I was presented with the issue: /opt/kotori/bin/python: /lib/x86_64-linux-gnu/libc.so.6: version "GLIBC_2.28" not found (required by /opt/kotori/bin/python)

I was able to resolve this again by running: virtualenv --python /usr/bin/python2.7 /opt/kotori

amotl commented 4 years ago

Dear Andrew,

yeah, I also found that glibc mismatch when testing the build for Debian 10 on Ubuntu 18. Do you believe we should also build and publish packages for Ubuntu 18 given the fact that there are now Docker images available [1]?

Apart from that, did the optimized packages work better for you than before? So, the fixup script will not be required anymore and the acquisition of this large number of packages is gone?

With kind regards, Andreas.

[1] https://getkotori.org/docs/setup/docker.html

amotl commented 4 years ago

I realize Ubuntu is not your preferred OS [...]

While we prefer vanilla Debian, we are finally giving in and just expanded our package repository to support Ubuntu 18 Bionic Beaver through 05cc89d8, see https://github.com/daq-tools/kotori/issues/7#issuecomment-594348406. We hope this further improves the installation experience for all Ubuntu users out there and will be happy to hear about the outcome for you.

Dewieinns commented 4 years ago

yeah, I also found that glibc mismatch when testing the build for Debian 10 on Ubuntu 18. Do you believe we should also build and publish packages for Ubuntu 18 given the fact that there are now Docker images available [1]?

Because the docker image was available I had decided it may not be necessary - but I already had things working with Ubuntu 18 and decided to stick with it.

Apart from that, did the optimized packages work better for you than before? So, the fixup script will not be required anymore and the acquisition of this large number of packages is gone?

I have not re-installed from scratch again. Installing only the basic packages necessary made a substantial difference in the number of packages installed. You have already addressed why in a separate issue.

While we prefer vanilla Debian, we are finally giving in and just expanded our package repository to support Ubuntu 18 Bionic Beaver through 05cc89d, see #7 (comment). We hope this further improves the installation experience for all Ubuntu users out there and will be happy to hear about the outcome for you.

I will be spinning up a different instance on a remote (not internal) server in the near future. I will certainly provide feedback when this happens.

amotl commented 4 years ago

Dear Andrew,

I just wanted to let you know that we've just updated the setup guidelines through ed63e4f5 in order to account for getting the most recent Mosquitto packages directly from their repository.

Cheers, Andreas.

[1] https://getkotori.org/docs/setup/debian-quickstart.html#register-the-mosquitto-package-repository

amotl commented 4 years ago

Dear Andrew,

I feel packaging works reasonably well right now. At some time in the future, we might want to look at #6.

Thank you very much for bothering me about the sad state of Kotori packaging. I hope others will find an easier time to get started.

Please let me know of you experience any other quirks with Kotori.

With kind regards, Andreas.