search

mastodon / mastodon

Your self-hosted, globally interconnected microblogging community
https://joinmastodon.org
GNU Affero General Public License v3.0
46.81k stars 6.91k forks source link

Debian package is necessary. #3576

Closed matyapiro31 closed 5 months ago

matyapiro31 commented 7 years ago

mastodon has no Debian package.

when debian package for mastodon is created, we're able to shrink Dockerfile simply and absorb the difference among version and distribution.

aral commented 7 years ago

I was just about to open an issue for this also. Having a Debian package would make it exponentially easier to set up your own instance and would result in a considerable rise in the instance count.

Imagine:

How to set up Mastodon on Ubuntu (CloudScale.ch, Digital Ocean, etc.)

  1. sudo apt-get update
  2. sudo apt-get install mastodon
matyapiro31 commented 7 years ago

I am expert (but just a few ) of debian package, and maybe it is easy to create for mastodon once, but a package you must maintain takes much time.Can be changed as the way to build be changing.

progval commented 7 years ago

maybe it is easy to create for mastodon once

I am not sure about this. Mastodon has 927 nodejs dependencies and 201 ruby dependencies.

Gargron commented 7 years ago

Mastodon has 927 nodejs dependencies and 168 ruby dependencies

This is not unusual for a project of this scale. This doesn't stop Gitlab from being a debian/ubuntu package.

Generally for this thread: https://github.com/chef/omnibus

matyapiro31 commented 7 years ago

I think it may not contain so many packages in node.js and ruby to assign on the package setting directly. Those package manager also need editing to specify dependencies.

guillaume-uH57J9 commented 7 years ago

I am not sure about this. Mastodon has 927 nodejs dependencies and 201 ruby dependencies.

That's good point. I would think twice about installing a package if apt-get asks me "Installing 1128 packages for dependencies. Do you want to continue ? "

Rationalizing nodjs/ruby dependencies in Mastodon (ie reducing their #), or making some of them optional, would go a long way toward packaging.

aral commented 7 years ago

@guillaume-uH57J9 Use case: I want to install Mastodon on my server.

For this use case, I don’t really care what the details of the installation are: Could be 1 dependency, could be 1 million dependencies – the beauty of having a package installer is that I don’t have to care; it’s a black box. My concern is: can I write a single command, hit OK, and boom – a little while later – I have Mastodon installed. Yay! (Much rejoicing, etc.) :)

That said, of course, by all means, please reduce dependencies, etc., if it streamlines the overall process but it’s not a prerequisite for this or the bottleneck that this issue aims to solve. (The bottleneck to new instance adoption is installation complexity and time.) The number of dependencies isn’t really a factor given the tremendous value a package will provide in terms of reducing installation time from hours to minutes – orders of magnitude – and upgrades to a single command.

matyapiro31 commented 7 years ago

Most important role of packaging is to separate the environment for running from the env for building. Can reduce npm package when running isnt what i know; maybe ruby package is unnecessary then.

aral commented 7 years ago

@matyapiro31 Again, as far as my use case for this goes (and I’m happy to open a separate issue if you feel I’m hijacking your use case) :) the most important reason for this is to reduce installation time on the most popular hosting platforms (Ubuntu-based) from hours to minutes (an order of magnitude change that will impact instance adoption rates substantially) and maintenance to a single command. If there are any advantages for developers, that’s an additional bonus but my concern here is in encouraging instance adoption.

matyapiro31 commented 7 years ago

For example, http-server is convenient package for testing, but not necessary when run:

$ npm install -g http-server

Mastodon doesn't use http-server , but there are package installed for similar reason.

matyapiro31 commented 7 years ago

@Gargron could you list up what packages are used now?

Gargron commented 7 years ago

I do not want to list or justify Mastodon's runtime dependencies. We do not try to make the project bigger on purpose, and everyone who is contributing to the codebase sees that we need the dependencies that we are using. What you are describing ("convenient package for testing, but not necessary when run") is already covered by environments: development, test, production. Dependencies are already sorted accordingly, and production dependencies do not include anything which is not used in production.

You also shouldn't worry about them, I think. What all of these "packaging" tools do, they manage system packages (think postgres, redis, node, protobuf). Ruby and Node dependencies still get installed the usual way, with no difference to manual installation. For a list of system dependencies, look at the production guide: https://github.com/tootsuite/documentation/blob/master/Running-Mastodon/Production-guide.md#general-dependencies

progval commented 7 years ago

everyone who is contributing to the codebase sees that we need the dependencies that we are using

I think some of them could be avoided. I am not very good at Javascript, but I think the dependency on classnames could be removed, for example.

It was added in 860ffc05602b148769b87d0cda39985feb9a8486, apparently just to replace button ${this.props.secondary ? 'button-secondary' : ''} ${this.props.block ? 'button--block' : ''} by:

classNames('button', this.props.className, {
      'button-secondary': this.props.secondary,
      'button--block': this.props.block,
    })

I know it is a small dependency, but it should be possible to remove it at very little cost.

matyapiro31 commented 7 years ago

maybe this helps us to create a package.

plus: /Aptfile and /Vagrantfile can be reduced by writing dependencies.

matyapiro31 commented 7 years ago

@Gargron Finally, you think package is necessary?

akuckartz commented 7 years ago

:+1:

guillaume-uH57J9 commented 7 years ago

I don’t really care what the details of the installation are: Could be 1 dependency, could be 1 million dependencies

Some users might not care, and it won't change much at installation time.

But for package maintainers this is day and night. Packaging an application + 1 dependency, and maintaining them, is a relatively simple/quick task. Packaging 1 application + 1 billion dependencies (and maintaining them) cannot be accomplished by one person. It'd require a whole team.

Jehops commented 7 years ago

For the FreeBSD package we punted on the node dependencies. All the OS and ruby/rails dependencies are covered with pkg install mastodon, but then the user still has to do, e.g., sudo su - mastodon; yarn install; RAILS_ENV=production rails assets:precompile. Doing the job of npm/yarn is too ambitious (for me) at this point; we would have to bake a similar infrastructure we have for dealing with ruby packages into the FreeBSD ports/package system.

While it's not a perfect package, most of the feedback I have received from Mastodon administrators running FreeBSD has been positive.

Gargron commented 7 years ago

@Jehops I actually don't really understand why bundler and yarn dependencies cannot be bundled in the package. There is literally no chance that running bundle install or yarn install will yield a different result unless the Mastodon lockfiles change (e.g. in a new release). That's the exact purpose of lockfiles - reproducible, deterministic bundling.

Jehops commented 7 years ago

@Gargron, for each rails dependency, there is a one-to-one correspondence with a FreeBSD port/package and each of those FreeBSD ports/packages handles dependencies. Basically, the FreeBSD port/package system is used in place of gem and bundler. That that sort of solution for the yarn dependencies would be nice, but a lot of work. It may be feasible in the short term to make one omnibus mastodon-yarn-dependencies package.

matyapiro31 commented 7 years ago

@Gargron mastodon's package usage design is not better to manage package. We are forced to use newer version of node, ruby, yarn etc than which in distribution packages. It makes us very complicated to control mastodon package. There are at least 5 package varieties: native packages for mastodon,npm packages, gem packages, native package for gem, native packages in sub repositories, and we must check if those packages has bad affect on other packages, this is the reason why very difficult to maintain package.

matyapiro31 commented 6 years ago

I now somehow understand how mastodon works, however it's still very difficult to create debian package.

Mastodon depends on many software that is not packaged correctly for debian. Supplying these packages with appropriate update is very difficult.

ghost commented 6 years ago

1) FYI there is a RFP for this on debian's side https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859741

2) "Mastodon depends on many software that is not packaged correctly for debian" If anyone has identified any of these packages, either this thread or the thread I linked to above would be appropriate places to mention them so we can start filing RFPs and getting those packages in debian, too.

3) "these prerequisites change all the time" That's the whole point of debian - to have a stable basis of development, something that we can actually audit and a known 'good state' that we can build from in case someone puts showstopper bugs in.

progval commented 6 years ago

If anyone has identified any of these packages, either this thread or the thread I linked to above would be appropriate places to mention them so we can start filing RFPs and getting those packages in debian, too.

Ruby:

nodejs:

ghost commented 6 years ago

Things you can check off

Things you can probably add to that list:

narcisgarcia commented 2 years ago

What about dependencies when upgrading Mastodon without .deb package?

limdingwen commented 1 year ago

FYI A package exists for Arch Linux and NixOS.

matyapiro31 commented 1 year ago

Debian Package system is very restricted. What customers want is the best package, that can be installed, updated, removed with just apt package.

narcisgarcia commented 1 year ago

There are diverse options for this: Request for package at official Debian repositories (examples: wordpress, mediawiki), create a self or hosted APT repository (examples: gpxsee, WineHQ, Skype, Gitlab, Jitsi), and publish simple .deb package (example: Skylobby, Adobe reader, Gigablast).

These three options do not exclude between them, as they can be steps to reach the best final situation (1.package, 2.self-repo, 3.Debian-repo).

laszlovl commented 1 year ago

I created a Github workflow that compiles Mastodon from source, packages it as a deb then publishes that deb as a release asset: https://github.com/laszlovl/mastodon-build

Note that installing this package just places the compiled version of Mastodon in /opt/mastodon, it doesn't provide configuration or start any services (it was meant for use with https://github.com/laszlovl/mastodon-ansible-scalable)

matyapiro31 commented 1 year ago

Debian packages are very complicated, it's not just writing shell script to install. It's not pacman, Debian is not Arch Linux. Making a Debian package has many rules, many script, many tests. If mastodon provides package, it should provide complete settings in 1 step installation.

narcisgarcia commented 1 year ago

First step is to package files without scripts and only taking care of separating local settings files from default packaged files. Many server software is packaged without completing settings in 1 step installation: Dovecot, Mailman, etc.

The right alternative to packaging is to deploy software as a standard CMS being possible to install it through FTP and not CLI, same as Wordpress only needing LAMPP without specific OS environment requirements.

Main difficult to install, maintain and upgrade software without package integration, is dependencies matter. Packages focused to distributions allow developer/packager to solve dependencies very well for people who deploy Mastodon nodes.

MarcusWolschon commented 1 year ago

I was just about to open an issue for this also. Having a Debian package would make it exponentially easier to set up your own instance and would result in a considerable rise in the instance count.

Imagine:

How to set up Mastodon on Ubuntu (CloudScale.ch, Digital Ocean, etc.)

1. `sudo apt-get update`

2. `sudo apt-get install mastodon`

Not just to set up. It would make it feasable to maintain a host for years for the administrator. Move to a new host, do desaster-recovery or otherwise set up an entire server again.

Mastodon is not the only service offered on the host, but the only one that is not in the package-management. Thus doesn't automatically get checked for regular updates or security relevant updates.

MarcusWolschon commented 1 year ago

What about dependencies when upgrading Mastodon without .deb package?

You don't. You really, really don't. Managing dependencies, maintaining all configuration as per the proper FHS (so it is backed up and restored by the admin along with all other services of the host) and informing about updates with special emphasis on security relevant upates is the very job of a package management.

I don't even consider setting up a Mastodon instance unless there is a maintained Debian package. Without, it doesn't play well with Main-, Web-, Database-, file hosting-, ... services on my host and I'll not have any part in hosting it for my users.

sjlongland commented 1 year ago

My grievance here is the system requirements needed to compile Mastodon, are orders of magnitude greater than that for it to run.

I run my own instance, on a VM that runs on my own hardware. The host is near maxxed out with RAM, and buying RAM for these machines is not as easy as "go down the shop". I have as much allocated to this VM as I can practically manage right now. At run-time, it works fine.

I find in particular, the usual build steps barf when I try to build the front-end assets. Like this:

$ RAILS_ENV=production bundle exec rake assets:precompile
yarn install v1.22.19
[1/6] Validating package.json...
[2/6] Resolving packages...
[3/6] Fetching packages...
[4/6] Linking dependencies...
warning Workspaces can only be enabled in private projects.
[5/6] Building fresh packages...
[6/6] Cleaning modules...
Done in 107.90s.
Compiling...
Compilation failed:
Browserslist: caniuse-lite is outdated. Please run:
  npx update-browserslist-db@latest
  Why you should do it regularly: https://github.com/browserslist/update-db#readme

<--- Last few GCs --->

[1369043:0x5746cc0]   218321 ms: Scavenge 478.8 (497.2) -> 477.0 (497.2) MB, 2.3 / 0.0 ms  (average mu = 0.748, current mu = 0.812) allocation failure
[1369043:0x5746cc0]   218346 ms: Scavenge 478.9 (497.2) -> 477.1 (497.4) MB, 2.8 / 0.0 ms  (average mu = 0.748, current mu = 0.812) allocation failure
[1369043:0x5746cc0]   218365 ms: Scavenge 479.1 (497.4) -> 477.3 (501.7) MB, 2.8 / 0.0 ms  (average mu = 0.748, current mu = 0.812) allocation failure

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb08e80 node::Abort() [node]
 2: 0xa1b70e  [node]
 3: 0xce1890 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xce1c37 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xe992a5  [node]
 6: 0xea8f6d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 7: 0xeabc6e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0xe6d1aa v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
 9: 0x11e5f96 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
10: 0x15d9c19  [node]

I have to rsync the work tree back to another machine, re-run the same steps there, then rsync the results back.

To say this is messy and error prone is an understatement.

What are my options?

As an instance admin, I really don't care if the "Debian package" is little more than a pre-compiled tarball that I can unpack somewhere, rsync over-the-top (with --exclude calls to preserve critical files), then re-start some services. That would be 100% better than what we have now.

CyberTailor commented 10 months ago

Prebuilt web assets (published in Releases) would be a nice addition

renchap commented 5 months ago

Closing this as we do not plan to support a Debian package ourselves.

If the community (or a Debian maintainer) wants to provide one they can, but our recommended way of deployments are either from git, or through our official container image.

narcisgarcia commented 5 months ago

Unlike @renchap closing tag, developers and implementers should go to https://github.com/laszlovl/mastodon-build to continue this subject.

MarcusWolschon commented 4 months ago

Unlike @renchap closing tag, developers and implementers should go to https://github.com/laszlovl/mastodon-build to continue this subject.

That one downloads and builds the source code. That's not a Debian binary package and doesn't name or manage it's dependencies. So it needs to be re-build every single time any package on the server is updated because it could be a dependency.

Looking for some other server that speaks ActivityPub. Something that runs in a LAMP environment perhaps. Mastodon just doesn't seem to be for people who don't want to have to maintain, backup, restore and operate services that build their own parallel universes away from the existing dependency management.

CyberTailor commented 4 months ago

@MarcusWolschon i suggest trying snac2 or honk, they are very minimal ActivityPub servers written in C

narcisgarcia commented 4 months ago

That one downloads and builds the source code. That's not a Debian binary package and doesn't name or manage it's dependencies. So it needs to be re-build every single time any package on the server is updated because it could be a dependency.

laszlovl's mastodon-build is a project to do what it says. Sure you can help developer to complete package's features. As you've seen, mastodon project is not packaging it for Debian; then "mastodon-build" is a project started to reach this goal. IT'S THE RIGHT DIRECTION ABOUT THIS THREAD SUBJECT.

...or maybe you are only looking for "free as in beer" software with everything already made to satisfy Github visitor.

MarcusWolschon commented 3 months ago

That one downloads and builds the source code. That's not a Debian binary package and doesn't name or manage it's dependencies. So it needs to be re-build every single time any package on the server is updated because it could be a dependency.

laszlovl's mastodon-build is a project to do what it says. Sure you can help developer to complete package's features. As you've seen, mastodon project is not packaging it for Debian; then "mastodon-build" is a project started to reach this goal. IT'S THE RIGHT DIRECTION ABOUT THIS THREAD SUBJECT.

...or maybe you are only looking for "free as in beer" software with everything already made to satisfy Github visitor.

I'm busy maintaining other software and will not host and administer services on my server for my users that doesn't play nice with the dependency management of everything else on that box. Therefore no Mastodon as ActivityPub implementation for me.