search

hometown-fork / hometown

A supported fork of Mastodon that provides local posting and a wider range of content types.
GNU Affero General Public License v3.0
734 stars 56 forks source link

Bump hometown to Mastodon 4.2.9 #1337

Open nachtjasmin opened 7 months ago

nachtjasmin commented 7 months ago

Update 2023-12-29: It is ready for review! Update 2023-12-30: I just deployed it on queer.group, so yes, this PR is already running in production.

This PR is a replacement for #1325, since I started from scratch. The last tag, v4.0.10+hometown-1.1.1 is used as the root for this PR.

For anyone who wants to support me, feel free: https://ko-fi.com/queergroup

Current TODO list:

And then it's time for the integration tests.

dariusk commented 7 months ago

Thank you for getting started on this!!!!!

github-actions[bot] commented 7 months ago

This pull request has merge conflicts that must be resolved before it can be merged.

nachtjasmin commented 7 months ago

All tags, including v4.2.1 have been merged. :partying_face:

However, all those linters are failing and probably for a good reason. I already found several places where either I or Git fucked up with the merges, therefore there's still some manual work to do.

Luckily it's not that bad, I think, most of the stuff looks very easy to fix. The v4.2.0 merge was quite a beast, no wonder that there are some errors. ^^

graue commented 7 months ago

Thank you so much for working on this!

pronoiac commented 7 months ago

I suggest adding:

Also, I poked around and realized, this includes a fix for our instance's translation issues! Thanks!

nachtjasmin commented 7 months ago

@pronoiac: I've cherry-picked the first PR, but not the second. I'd like to, but it depends on stuff which is only in the next version of Mastodon. Guess, you have to wait a little more then. ^^

(Funfact: I've tried to merge mastodon/main into this branch and the list of conflicts is surprisingly low, so even if Mastodon 4.3-beta1 would be released tomorrow, it wouldn't take long to incorporate it into this branch :3)

And for those who are interested about the current state: I've edited the todo list in the first post, it's getting smaller and smaller! :nerd_face:

nachtjasmin commented 6 months ago

It's ready! :partying_face:

I would say that it's ready. I've deployed a test instance with the changes (limited federation, only federating with queer.group right now) and tested several of Hometown's features:

Although I have tested it with a database dump of an existing installation, I am going to do this again (tomorrow). Just to be sure that exclusive lists are migrated properly. But if that's the case, I think it's safe to merge this.

Importing the complete database dump of queer.group and running RAILS_ENV=production bundle exec rails db:migrate works like a charm. My exclusive lists were migrated properly, everything seems to work as intended.

dariusk commented 6 months ago

Hello! I am running this in a local dev environment and it seems to work pretty well! Thank you again for the help.

I wanted to confirm: the custom Hometown mobile view no longer works in this branch, is that correct? It's the view that in base Hometown looks like this:

image

Note the hamburger menu at the top right that you can press to retract the menu bar, plus the lack of a big banner and user avatar in a bar at the top. What I see locally running this PR is this:

image

Is that expected? AKA do I have to re-implement the custom layout?

(I've also noticed some CSS bugs in the pastel themes that I'll need to sort out too.)

nachtjasmin commented 6 months ago

@dariusk Yes, I removed the custom Hometown-layout stuff completely to make it easier to deal with all the merges of Mastodon 4.2 especially. Maybe, the custom sidebar can be even done with some CSS-only hacks. At least with Tangerine, the menu can be moved to the bottom now which looks quite nice as well.

And I haven't updated the pastel themes yet (actually, I totally forgot about them) and with the upcoming changes of Mastodon 4.3 (new icons, probably just a bunch of new UI changes as well), I don't feel like custom themes can be maintained easily, especially if it's "just" the color scheme. I heard about other people who want their own color schemes and stuff and the situation right now is painful.

I had and still have the plan to create a PR for upstream mastodon that just migrates the SASS stuff to CSS variables (or at least merges all colors into one file), so that custom color schemes can be done quickly by overriding some variables.

dariusk commented 6 months ago

Great - I have already fixed the custom themes so we are good there. Tomorrow I'm going to work on retractable side bar stuff and look at the mobile layout in general. I will ship this on friend camp when that work is done, and if it looks good there, having it working on your server and mine makes me feel very good about shipping this. Again, thank you for all your hard work on this and I will keep you updated in this thread on my progress.

mnvr commented 5 months ago

I had and still have the plan to create a PR for upstream mastodon that just migrates the SASS stuff to CSS variables (or at least merges all colors into one file), so that custom color schemes can be done quickly by overriding some variables.

That'll be great indeed! I had tried going on that route, eventually I gave up because of the complexity in keeping that in sync with Mastodon. What's funny is their SASS is almost as big as the CSS it produces, so all that indirection is not even buying much of anything.

I recently tried creating a theme generator, and my finding was that to get a basic theme, we just need 3 core CSS variables:

And two more optional, possibly derived, ones:

Rest of the colors can be derived from it (either at theme creation time, say using the colorjs.io library, or letting the browser do it using CSS color-mix).

The resulting theme won't be spectacular, but it'll be useable. What I had hopes from this approach was that it'll allow me to create a theme that automatically adapts to light/dark mode, for folks like myself whose computers switch between light/dark mode at morning/evening.

However, eventually I gave up - the approach I was trying is too hacky as things stand. If however, Mastodon were to instead start using CSS variables, such themeing would become much easier (to do, and to maintain).

tl;dr; +1!

pronoiac commented 2 months ago

Would you be up for looking at https://github.com/mastodon/mastodon/pull/28488 , re: HEIC/HEIF images?

nachtjasmin commented 2 months ago

@pronoiac Just cherry-picked it into the branch and deployed it to our instance, it's working fine. So, in case you're running your instance of this branch, the usual git pull && bundle install should do the trick here. ^^

pronoiac commented 2 months ago

@nachtjasmin oops, lost a draft comment when I went looking for the other WIP PR...

I'm on managed hosting, so it's currently out of my hands.

Not to be uselessly helpful, or pry, but can I help? Running a staging / test server? Donations? Moral support?

ghost commented 1 month ago

@dariusk do you know if you will have some time this summer to merge this and do a proper release? If no, does something else than what have been done by @nachtjasmin (thank you so much!) can be done to help?

dariusk commented 1 month ago

I will have time next week, actually!

leitmedium commented 1 week ago

Are there any current plans or foreseeable updates to the Mastodon version that Hometown is based on?

nachtjasmin commented 1 week ago

Or, to reframe the question: what are the blockers of merging this PR? And can we assist you somehow, @dariusk?

Cause yes, this PR lost some of the features of Hometown, e.g. the sidebar changes on mobile and it has broken pastel themes. To me those are only mild inconveniences, but I understand that they might have a higher priority to you as it is your project.

but that’s just a guess, so if there are other things to tackle, feel free to tell me! :3

dariusk commented 1 week ago

Hi @nachtjasmin -- literally sitting down tonight to actually do this. Life has been a bit of a whirlwind and unfortunately keeping Hometown up to date took a back burner. FWIW I should probably give you merge/release rights but I suppose that's a conversation for another venue/time. Right now I want to get us up to date and ready for the security patch by getting us up to speed with v4.2. I'll have more comments here over the next few hours while I do the work of incorporating your patch.

dariusk commented 1 week ago

Okay @nachtjasmin, I believe I have solved the rebase issues in the branch https://github.com/hometown-fork/hometown/tree/hometown-4.2-merge and I managed to upgrade a local hometown-4.0.15+1.1.1 install to 4.2.9 and the basic Hometown features do still work, and the exclusive lists migration works great, thank you for that. hometown-dev is not up to date but that branch is enough to apply security patches to on Thursday and then do a release. It's very late here so I am going to call it a night but in the morning I am going to attempt to deploy the 4.2 merge to Friend Camp to make extra sure it works.

ClearlyClaire commented 1 week ago

Chiming here to notify you that we updated the stable-4.2 branch with a few more backported fixes. Those should probably be merged before the release of the security fixes, but they seem to apply cleanly on your hometown-4.2-merge branch, and I don't think they should affect Hometown features.

Same thing with the security patches, I haven't checked whether they interfere with Hometown features, but I don't think they should, and they also apply cleanly, so merging them should not be an issue, and you should be on track for the release!

nachtjasmin commented 1 week ago

@ClearlyClaire Thank you for letting us know! I've merged the stable-4.2 branch into this again, went smooth as butter. :+1:

nachtjasmin commented 1 week ago

I just realised that I accidentally leaked stuff from my personal fork into this branch. I don't know when it happened. Anyway, I removed said commits, sorry for any confusion caused by this.

dariusk commented 1 week ago

Oops! Can you give an example personal commit for me to look out for just to make extra sure it's not in the final release or my test builds?

nachtjasmin commented 1 week ago

af8b8ee would be one such example, I think it's in your branch as well. :see_no_evil:

dariusk commented 1 week ago

Okay I did a force update on https://github.com/hometown-fork/hometown/tree/hometown-4.2-merge -- going to attempt to fix the pastel themes since those are the main thing broken.

dariusk commented 1 week ago

@nachtjasmin Interesting -- it seems like no user themes (pastel or not) are saving state for me. Like I go to change my user set theme and no matter what I change it to it always reverts to Mastodon (Dark). But if I change the site theme that does change and save it for everyone. Can you reproduce this? Or is this just a problem for me...

dariusk commented 1 week ago

Hmm looks like a rollback is happening here when it tries to save the setting...

09:20:36 web.1     |   Setting Load (0.3ms)  SELECT "settings".* FROM "settings" WHERE (thing_type is NULL and thing_id is NULL) AND "settings"."var" = $1 ORDER BY "settings"."id" ASC LIMIT $2  [["var", "theme"], ["LIMIT", 1]]
09:20:36 web.1     |   ↳ app/models/setting.rb:28:in `block in []'
09:20:36 web.1     | Cache write: rails_settings_cached/8d13a7825611181d5a70a3c9add28ef1/theme ({:compress=>true, :compress_threshold=>1024})
09:20:36 web.1     | Cache read: rails_settings_cached/8d13a7825611181d5a70a3c9add28ef1/theme ({:compress=>true, :compress_threshold=>1024})
09:20:36 web.1     | Cache fetch_hit: rails_settings_cached/8d13a7825611181d5a70a3c9add28ef1/theme ({:compress=>true, :compress_threshold=>1024})
09:20:36 web.1     |   UserRole Load (0.1ms)  SELECT "user_roles".* FROM "user_roles" WHERE "user_roles"."id" = $1 LIMIT $2  [["id", 3], ["LIMIT", 1]]
09:20:36 web.1     |   ↳ app/models/user.rb:159:in `role'
09:20:36 web.1     |   TRANSACTION (0.1ms)  ROLLBACK
09:20:36 web.1     |   ↳ app/controllers/settings/preferences/base_controller.rb:7:in `update'
09:20:36 web.1     |   Rendering layout layouts/admin.html.haml

~Am I reading this right... it thinks it hit the cache so it doesn't need to change anything?~ no, cache is fine, this is what it looks like on vanilla 4.2.9:

10:10:41 web.1     |   Setting Load (0.2ms)  SELECT "settings".* FROM "settings" WHERE (thing_type is NULL and thing_id is NULL) AND "settings"."var" = $1 ORDER BY "settings"."id" ASC LIMIT $2  [["var", "theme"], ["LIMIT", 1]]
10:10:41 web.1     |   ↳ app/models/setting.rb:28:in `block in []'
10:10:41 web.1     | Cache write: rails_settings_cached/e23eceea1dbbecf09d779eb1d9007c03/theme ({:compress=>true, :compress_threshold=>1024})
10:10:41 web.1     | Cache read: rails_settings_cached/e23eceea1dbbecf09d779eb1d9007c03/theme ({:compress=>true, :compress_threshold=>1024})
10:10:41 web.1     | Cache fetch_hit: rails_settings_cached/e23eceea1dbbecf09d779eb1d9007c03/theme ({:compress=>true, :compress_threshold=>1024})
10:10:41 web.1     |   UserRole Load (0.1ms)  SELECT "user_roles".* FROM "user_roles" WHERE "user_roles"."id" = $1 LIMIT $2  [["id", 3], ["LIMIT", 1]]
10:10:41 web.1     |   ↳ app/models/user.rb:159:in `role'
10:10:41 web.1     |   User Update (0.2ms)  UPDATE "users" SET "updated_at" = $1, "settings" = $2 WHERE "users"."id" = $3  [["updated_at", "2024-07-03 17:10:41.849632"], ["settings", "{\"theme\":\"default\",\"web.advanced_layout\":false,\"web.trends\":true,\"web.use_blurhash\":true,\"web.use_pending_items\":false,\"web.use_system_font\":false,\"web.disable_swiping\":false,\"web.delete_modal\":true,\"web.reblog_modal\":false,\"web.unfollow_modal\":true,\"web.reduce_motion\":false,\"web.expand_content_warnings\":false,\"web.display_media\":\"default\",\"web.auto_play\":false}"], ["id", 1]]
10:10:41 web.1     |   ↳ app/controllers/settings/preferences/base_controller.rb:7:in `update'
10:10:41 web.1     |   TRANSACTION (0.5ms)  COMMIT

See how it goes straight from the UserRole Load into User Update without a rollback...

dariusk commented 1 week ago

Oh I think I am starting to see... Mastodon introduced UserRole since v4.0. There is probably a missing migration or something where users lack an appropriate UserRole so the validation doesn't happen and triggers a rollback?

dariusk commented 1 week ago

So I have confirmed that this error only occurs when a server has been migrated from Hometown running v4.0.15 to the new v4.2.9 merge branch. If you install it clean it's fine, but something about the migration is an issue. After migration there is indeed a user_roles table, and it has the correct data, and the account being queried in the UserRole check even has the correct role_id so I am not sure what the problem is here...

dariusk commented 1 week ago

Lol okay this is a problem with email addresses on the default admin account!! In 4.0.15 it was admin@localhost:3000 and now it's like "that is not a valid email address". Fixing this now...

dariusk commented 1 week ago

Okay I changed my admin email to admin@localhost and now everything is saving just fine, siiiiigh. @ClearlyClaire, this branch now includes stuff from stable-4.2 so we are ready to apply the patch tomorrow. Since I haven't actually done a 4.2 release yet, I am just going to advise everyone to update to 4.2 tomorrow and do the release all at once.

pronoiac commented 1 week ago

It's been 16 months since Mastodon 4.1.0 dropped, and over 2k commits. Releasing today, maybe with a note about tomorrow's expected security release, could help surface any issues sooner. It could give the hosted services some room to work and do some indexing, too.

(Apologies if that's just my anxiety.)

dariusk commented 1 week ago

@pronoiac Hmm I see your point but I do not have physical availability to monitor a release until tomorrow morning anyway. I think it's more important that I'm around to help put out fires than to do a prerelease and disappear until a second actual release tomorrow morning. But if anyone wants to try out the prerelease, they can run this branch:

https://github.com/hometown-fork/hometown/tree/hometown-4.2-merge

It's currently running on Friend Camp just fine. Here is my draft installation guide for anyone who wants to try it:

Hometown v4.2.9+1.1.1 release (not a real release, I'm waiting for the security patch tomorrow)

This is a security release that keeps us up to date with Mastodon v4.2.9. Thank you once again to [@nachtjasmin](https://github.com/nachtjasmin) for the help with this merge.

Changelog

Please [see the Mastodon 4.2.0 release notes for details](https://github.com/mastodon/mastodon/releases/tag/v4.2.0). You can check the release notes up to 4.2.9 as well, though they are mostly bugfixes minus a big moderation change that I highlight below.

The really big changes for users are:

Upgrade steps

These are instructions for upgrading from Hometown v4.0.15+1.1.1.

As always, make sure you have backups of the database before performing any upgrades. A postgres backup command would look something like this: 

pg_dump -Fc -U postgres mastodon_production > name_of_the_backup.dump

Dependencies

External dependencies have changed since v4.1.7, with the Ruby, PostgreSQL and Node.js minimum version being higher.

Tip

If your uploaded images are broken after the upgrade, it means your installed ImageMagick version is older than the new minimum version (6.9.7-7), for example if you are running Ubuntu 18.04. If this happens, you can find more information and ways to fix it [on this page](https://github.com/mastodon/mastodon/issues/25776).

Moderation and registration changes

Important

This update changes registrations to be closed by default.

Running a social media platform where anyone can sign up without active moderation is dangerous.

We are changing the default, so that opening registrations is always a conscious choice. If you have never changed or saved the registrations mode yourself, this update will switch your server to not accepting new users. Simply change the setting again after the update if you wish to restore the old behavior.

Database replica configuration

The way Mastodon handles read replicas has changed, removing the makara gem and using native Rails support instead.

This changes how database replicas are configured. Instead of editing config/database.yml, you should use an unmodified one and use the REPLICA_DB_NAME, along with REPLICA_DB_USER, REPLICA_DB_PASS, REPLICA_DB_HOST and REPLICA_DB_PORT, if they differ from the primary database.

If you are using DATABASE_URL, you can configure your read replica in a similar way using REPLICA_DATABASE_URL.

If you aren't using database replicas or have no idea what this is about... don't sweat it, you're fine.

Streaming server changes

We have dropped built-in clustering support from the streaming server, which means, depending on the load you are facing, that you may need to run multiple instances of it and configure a load-balancer.

Unless you are using Docker, it is recommended that you update your mastodon-streaming unit scripts with the ones we provide:

  1. sudo cp ~mastodon/live/dist/mastodon-streaming*.service /etc/systemd/system/
  2. sudo systemctl daemon-reload
  3. sudo systemctl restart mastodon-streaming

If you then need to run more than one mastodon-streaming server, you can:

  1. Start a new instance with sudo systemctl start mastodon-streaming@port (e.g. mastodon-streaming@4001)
  2. Edit your nginx configuration file to add the new server to the load-balancing (an example is provided in the comments in dist/nginx.conf)

Automatic update checking

Starting from this release, Mastodon will periodically check for updates by querying https://api.joinmastodon.org/update-check every 30 minutes in a background job.

That URL can be changed using the UPDATE_CHECK_URL environment variable, and the feature outright disabled by setting that variable to an empty string (UPDATE_CHECK_URL=).

Update steps

  1. If you are using rbenv, [update the list of available versions](https://github.com/rbenv/ruby-build/wiki#updating-ruby-build) and install Ruby 3.2.3 by doing RUBY_CONFIGURE_OPTS=--with-jemalloc rbenv install in the Mastodon install directory (e.g. /home/mastodon/live)
  2. Ensure that you are using Node.js v16 or newer -- I (Darius) actually encountered errors on 16 and needed to upgrade to v18 to make everything work.
  3. Install dependencies: bundle install and yarn install --frozen-lockfile
  4. Precompile the assets: RAILS_ENV=production bundle exec rails assets:precompile
  5. Run the pre-deployment database migrations by specifying the SKIP_POST_DEPLOYMENT_MIGRATIONS=true environment variable: SKIP_POST_DEPLOYMENT_MIGRATIONS=true RAILS_ENV=production bundle exec rails db:migrate
  6. Restart all Mastodon processes
  7. Run the post-deployment database migrations: RAILS_ENV=production bundle exec rails db:migrate
  8. If you use Elasticsearch, rebuild the search indexes with RAILS_ENV=production bin/tootctl search deploy --reset-chewy
graue commented 1 week ago

Starting from this release, Mastodon will periodically check for updates by querying https://api.joinmastodon.org/update-check

Does that URL also know about Hometown releases? or should this be changed to account for running a fork?

pronoiac commented 1 week ago

What that api returns right now, prettified:

{
  "updatesAvailable": [
    {
      "version": "4.2.9",
      "type": "patch",
      "releaseNotes": "https://github.com/mastodon/mastodon/releases/tag/v4.2.9",
      "urgent": false
    }
  ]
}
dariusk commented 1 week ago

It does not - I can probably provide a new URL for Hometown releases but it wasn't priority for this security push.

On Wed, Jul 3, 2024, 6:57 PM James Young @.***> wrote:

What that api returns right now, prettified:

{ "updatesAvailable": [ { "version": "4.2.9", "type": "patch", "releaseNotes": "https://github.com/mastodon/mastodon/releases/tag/v4.2.9", "urgent": false } ] }

— Reply to this email directly, view it on GitHub https://github.com/hometown-fork/hometown/pull/1337#issuecomment-2207862636, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACBBVQKY5BFOTAME7EWOQLZKSTZTAVCNFSM6AAAAAA7O5HSMWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBXHA3DENRTGY . You are receiving this because you were mentioned.Message ID: @.***>

reesericci commented 3 days ago

Is there a clean path to migrate from this to v4.2.10? Just attempted it this morning and ran into a whole host of issues myself

pronoiac commented 3 days ago

@reesericci I suggest filing a new issue, and including details of the issues there

reesericci commented 3 days ago

I more just ended up hurting myself with git & bundler than anything specific (updated the dependencies too far, etc)

nachtjasmin commented 3 days ago

clean path to migrate from this

The recent Hometown release already includes the patch, so you should be fine following the instructions there.

This PR is also part of said release.

reesericci commented 2 days ago

That worked for me!