search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.89k stars 1.77k forks source link

Support ARM64 #1585

Open androidacy-user opened 2 years ago

androidacy-user commented 2 years ago

Self-Hosted Version

n/a

CPU Architecture

arm64

Docker Version

20.10.12

Docker Compose Version

1.29.2

Steps to Reproduce

  1. Try to run ./install.sh on a relatively fresh install of ubuntu 22.04
  2. Install fails

Expected Result

Install succeeds

Actual Result

Hits an error:

Step 4/5 : RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then     /usr/src/sentry/enhance-image.sh; fi
 ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
 ---> Running in 2ae646b00261
exec /bin/sh: exec format error

Note we don't have an enhance-image.sh anywhere in the directory

peterdk commented 2 months ago

I understand that. It would be nice, but I don't see how we as (I assume?!) non paying users can 'demand' this. I am happy with Sentry, I use it for selfhosted in China, since data can't leave the country there. Would love to see ARM support, since that makes the hosting cheaper on AWS. But yeah, can't blame you.

I could benefit Sentry itself for cheaper hosting of their own infrastructure in the future, but that I have no insight in.

johnsturgeon commented 2 months ago

I know this is not what you all want to hear, but this is not something we are prioritizing at the moment.

That is exactly what I want to hear. Truth. If it's inconvenient.

Thank you!

halettedurand commented 2 months ago

We switched to Bugsink; much simpler than Sentry, but that also means much less heavy (we run it on a single 2GB instance!)

androidacy-user commented 2 months ago

We switched to Bugsink; much simpler than Sentry, but that also means much less heavy (we run it on a single 2GB instance!)

Looks like that's in private beta

We were using countly on an x86 server previously which includes error tracking and such but they don't support ARM64 either

vanschelven commented 1 month ago

We switched to Bugsink; much simpler than Sentry, but that also means much less heavy (we run it on a single 2GB instance!)

Looks like that's in private beta

Not anymore; and we even have a published ARM Docker image ready to go

Yes we are but [1] it just means "rough around the edges, looking for early adopters, one click away and [2] we won't be in that state for much longer.

ezhevita commented 1 month ago

There was Sentry-ARM, but it appears to be archived, very odd.

@crmacca I've archived it because there was a pull request that supposedly introduced arm64 support (https://github.com/getsentry/self-hosted/pull/1538), I guess it actually didn't?... Also I no longer have a free arm64 server to host it on so there was no personal need to support it.

I may revisit it in the near future considering that users still need a working option.

vanschelven commented 1 month ago

a pull request that supposedly introduced arm64 support [..] actually didn't?...

If I understand correctly, it did in fact introduce arm64 support, but only for the single "sentry" image (and not the 5 to 10 or so images that it depends on). I swear there's a funny punchline here but I'm not that funny.

ezhevita commented 1 month ago

I've updated my fork to the latest version (24.9.0) however I don't have a device to test it on, so feel free to report any issues: https://github.com/Sentry-ARM/self-hosted/tree/arm64

By the way, Sentry is much more arm64-ready now than it was 3 years ago, couple of Dockerfile tweaks and everything compiles properly, so great job on doing all of this progress!

Shaggy84675 commented 1 month ago

Probably a dumb question, but am I missing something? It wants an authentication for docker images:

▶ Fetching and updating Docker images ...
Error response from daemon: Head "https://ghcr.io/v2/sentry-arm/sentry/manifests/24.9.0": unauthorized

▶ Building and tagging Docker images ...

#0 building with "default" instance using docker driver

#1 [web internal] load build definition from Dockerfile
#1 transferring dockerfile: 463B done
#1 WARN: InvalidDefaultArgInFrom: Default value for ARG ${SENTRY_IMAGE} results in empty or invalid base image name (line 2)
#1 DONE 0.0s

#2 [web internal] load metadata for ghcr.io/sentry-arm/sentry:24.9.0
#2 ERROR: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Asentry-arm%2Fsentry%3Apull&service=ghcr.io: 401 Unauthorized
------
 > [web internal] load metadata for ghcr.io/sentry-arm/sentry:24.9.0:
------
failed to solve: ghcr.io/sentry-arm/sentry:24.9.0: failed to resolve source metadata for ghcr.io/sentry-arm/sentry:24.9.0: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Asentry-arm%2Fsentry%3Apull&service=ghcr.io: 401 Unauthorized
Error in install/build-docker-images.sh:6.
'$dcb --force-rm web' exited with status 17
-> ./install.sh:main:34
--> install/build-docker-images.sh:source:6

Cleaning up...
ezhevita commented 1 month ago

Probably a dumb question, but am I missing something? It wants an authentication for docker images:

@Shaggy84675 should be fixed now, Github sets private visibility for packages by default and I was way too tired to notice and fix that for the last couple of images

Shaggy84675 commented 1 month ago

Probably a dumb question, but am I missing something? It wants an authentication for docker images:

@Shaggy84675 should be fixed now, Github sets private visibility for packages by default and I was way too tired to notice and fix that for the last couple of images

@ezhevita Thanks! Got further, probably need to enable snuba package as well :)

▶ Bootstrapping and migrating Snuba ...
 Network sentry-self-hosted_default  Creating
 Network sentry-self-hosted_default  Created
 Container sentry-self-hosted-kafka-1  Creating
 Container sentry-self-hosted-clickhouse-1  Creating
 Container sentry-self-hosted-redis-1  Creating
 Container sentry-self-hosted-clickhouse-1  Created
 Container sentry-self-hosted-redis-1  Created
 Container sentry-self-hosted-kafka-1  Created
 Container sentry-self-hosted-redis-1  Starting
 Container sentry-self-hosted-kafka-1  Starting
 Container sentry-self-hosted-clickhouse-1  Starting
 Container sentry-self-hosted-clickhouse-1  Started
 Container sentry-self-hosted-kafka-1  Started
 Container sentry-self-hosted-redis-1  Started
 snuba-api Pulling
 snuba-api Error Head "https://ghcr.io/v2/sentry-arm/snuba/manifests/24.9.0": denied
Error response from daemon: Head "https://ghcr.io/v2/sentry-arm/snuba/manifests/24.9.0": denied
Error in install/bootstrap-snuba.sh:3.
'$dcr snuba-api bootstrap --no-migrate --force' exited with status 18
-> ./install.sh:main:35
--> install/bootstrap-snuba.sh:source:3

Cleaning up...
ezhevita commented 1 month ago

@ezhevita Thanks! Got further, probably need to enable snuba package as well :)

@Shaggy84675 I've somehow completely forgotten to build and push snuba image, should be available now also tested install.sh - it has executed without any errors, so no further installation errors expected

Shaggy84675 commented 1 month ago

@ezhevita Can confirm, everything passed so it should be good now. Thanks!

aldy505 commented 1 month ago

I've updated my fork to the latest version (24.9.0) however I don't have a device to test it on, so feel free to report any issues: https://github.com/Sentry-ARM/self-hosted/tree/arm64

By the way, Sentry is much more arm64-ready now than it was 3 years ago, couple of Dockerfile tweaks and everything compiles properly, so great job on doing all of this progress!

@ezhevita Very nice! I'm pinging the team on Discord to ask whether we want to have it on upstream. so people won't need to mess their head around this anymore.

aldy505 commented 1 month ago

Since it's been a week and the devinfra team hasn't get back to me, and since not everyone in here uses Discord, these are some of the things we need to do in order to get this issue closed 😆:

TLDR version:

  1. Modify the Dockerfile in each of the repo (sentry, snuba, vroom, relay, symbolicator) to accept arch platform arguments.
  2. Modify the build-publish mechanism on each repo to also build and publish arm64 to DockerHub.
  3. Modify craft to also copy every amd64 and arm64 (if any) manifest to DockerHub for monthly self-hosted release.

Long version:

So on every push to master, every repository would be publishing an artifact (per commit release based) to DockerHub.

They actually have a gocd pipeline that deploy this into GCR, but since I assume for production usage they don't use the image from DockerHub at all, we should be fine as long we don't touch the gocd pipeline script. One example on relay: https://github.com/getsentry/relay/blob/2f2e7d06b5eff9af17adee1b49368c3a5d88a32e/gocd/pipelines/relay.yaml#L35-L53

What we need to do is to modify those publish to DockerHub script.

  1. If the script is pulling directly from GCR and re-push it to DockerHub, we need to build it on the Github Actions, using docker buildx ... --platform linux/amd64,linux/arm64 (see https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/), but we also need to experiment on choosing the right Github Actions runner for this.
  2. If the script is pushing directly to DockerHub, we only need to ensure it also builds an arm64 image.

Since every repository is a public one, it should not cost any money to run the Github Actions runner, therefore we're not going to cost Sentry (the company) anything.

That only resolves publishing the multi-arch image to DockerHub, we'll also need to modify this: https://github.com/getsentry/craft/blob/a83b19a2667b6a8144780aa806695f8b95a3b3ef/src/targets/docker.ts#L104-L109. TLDR, it executes some kind of docker buildx imagetools create --tag <GIT SHA FROM NIGHTLY> 24.9.0 (24.9.0 refers to the regular self-hosted tag).

The bad news is that using buildx imagetools can only be done for a single architecture image, see here: https://github.com/docker/buildx/issues/1744#issuecomment-1513579190. But there's a workaround at the comments too, so we'll still need to modify craft a bit.

One thing to note on modifying craft, is that Sentry also have a bunch of other images being published to DockerHub, like getsentry/chartcuterie, getsentry/sentry-cli and even getsentry/craft itself. We need to carefully make sure that we're publishing the correct arch manifest. (If the bad news with buildx imagetools previously still causing an issue).

Well then of course, after understanding the build-publish process, the first thing that we need to do is to modify the Dockerfile in each reposiotory (vroom, sentry, snuba, symbolicator, relay) to be multi-arch friendly, with amd64 as default for now. As done previously by @ezhevita on their Sentry-ARM organization.

Even if we're finally able to put this thing working, I totally assume that moving forward, Sentry (the company) wouldn't want to maintain the arm64 images, since they're not using it on their production environment. So the best we can do is to maintain this as a community effort.

abriginets commented 1 month ago

Sentry (the company) wouldn't want to maintain the arm64 images

Does it all even make sense then? Keeping the fork synced sounds like less effort and interference for everyone.

P.S. still not sure why Sentry the company does not prioritize ARM64 support considering how drastically it would cut their hardware costs ¯\_(ツ)_/¯

maximal commented 4 weeks ago

Wow! @aldy505, that’s quite a work you did summarizing all of that. Thanks!

aldy505 commented 3 weeks ago

Finnaly heard back from the team, forwarding the raw message here:

Update: Apparently our team has previously built out our service images for arm, but we disabled it since the build times were very long. Let's revisit this in the future when this is available for us to use: https://github.blog/news-insights/product-news/arm64-on-github-actions-powering-faster-more-efficient-build-systems/

(end of the raw message, below this line is my statement)

It makes sense though to disable it since Sentry always build and push every commit as a Docker image. From the GitHub blog, they said the ARM runner will be available for open source project by the end of the year, in which it's 2-3 months from now. It should be okay for us to wait for this, I guess.

benedikt-bartscher commented 1 day ago

It seems like arm runners are now available: https://github.blog/changelog/2024-09-03-github-actions-arm64-linux-and-windows-runners-are-now-generally-available/ 🚀

aldy505 commented 1 day ago

It seems like arm runners are now available: https://github.blog/changelog/2024-09-03-github-actions-arm64-linux-and-windows-runners-are-now-generally-available/ 🚀

@benedikt-bartscher Not yet. Quoting:

Arm64 Linux and Windows GitHub-hosted runners for Actions are now generally available. [...] Arm64 runners are available to customers on our Team and Enterprise Cloud plans.

Sentry is not on the Team and Enterprise Cloud plans. We're waiting for it to be available for public repositories on the free plan.

benedikt-bartscher commented 21 hours ago

Thank you for clarifying @aldy505