search

orhun / git-cliff

A highly customizable Changelog Generator that follows Conventional Commit specifications ⛰️
https://git-cliff.org
Apache License 2.0
9.02k stars 185 forks source link

ARM64 docker images take 6h+ to build #879

Open M0NsTeRRR opened 1 week ago

M0NsTeRRR commented 1 week ago

Is there an existing issue for this?

Description of the bug

https://github.com/orhun/git-cliff/actions/runs/10978847295/job/30482488830

The github action failed to build and push 2.6.0 tag to ghcr.io

The Docker job times out after 360 minutes, as described in the GitHub documentation

Steps To Reproduce

.

Expected behavior

.

Screenshots / Logs

No response

Software information

Github action

Additional context

No response

orhun commented 1 week ago

Thanks for reporting!

Yup, I tried running the workflow 2 more times, they have all failed: https://github.com/orhun/git-cliff/actions/runs/10978847295

I think it is timing out due to QEMU/AMD builds :/

According to this docs, it is not possible to increase this limit.

Job execution time - Each job in a workflow can run for up to 6 hours of execution time. If a job reaches this limit, the job is terminated and fails to complete.

Not sure what can I do expect disabling them temporarily, which I did in cde2a8e3222f5e8f8bdd9ae841fd0e5c42f68846

M0NsTeRRR commented 6 days ago

@orhun Looks like an issue with github action, the docker action is running now (4 completed) ? (could you rerun the 2.6.0 docker action)

orhun commented 5 days ago

Yup, I need to create a new release soon. I don't think it is possible to trigger the pipeline again with the new changes in the workflow file :)

orhun commented 5 days ago

In the meantime you can probably use one of the latest versions.

M0NsTeRRR commented 5 days ago

Yup, I close this one as the issue seems resolved :)

orhun commented 5 days ago

I refactored the Docker builds in #888 based on these docs - it should be faster in theory but I feel like there are still some rough edges in the workflow.

orhun commented 4 days ago

It takes 6hr+ even with the build matrix: https://github.com/orhun/git-cliff/actions/runs/11053756749/job/30708900152 💀

LtdSauce commented 3 days ago

Hey!

As i have the impression i caused some trouble with my comment (that i deleted... which might have been a mistake), i thought i share my rationale why i removed that comment.

When i saw the change removing the arm64 image build, i thought that it was due to the fact, that building the images and publishing them was all done on the same node. But then i found the setup-buildx-action docs, which mention that this is already creating one node for each architecture. Which is basically what this suggests as well. Thus, i deleted my comment in the hope that i had not done any harm :sweat_smile: Sorry 'bout that!

Also i guess in the current state this is not yet achievable as github has not yet released its official arm64 support as of this comment. As soon as that is available for OSS this could be used in the open PR to use a native arm64 node to build the arm64 image. This suggests this will not be within this year. (In fact this was the second reason i deleted my comment... next time i will just update it!)

M0NsTeRRR commented 3 days ago

From the log I'm reading, it looks like there is an issue with shellexpand, right? It takes one hour from the start to reach shellexpand, and the compilation takes at least 5 hours (it looks really long from my POV) ? https://github.com/orhun/git-cliff/actions/runs/11053756749/job/30708900152#step:10:1072

Maybe we could add more verbose to cargo build to see if we have something useful ?

But maybe as @LtdSauce pointed out it's just the QEMU build that is really slow as expected :/

For now as a quick fix we could disable the ARM build until we find a solution.

M0NsTeRRR commented 3 days ago

Maybe docker buildx on macos arm runner (for arm build) are better ?

https://github.com/actions/runner-images

LtdSauce commented 3 days ago

Just to share some insights: on my DAYJOB we recently had to switch from qemu emulation for building arm64 images as well. In our case we were only doing apt install commands, but that already increased the build time from 10 minutes for our amd64 builds to 60 minutes for our qemu emulated arm64 builds.

orhun commented 3 days ago

As i have the impression i caused some trouble with my comment (that i deleted... which might have been a mistake), i thought i share my rationale why i removed that comment.

No worries!

Also i guess in the current state this is not yet achievable as github has not yet released its official arm64 support as of this comment.

Ah, thanks for sharing the links. I subscribed to those issues and I hope it happens one day 🙏🏼

it looks like there is an issue with shellexpand, right?

I think the issue is not about a specific crate, it is related to QEMU emulated builds.

Maybe we could add more verbose to cargo build to see if we have something useful ? Maybe docker buildx on macos arm runner (for arm build) are better ?

I tried your suggestion in #888 but I guess setup-qemu-action is not available for the macOS runner: https://github.com/orhun/git-cliff/actions/runs/11072292415/job/30766175378?pr=888

M0NsTeRRR commented 3 days ago

I think the issue is not about a specific crate, it is related to QEMU emulated builds.

I just find it strange at first glance that such package takes more than 5 hours to build compared to others.

I tried your suggestion in #888 but I guess setup-qemu-action is not available for the macOS runner: https://github.com/orhun/git-cliff/actions/runs/11072292415/job/30766175378?pr=888

Sad, yes I saw the same error in setup-qemu-action repository discussion.

orhun commented 3 days ago

Can you send a link of that discussion?

Man... all I wanted was to build arm64 images...

LtdSauce commented 3 days ago

I thought qemu is not needed when using the arm64 MacOs runner? The architecture is already native and afaik you should be able to do docker builds on MacOs that build a Linux image already?

M0NsTeRRR commented 3 days ago

Can you send a link of that discussion?

Man... all I wanted was to build arm64 images...

https://github.com/docker/setup-qemu-action/discussions/137 https://github.com/docker/setup-qemu-action/issues/24

M0NsTeRRR commented 3 days ago

I thought qemu is not needed when using the arm64 MacOs runner? The architecture is already native and afaik you should be able to do docker builds on MacOs that build a Linux image already?

Yes with two differents job instead of a matrix 👍

orhun commented 3 days ago

I thought qemu is not needed when using the arm64 MacOs runner? The architecture is already native and afaik you should be able to do docker builds on MacOs that build a Linux image already?

Ah damn, right...

LtdSauce commented 3 days ago

FTR: This seems to be the way it is until GitHub launches their linux/arm64 runners.

There seems to be no way for macos runners on arm64 to get docker to build.