search

metal3-io / ironic-image

Container image to run OpenStack Ironic as part of Metal³
Apache License 2.0
56 stars 116 forks source link

🌱 increase ironic-image build timeout to 3600s #519

Closed tuminoid closed 3 months ago

tuminoid commented 3 months ago

Ironic-image build locally, with a fast disk, fast CPU, and fast net takes 500-600s, depending on pre-pulls, repository speeds etc. New cloud has none of those reliably, so it times out at 1000s far too often. Also scheduling time is considered when images are built in Jenkins, and often scheduling takes a long time. Thanks to #505 we get notifications of these failures now.

I will cherry-pick this to release branches as well.

tuminoid commented 3 months ago

/override metal3-centos-e2e-integration-test-main metal3-ubuntu-e2e-integration-test-main GH action.

metal3-io-bot commented 3 months ago

@tuminoid: Overrode contexts on behalf of tuminoid: metal3-centos-e2e-integration-test-main, metal3-ubuntu-e2e-integration-test-main

In response to [this](https://github.com/metal3-io/ironic-image/pull/519#issuecomment-2159934399): >/override metal3-centos-e2e-integration-test-main metal3-ubuntu-e2e-integration-test-main >GH action. Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
tuminoid commented 3 months ago

/cc @kashifest @Rozzii

tuminoid commented 3 months ago

/cherry-pick release-24.1

metal3-io-bot commented 3 months ago

@tuminoid: once the present PR merges, I will cherry-pick it on top of release-24.1 in a new PR and assign it to you.

In response to [this](https://github.com/metal3-io/ironic-image/pull/519#issuecomment-2159935163): >/cherry-pick release-24.1 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
tuminoid commented 3 months ago

/cherry-pick release-24.0

metal3-io-bot commented 3 months ago

@tuminoid: once the present PR merges, I will cherry-pick it on top of release-24.0 in a new PR and assign it to you.

In response to [this](https://github.com/metal3-io/ironic-image/pull/519#issuecomment-2159935281): >/cherry-pick release-24.0 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
tuminoid commented 3 months ago

/cherry-pick release-23.1

metal3-io-bot commented 3 months ago

@tuminoid: once the present PR merges, I will cherry-pick it on top of release-23.1 in a new PR and assign it to you.

In response to [this](https://github.com/metal3-io/ironic-image/pull/519#issuecomment-2159935410): >/cherry-pick release-23.1 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
tuminoid commented 3 months ago

/cc @NymanRobin

Rozzii commented 3 months ago

/lgtm

Rozzii commented 3 months ago

The change is good. However, please note that there have been some issues with VM scheduling for the jobs. For instance, this job took 15 hours when scheduling time is included: Build #62.

Maybe these jobs could be moved from Jenkins to github action runners in the future to improve reliability

WOW that is not expected IMO, anyways yeah I would guess container image building could be a github action, I think that was considered originally too.

tuminoid commented 3 months ago

The change is good. However, please note that there have been some issues with VM scheduling for the jobs. For instance, this job took 15 hours when scheduling time is included: Build #62.

Maybe these jobs could be moved from Jenkins to github action runners in the future to improve reliability

Yes agreed. We can see that after scheduling, the build itself took almost 21minutes (~1250 sec), which would've timeouted as well, so this fix is needed. It would then leave 30mins for scheduling as it is counted as well. Similarly for sushy/vmbc, the safety allows ~10 minutes of scheduling. We should be able to accomplish that in Jenkins or with runners. Jenkins scheduling we can fix ourselves, but runners would be at mercy of CNCF and other runner users.

edit: Correcting myself: for runners scheduling is not an issue, since if the workflow isn't scheduled, the time isn't running. Doh :)

tuminoid commented 3 months ago

/cc @dtantsur @elfosardo

dtantsur commented 3 months ago

/approve

metal3-io-bot commented 3 months ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dtantsur, kashifest, NymanRobin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/metal3-io/ironic-image/blob/main/OWNERS)~~ [dtantsur] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
metal3-io-bot commented 3 months ago

@tuminoid: new pull request created: #520

In response to [this](https://github.com/metal3-io/ironic-image/pull/519#issuecomment-2159935163): >/cherry-pick release-24.1 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
metal3-io-bot commented 3 months ago

@tuminoid: new pull request created: #521

In response to [this](https://github.com/metal3-io/ironic-image/pull/519#issuecomment-2159935281): >/cherry-pick release-24.0 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
metal3-io-bot commented 3 months ago

@tuminoid: new pull request created: #522

In response to [this](https://github.com/metal3-io/ironic-image/pull/519#issuecomment-2159935410): >/cherry-pick release-23.1 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.