search

fedora-copr / copr

RPM build system - upstream for https://copr.fedorainfracloud.org/
115 stars 59 forks source link

Copr build fails because an ephemeral machine fails or SSH drops? #3422

Closed praiskup closed 1 week ago

praiskup commented 1 week ago 
[2024-09-24 11:25:10,065][  INFO][PID:1333655] Downloading the builder-live.log file, attempt 1
[2024-09-24 11:25:10,067][  INFO][PID:1333655] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c1d4 copr-rpmbuild-log
[2024-09-24 11:25:15,074][  INFO][PID:1333655] Status: False
[2024-09-24 11:35:15,267][  INFO][PID:1333655] Status: False
[2024-09-24 11:45:15,394][  INFO][PID:1333655] Status: False
[2024-09-24 11:55:15,558][  INFO][PID:1333655] Status: False
[2024-09-24 12:05:15,785][  INFO][PID:1333655] Status: False
[2024-09-24 12:15:16,218][  INFO][PID:1333655] Status: False
[2024-09-24 12:25:16,666][  INFO][PID:1333655] Status: False
[2024-09-24 12:35:16,908][  INFO][PID:1333655] Status: False
[2024-09-24 12:45:17,238][  INFO][PID:1333655] Status: False
...

The output Status: False is weird. I don't understand what is happening.

Then, another build has been able to finish (fail) with:

[2024-09-24 11:35:14,369][  INFO][PID:1333803] Status: False
[2024-09-24 11:45:14,491][  INFO][PID:1333803] Status: False
[2024-09-24 11:55:14,668][  INFO][PID:1333803] Status: False
[2024-09-24 12:05:14,975][  INFO][PID:1333803] Status: False
[2024-09-24 12:15:15,379][  INFO][PID:1333803] Status: False
[2024-09-24 12:25:15,842][  INFO][PID:1333803] Status: False
[2024-09-24 12:29:07,827][  INFO][PID:1333803] Downloading results from builder
[2024-09-24 12:29:07,830][  INFO][PID:1333803] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a2]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:3408/fedora-40-ppc64le/08065289-copr-rpmbuild started
[2024-09-24 12:29:07,833][  INFO][PID:1333803] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a2]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:3408/fedora-40-ppc64le/08065289-copr-rpmbuild/ &> /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:3408/fedora-40-ppc64le/08065289-copr-rpmbuild/build-08065289.rsync.log
[2024-09-24 12:29:08,651][  INFO][PID:1333803] rsyncing finished.
[2024-09-24 12:29:08,652][  INFO][PID:1333803] Releasing VM back to pool
[2024-09-24 12:29:08,665][  INFO][PID:1333803] Searching for 'success' file in resultdir
[2024-09-24 12:29:08,665][ ERROR][PID:1333803] Build failed: Backend process error: No success file => build failure
praiskup commented 1 week ago

This has been fixed by 04d375be0790c1c272f596cf150f3dd00a63ba2d (nicer commit message), and #3425 may make it even more user friendly. The original problem was caused by hanging testsuite (some infinite recursion, or infinite memory consumption that eventually failed the bilder).