search

fedora-copr / copr

RPM build system - upstream for https://copr.fedorainfracloud.org/
111 stars 58 forks source link

User SSH access to builders closed sooner than expected #3250

Open FrostyX opened 2 months ago

FrostyX commented 2 months ago

It seems that Copr kills builders with user SSH access after an hour even though a user requested more.

Prolonging for 10 hours:

[root@aws-x86-64-spot-prod-06866057-20240506-121748 ~]# copr-builder prolong --hours 10
Prolonged to 2024-05-06 23:33:39.627366

Let's see how long I can keep connected:

[root@aws-x86-64-spot-prod-06866057-20240506-121748 ~]# while true; do date; sleep 60; done
Mon May  6 12:36:53 PM UTC 2024
Mon May  6 12:37:53 PM UTC 2024
Mon May  6 12:38:53 PM UTC 2024
...
Mon May  6 01:32:53 PM UTC 2024
Mon May  6 01:33:53 PM UTC 2024
Mon May  6 01:34:53 PM UTC 2024
Connection to 54.224.58.243 closed by remote host.
Connection to 54.224.58.243 closed.

The whole builder seems to be terminated because I cannot reconnect:

[jkadlcik@zeratul ~]$ ssh root@54.224.58.243
ssh: connect to host 54.224.58.243 port 22: Connection timed out
kwk commented 2 months ago

Thank you for looking into this.

FrostyX commented 2 months ago

@kwk I successfully reproduced the issue on the first try once you reported it to me. However, when I wanted to fix the issue last week, I couldn't reproduce it anymore. Maybe this happens only occasionally under some weird circumstances.

My plan is to improve the logging around VM expiration so that once this happens, we better understand what happened. Until that, please give it another chance and keep us posted.