failure is not displayed on machine list, only on package view

[tpokorra]

MariaDB [lbs]> select * from build order by started desc limit 10;
+-------+----------+--------------+-------------+--------+----------------------------------+------------+--------+---------+-------+----------+-------------+------------------------+------------------------------------+---------------------+---------------------+---------+--------------+-------------+
| id    | status   | username     | projectname | secret | packagename                      | branchname | distro | release | arch  | avoidlxc | avoiddocker | dependsOnOtherProjects | buildmachine                       | started             | finished            | hanging | buildsuccess | buildnumber |
+-------+----------+--------------+-------------+--------+----------------------------------+------------+--------+---------+-------+----------+-------------+------------------------+------------------------------------+---------------------+---------------------+---------+--------------+-------------+
| 28966 | FINISHED | tpokorra     | lbs         | f      | lxc-scripts                      | master     | ubuntu | focal   | amd64 |        1 |           0 |                        | build04docker.lbs.solidcharity.com | 2020-09-19 06:41:01 | 2020-09-19 06:42:53 |       0 | success      |           3 |
| 28965 | FINISHED | tpokorra     | lbs         | f      | lxc-scripts                      | master     | ubuntu | focal   | amd64 |        1 |           0 |                        | build04docker.lbs.solidcharity.com | 2020-09-19 06:29:11 | 2020-09-19 06:31:48 |       0 | success      |           2 |
| 28955 | BUILDING | solidcharity | openpetra   | f      | openpetra-test-mysql             | test       | debian | buster  | amd64 |        0 |           1 |                        | build04lxd.lbs.solidcharity.com    | 2020-09-19 06:27:01 | 0000-00-00 00:00:00 |       0 | NULL         |        NULL |

there are 2 finished jobs the failed job looks like if it is still building

package: looks into the logs: {{"Succeeded" if package['logs'][buildtarget+"-"+branchname][buildnumber]["resultcode"] == "success" else "Failure"}}

[tpokorra]

the log looks like this:

[00:05:12] Permission denied, please try again.
[00:05:12] Permission denied, please try again.
[00:05:12] Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
[00:05:22] LBSERROR: Problem with startmachine
[00:05:36] This build took about 5 minutes

It seems, the machine was released, but the job not marked as finished https://github.com/SolidCharity/LightBuildServer/blob/9400b030c4b13bbff38c353811ce1ba43dc886bb/lib/Build.py#L216

from journalctl:

Sep 19 07:22:50 180-lbs.solidcharity.com gunicorn[36722]: [00:03:48] now running: ssh -f -o "StrictHostKeyChecking no" -o Port=22 -i /etc/lightbuildserver/container/container_rsa root@10.0.4.5 "export LC_ALL=C; (echo "container is running") 2>&1; echo \$?"
Sep 19 07:23:00 180-lbs.solidcharity.com gunicorn[36722]: [00:03:58] LBSERROR: Problem with startmachine
Sep 19 07:23:00 180-lbs.solidcharity.com gunicorn[36722]: ReleaseMachine build04lxd.lbs.solidcharity.com
Sep 19 07:23:00 180-lbs.solidcharity.com gunicorn[36722]: [00:00:00] now running: ssh -f -o "StrictHostKeyChecking no" -p 2222 -i /etc/lightbuildserver/container/container_rsa root@build04lxd.lbs.solidcharity.com "export LC_ALL=C; (lxc stop l005-build04lxd-lbs-solidcharity-com && sleep 10) 2>&1; echo \$?"
Sep 19 07:23:01 180-lbs.solidcharity.com systemd[1]: Started Session c524 of user uwsgi.
Sep 19 07:23:01 180-lbs.solidcharity.com CROND[37715]: (uwsgi) CMD (/usr/share/lightbuildserver/cron.sh)
Sep 19 07:23:12 180-lbs.solidcharity.com gunicorn[36722]: [00:04:10] This build took about 4 minutes
Sep 19 07:24:01 180-lbs.solidcharity.com systemd[1]: Started Session c525 of user uwsgi.

[tpokorra]

running lbs.py directly, it seems the ssh is waiting for the root password, and blocking the whole process.

[tpokorra]

when running directly from the console, I see this error:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "./../lib/Build.py", line 230, in buildpackage
    self.logger.email(self.config['lbs']['EmailFromAddress'], userconfig['EmailToAddress'], "LBS Result for " + projectname + "/" + packagename, self.config['lbs']['LBSUrl'] + "/logs/" + logpath + "/" + str(buildnumber))
  File "./../lib/Logger.py", line 135, in email
    server.login(self.emailuser, self.emailpassword)
  File "/usr/lib64/python3.6/smtplib.py", line 730, in login
    raise last_exception
  File "/usr/lib64/python3.6/smtplib.py", line 721, in login
    initial_response_ok=initial_response_ok)
  File "/usr/lib64/python3.6/smtplib.py", line 642, in auth
    raise SMTPAuthenticationError(code, resp)
smtplib.SMTPAuthenticationError: (535, b'5.7.8 Error: authentication failed: authentication failure')