ejoerns / poky-nfsroot

nfs-export-updater patches for poky
Other
1 stars 1 forks source link

bitbake deadlock with multiple pending operations #4

Open Bastian-Krause opened 9 months ago

Bastian-Krause commented 9 months ago

When running another "bitbake" while the nfs-export-updater has not finished it's current operation yet, we seem to be running in some kind of deadlock situation:

bitbake stuck without any progress for minutes:

NOTE: Tasks Summary: Attempted 3559 tasks of which 3558 didn't need to be rerun and 1 failed.
NOTE: Writing buildhistory
NOTE: Writing buildhistory took: 4 seconds
NOTE: Notifying nfs-export-updater server..
Setscene tasks: 1384 of 1384
No currently running tasks (1792 of 3604)  49% |#######################################################################
Setscene tasks: 1384 of 1384
No currently running tasks (1792 of 3604)  49% |##################################                                    |

nfs-export-updater:

INFO: Package index updated.
INFO:nfs-export-updater:Package index updated.
INFO:BitBake:Reconnecting to bitbake server...
INFO:BitBake:No reply from server in 30s
INFO:BitBake:Retrying server connection (#1)... (16:30:47.189585)
NOTE: Reconnecting to bitbake server...
NOTE:BitBake:Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE:BitBake:No reply from server in 30s
NOTE: Retrying server connection (#1)... (16:30:47.189585)
NOTE:BitBake:Retrying server connection (#1)... (16:30:47.189585)
INFO:BitBake:Reconnecting to bitbake server...
INFO:BitBake:No reply from server in 30s
INFO:BitBake:Retrying server connection (#2)... (16:31:47.245574)
NOTE: Reconnecting to bitbake server...
NOTE:BitBake:Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE:BitBake:No reply from server in 30s
NOTE: Retrying server connection (#2)... (16:31:47.245574)
NOTE:BitBake:Retrying server connection (#2)... (16:31:47.245574)
INFO:BitBake:Reconnecting to bitbake server...
INFO:BitBake:No reply from server in 30s
INFO:BitBake:Retrying server connection (#3)... (16:32:47.301563)
NOTE: Reconnecting to bitbake server...
NOTE:BitBake:Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE:BitBake:No reply from server in 30s
NOTE: Retrying server connection (#3)... (16:32:47.301563)
NOTE:BitBake:Retrying server connection (#3)... (16:32:47.301563)
INFO:BitBake:Reconnecting to bitbake server...
INFO:BitBake:No reply from server in 30s
INFO:BitBake:Retrying server connection (#4)... (16:33:47.358051)
NOTE: Reconnecting to bitbake server...
NOTE:BitBake:Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE:BitBake:No reply from server in 30s
NOTE: Retrying server connection (#4)... (16:33:47.358051)
NOTE:BitBake:Retrying server connection (#4)... (16:33:47.358051)
INFO:BitBake:Reconnecting to bitbake server...
INFO:BitBake:No reply from server in 30s
INFO:BitBake:Retrying server connection (#5)... (16:34:52.376032)
NOTE: Reconnecting to bitbake server...
NOTE:BitBake:Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE:BitBake:No reply from server in 30s
NOTE: Retrying server connection (#5)... (16:34:52.376032)
NOTE:BitBake:Retrying server connection (#5)... (16:34:52.376032)
INFO:BitBake:Reconnecting to bitbake server...
INFO:BitBake:No reply from server in 30s
Bastian-Krause commented 9 months ago

With a little patience, the situation resolved itself.

bitbake:

NOTE: Received response: Server received: bitbake_done

Summary: 1 task failed:
  [...]/recipes-kernel/linux/linux-customer_6.5.bb:do_compile
Summary: There was 1 WARNING message.
Summary: There were 2 ERROR messages, returning a non-zero exit code.

nfs-export-updater:

INFO:BitBake:Retrying server connection (#6)... (16:35:57.433600)
NOTE: Reconnecting to bitbake server...
NOTE:BitBake:Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE:BitBake:No reply from server in 30s
NOTE: Retrying server connection (#6)... (16:35:57.433600)
NOTE:BitBake:Retrying server connection (#6)... (16:35:57.433600)
INFO:BitBake:Reconnecting to bitbake server...
INFO:BitBake:No reply from server in 30s
INFO:BitBake:Retrying server connection (#7)... (16:37:02.496375)
NOTE: Reconnecting to bitbake server...
NOTE:BitBake:Reconnecting to bitbake server...
NOTE: No reply from server in 30s
NOTE:BitBake:No reply from server in 30s
NOTE: Retrying server connection (#7)... (16:37:02.496375)
NOTE:BitBake:Retrying server connection (#7)... (16:37:02.496375)
INFO:BitBake:Reconnecting to bitbake server...
INFO:BitBake:No reply from server in 30s
INFO:BitBake:Retrying server connection (#8)... (16:38:07.553593)
CRITICAL:BitBake.Main:Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log).
ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-25' coro=<update_nfsroot() done, defined at YOCTO.BSP/poky/scripts/lib/nfs_export_updater/pkgindex.py:240> exception=BBHandledException()>
Traceback (most recent call last):
  File "YOCTO.BSP/poky/scripts/lib/nfs_export_updater/pkgindex.py", line 242, in update_nfsroot
    update_packages(rootfs_recipe, nfsroot)
  File "YOCTO.BSP/poky/scripts/lib/nfs_export_updater/pkgindex.py", line 181, in update_packages
    tinfoil.prepare(quiet=1)
  File "YOCTO.BSP/poky/bitbake/lib/bb/tinfoil.py", line 394, in prepare
    self.server_connection, ui_module = setup_bitbake(config_params, extrafeatures)
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "YOCTO.BSP/poky/bitbake/lib/bb/main.py", line 505, in setup_bitbake
    bb.fatal("Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log).")
  File "YOCTO.BSP/poky/bitbake/lib/bb/__init__.py", line 182, in fatal
    raise BBHandledException()
bb.BBHandledException
Received data: bitbake_done
Handling 'bitbake_done'
Sending respone...
Client disconnected
INFO:BitBake:Reconnecting to bitbake server...
Loading cache: 100% |###################################################################################| Time: 0:00:00
Parsing recipes: 100% |#################################################################################| Time: 0:00:00
INFO: Updating package index...
[...]

Something's seems a little fishy here, but as long as it resolves itself, I'm a happy nfsroot user :)