search

buildstream-migration / bst-staging

GNU Lesser General Public License v2.1
0 stars 0 forks source link

Stack trace when running out of disk space #623

Closed Cynical-Optimist closed 4 years ago

Cynical-Optimist commented 4 years ago

See original issue on GitLab In GitLab by [Gitlab user @tristanvb] on Sep 3, 2018, 08:13

Summary

When running a build of GNOME using BuildStream master, instead of automatically cleaning up the artifact cache as expected, I received unhandled exceptions due to no remaining space left on my disk.

Steps to reproduce

Run a build of gnome-build-meta without enough remaining disk space to cache the new artifacts.

What is the current bug behavior?

Stack trace

What is the expected correct behavior?

Either the artifact cache makes room for the new incoming artifacts by automatically cleaning it up, or I get an elegant failure message telling me that all the artifacts needed for my build don't fit on the disk.

Relevant logs and/or screenshots


========================================================================================= 02:07:16 gnome (470/471) =========================================================================================
                                                                 (Tracked:77 166 0)→ (Pulled:0 470 0)→ (Fetched:138 332 0)→ (Built:107 0 0)                                                                 
                                                                                          ~~~~~ Active Tasks ~~~~~                                                                                          
[00:13:00][Build:freedesktop-sdk.bst:base/cmake.bst  ] [00:08:08][Build:freedesktop-sdk.bst:base/nss.bst] [00:06:29][Build:freedesktop-sdk.bst:base/gettext.bst]
[00:03:08][Build:freedesktop-sdk.bst:base/locales.bst] 
Process Process-1415:
Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 421, in add_object
    out.write(chunk)
  File "/usr/lib/python3.5/tempfile.py", line 622, in func_wrapper
    return func(*args, **kwargs)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 434, in add_object
    os.link(out.name, objpath)
  File "/usr/lib/python3.5/tempfile.py", line 640, in __exit__
    result = self.file.__exit__(exc, value, tb)
OSError: [Errno 28] No space left on device

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 407, in timed_activity
    yield
  File "/codethink/GNOME/buildstream/buildstream/plugin.py", line 556, in timed_activity
    yield
  File "/codethink/GNOME/buildstream/buildstream/element.py", line 1650, in _assemble
    self.__artifacts.commit(self, assembledir, self.__get_cache_keys_for_commit())
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 118, in commit
    tree = self._create_tree(content)
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 640, in _create_tree
    self._create_tree(full_path, digest=dirnode.digest)
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 640, in _create_tree
    self._create_tree(full_path, digest=dirnode.digest)
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 640, in _create_tree
    self._create_tree(full_path, digest=dirnode.digest)
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 640, in _create_tree
    self._create_tree(full_path, digest=dirnode.digest)
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 640, in _create_tree
    self._create_tree(full_path, digest=dirnode.digest)
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 644, in _create_tree
    self.add_object(path=full_path, digest=filenode.digest)
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 441, in add_object
    raise ArtifactError("Failed to hash object: {}".format(e)) from e
buildstream._exceptions.ArtifactError: Failed to hash object: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 398, in _child_action
    result = self.child_process()
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/elementjob.py", line 94, in child_process
    return self._action_cb(self._element)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/queues/buildqueue.py", line 70, in process
    element._assemble()
  File "/codethink/GNOME/buildstream/buildstream/element.py", line 1650, in _assemble
    self.__artifacts.commit(self, assembledir, self.__get_cache_keys_for_commit())
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/codethink/GNOME/buildstream/buildstream/plugin.py", line 556, in timed_activity
    yield
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 415, in timed_activity
    self.message(message)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 350, in message
    self._record_message(message)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 552, in _record_message
    self._log_handle.flush()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 477, in recorded_messages
    yield self._log_filename
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 432, in _child_action
    logfile=filename)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/elementjob.py", line 106, in message
    **args))
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 350, in message
    self._record_message(message)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 552, in _record_message
    self._log_handle.flush()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 448, in _child_action
    self._child_shutdown(RC_OK)

BUG: Message handling out of sync, unable to retrieve failure message for element autotools element at base/cmake.bst [line 1 column 0]

========================================================================================= 02:07:17 gnome (470/471) =========================================================================================
                                                                 (Tracked:77 166 0)→ (Pulled:0 470 0)→ (Fetched:138 332 0)→ (Built:107 0 1)                                                                 
                                                                                          ~~~~~ Active Tasks ~~~~~                                                                                          
[00:08:09][Build:freedesktop-sdk.bst:base/nss.bst  ] [00:06:29][Build:freedesktop-sdk.bst:base/gettext.bst] [00:03:08][Build:freedesktop-sdk.bst:base/locales.bst]
[00:00:00][Build:freedesktop-sdk.bst:base/gnupg.bst] 
Process Process-1437:
Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 477, in recorded_messages
    yield self._log_filename
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 394, in _child_action
    self.message(MessageType.START, self.action_name, logfile=filename)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/elementjob.py", line 106, in message
    **args))
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 350, in message
    self._record_message(message)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 552, in _record_message
    self._log_handle.flush()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 448, in _child_action
    self._child_shutdown(RC_OK)

BUG: Message handling out of sync, unable to retrieve failure message for element autotools element at base/gnupg.bst [line 1 column 0]

========================================================================================= 02:07:17 gnome (470/471) =========================================================================================
                                                                 (Tracked:77 166 0)→ (Pulled:0 470 0)→ (Fetched:138 332 0)→ (Built:107 0 2)                                                                 
                                                                                          ~~~~~ Active Tasks ~~~~~                                                                                          
[00:08:09][Build:freedesktop-sdk.bst:base/nss.bst     ] [00:06:29][Build:freedesktop-sdk.bst:base/gettext.bst] [00:03:08][Build:freedesktop-sdk.bst:base/locales.bst]
[00:00:00][Build:freedesktop-sdk.bst:base/automake.bst] 
Process Process-1438:
Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 477, in recorded_messages
    yield self._log_filename
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 394, in _child_action
    self.message(MessageType.START, self.action_name, logfile=filename)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/elementjob.py", line 106, in message
    **args))
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 350, in message
    self._record_message(message)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 552, in _record_message
    self._log_handle.flush()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 448, in _child_action
    self._child_shutdown(RC_OK)

BUG: Message handling out of sync, unable to retrieve failure message for element autotools element at base/automake.bst [line 1 column 0]

========================================================================================= 02:07:17 gnome (470/471) =========================================================================================
                                                                 (Tracked:77 166 0)→ (Pulled:0 470 0)→ (Fetched:138 332 0)→ (Built:107 0 3)                                                                 
                                                                                          ~~~~~ Active Tasks ~~~~~                                                                                          
[00:08:09][Build:freedesktop-sdk.bst:base/nss.bst  ] [00:06:30][Build:freedesktop-sdk.bst:base/gettext.bst] [00:03:08][Build:freedesktop-sdk.bst:base/locales.bst]
[00:00:00][Build:freedesktop-sdk.bst:base/groff.bst] 
Process Process-1439:
Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 477, in recorded_messages
    yield self._log_filename
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 394, in _child_action
    self.message(MessageType.START, self.action_name, logfile=filename)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/elementjob.py", line 106, in message
    **args))
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 350, in message
    self._record_message(message)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 552, in _record_message
    self._log_handle.flush()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 448, in _child_action
    self._child_shutdown(RC_OK)

BUG: Message handling out of sync, unable to retrieve failure message for element autotools element at base/groff.bst [line 1 column 0]

========================================================================================= 02:07:17 gnome (470/471) =========================================================================================
                                                                 (Tracked:77 166 0)→ (Pulled:0 470 0)→ (Fetched:138 332 0)→ (Built:107 0 4)                                                                 
                                                                                          ~~~~~ Active Tasks ~~~~~                                                                                          
[00:08:09][Build:freedesktop-sdk.bst:base/nss.bst  ] [00:06:30][Build:freedesktop-sdk.bst:base/gettext.bst] [00:03:08][Build:freedesktop-sdk.bst:base/locales.bst]
[00:00:00][Build:freedesktop-sdk.bst:base/bison.bst] 
Process Process-1440:
Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 477, in recorded_messages
    yield self._log_filename
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 394, in _child_action
    self.message(MessageType.START, self.action_name, logfile=filename)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/elementjob.py", line 106, in message
    **args))
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 350, in message
    self._record_message(message)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 552, in _record_message
    self._log_handle.flush()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 448, in _child_action
    self._child_shutdown(RC_OK)

BUG: Message handling out of sync, unable to retrieve failure message for element autotools element at base/bison.bst [line 1 column 0]

========================================================================================= 02:07:17 gnome (470/471) =========================================================================================
                                                                 (Tracked:77 166 0)→ (Pulled:0 470 0)→ (Fetched:138 332 0)→ (Built:107 0 5)                                                                 
                                                                                          ~~~~~ Active Tasks ~~~~~                                                                                          
[00:08:09][Build:freedesktop-sdk.bst:base/nss.bst] [00:06:30][Build:freedesktop-sdk.bst:base/gettext.bst] [00:03:08][Build:freedesktop-sdk.bst:base/locales.bst]
[00:00:00][Build:freedesktop-sdk.bst:base/ed.bst ] 
Process Process-1441:
Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 477, in recorded_messages
    yield self._log_filename
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 394, in _child_action
    self.message(MessageType.START, self.action_name, logfile=filename)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/elementjob.py", line 106, in message
    **args))
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 350, in message
    self._record_message(message)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 552, in _record_message
    self._log_handle.flush()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 448, in _child_action
    self._child_shutdown(RC_OK)

BUG: Message handling out of sync, unable to retrieve failure message for element autotools element at base/ed.bst [line 1 column 0]

========================================================================================= 02:07:17 gnome (470/471) =========================================================================================
                                                                 (Tracked:77 166 0)→ (Pulled:0 470 0)→ (Fetched:138 332 0)→ (Built:107 0 6)                                                                 
                                                                                          ~~~~~ Active Tasks ~~~~~                                                                                          
[00:08:09][Build:freedesktop-sdk.bst:base/nss.bst] [00:06:30][Build:freedesktop-sdk.bst:base/gettext.bst] [00:03:09][Build:freedesktop-sdk.bst:base/locales.bst]
[00:00:00][Build:freedesktop-sdk.bst:base/gdb.bst] 
Process Process-1442:
Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 477, in recorded_messages
    yield self._log_filename
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 394, in _child_action
    self.message(MessageType.START, self.action_name, logfile=filename)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/elementjob.py", line 106, in message
    **args))
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 350, in message
    self._record_message(message)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 552, in _record_message
    self._log_handle.flush()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 448, in _child_action
    self._child_shutdown(RC_OK)

BUG: Message handling out of sync, unable to retrieve failure message for element autotools element at base/gdb.bst [line 1 column 0]

========================================================================================= 02:07:17 gnome (470/471) =========================================================================================
                                                                 (Tracked:77 166 0)→ (Pulled:0 470 0)→ (Fetched:138 332 0)→ (Built:107 0 7)                                                                 
                                                                                          ~~~~~ Active Tasks ~~~~~                                                                                          
[00:08:09][Build:freedesktop-sdk.bst:base/nss.bst     ] [00:06:30][Build:freedesktop-sdk.bst:base/gettext.bst] [00:03:09][Build:freedesktop-sdk.bst:base/locales.bst]
[00:00:00][Build:freedesktop-sdk.bst:base/intltool.bst] 
Process Process-1443:
Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 477, in recorded_messages
    yield self._log_filename
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 394, in _child_action
    self.message(MessageType.START, self.action_name, logfile=filename)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/elementjob.py", line 106, in message
    **args))
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 350, in message
    self._record_message(message)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 552, in _record_message
    self._log_handle.flush()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 448, in _child_action
    self._child_shutdown(RC_OK)

BUG: Message handling out of sync, unable to retrieve failure message for element autotools element at base/intltool.bst [line 1 column 0]

========================================================================================= 02:07:18 gnome (470/471) =========================================================================================
                                                                 (Tracked:77 166 0)→ (Pulled:0 470 0)→ (Fetched:138 332 0)→ (Built:107 0 8)                                                                 
                                                                                          ~~~~~ Active Tasks ~~~~~                                                                                          
[00:08:10][Build:freedesktop-sdk.bst:base/nss.bst   ] [00:06:30][Build:freedesktop-sdk.bst:base/gettext.bst] [00:03:09][Build:freedesktop-sdk.bst:base/locales.bst]
[00:00:00][Build:freedesktop-sdk.bst:base/nettle.bst] 
Process Process-1444:
Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 477, in recorded_messages
    yield self._log_filename
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 394, in _child_action
    self.message(MessageType.START, self.action_name, logfile=filename)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/elementjob.py", line 106, in message
    **args))
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 350, in message
    self._record_message(message)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 552, in _record_message
    self._log_handle.flush()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 448, in _child_action
    self._child_shutdown(RC_OK)

BUG: Message handling out of sync, unable to retrieve failure message for element autotools element at base/nettle.bst [line 1 column 0]

Process Process-1434:
Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 407, in timed_activity
    yield
  File "/codethink/GNOME/buildstream/buildstream/plugin.py", line 556, in timed_activity
    yield
  File "/codethink/GNOME/buildstream/buildstream/buildelement.py", line 186, in assemble
    self.__run_command(sandbox, cmd, command_name)
  File "/codethink/GNOME/buildstream/buildstream/buildelement.py", line 245, in __run_command
    collect=self.get_variable('install-root'))
buildstream.element.ElementError: Command 'if [ -n "bst_build_dir" ]; then
  cd bst_build_dir
fi
make -j1 DESTDIR="/buildstream-install" install' failed with exitcode 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/element.py", line 1540, in _assemble
    collect = self.assemble(sandbox)
  File "/codethink/GNOME/buildstream/buildstream/buildelement.py", line 186, in assemble
    self.__run_command(sandbox, cmd, command_name)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/codethink/GNOME/buildstream/buildstream/plugin.py", line 556, in timed_activity
    yield
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 415, in timed_activity
    self.message(message)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 350, in message
    self._record_message(message)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 552, in _record_message
    self._log_handle.flush()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 398, in _child_action
    result = self.child_process()
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/elementjob.py", line 94, in child_process
    return self._action_cb(self._element)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/queues/buildqueue.py", line 70, in process
    element._assemble()
  File "/codethink/GNOME/buildstream/buildstream/element.py", line 1584, in _assemble
    os.mkdir(assembledir)
OSError: [Errno 28] No space left on device: '/home/tristan/.cache/buildstream/build/base-gettext-8l4uct30/artifact'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 477, in recorded_messages
    yield self._log_filename
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 432, in _child_action
    logfile=filename)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/elementjob.py", line 106, in message
    **args))
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 350, in message
    self._record_message(message)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 552, in _record_message
    self._log_handle.flush()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 448, in _child_action
    self._child_shutdown(RC_OK)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/codethink/GNOME/buildstream/buildstream/_context.py", line 480, in recorded_messages
    self._log_filename = None
OSError: [Errno 28] No space left on device

BUG: Message handling out of sync, unable to retrieve failure message for element autotools element at base/gettext.bst [line 1 column 0]

Possible fixes

Other relevant information

The content of my buildstream.conf file:


cache:
  quota: 20G
Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 3, 2018, 09:45

marked this issue as related to #609

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tlater] on Sep 3, 2018, 10:28

Yep, similar to #609, I suspect we're simply running out of headroom.

I wonder if I can get docker to simulate an almost-full disk so I can find all the code that will cause exceptions in...

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 3, 2018, 12:04

Update:

The result is:

[--:--:--][][] START   cleanup.24886.log
[--:--:--][2701f46e][ pull:core/gnome-backgrounds.bst    ] INFO    Downloaded artifact 2701f46e
[00:03:34][2701f46e][ pull:core/gnome-backgrounds.bst    ] SUCCESS gnome/core-gnome-backgrounds/2701f46e-pull.23674.log
[--:--:--][81d45800][ pull:sdk/yelp.bst                  ] START   gnome/sdk-yelp/81d45800-pull.25410.log
[--:--:--][81d45800][ pull:sdk/yelp.bst                  ] INFO    Pulling 81d45800 <- https://testcache.codethink.co.uk:11001
[00:00:25][][] BUG     cleanup

    An unhandled exception occured:

    Traceback (most recent call last):
      File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 398, in _child_action
        result = self.child_process()
      File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/cleanupjob.py", line 30, in child_process
        return self._cache.clean()
      File "/codethink/GNOME/buildstream/buildstream/_artifactcache/artifactcache.py", line 257, in clean
        size = self.remove(to_remove)
      File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 564, in remove
        pruned = self.prune()
      File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 586, in prune
        self._reachable_refs_dir(reachable, tree)
      File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 732, in _reachable_refs_dir
        with open(self.objpath(tree), 'rb') as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/home/tristan/.cache/buildstream/artifacts/cas/objects/7f/4c864e8edd35a496c17285a1035b7221d3d317f90deb65a964c1d77c5c4364'

========================================================================================= 02:03:03 gnome (471/471) =========================================================================================
                                                                           (Pulled:355 24 0)→ (Fetched:0 379 0)→ (Built:4 355 0)                                                                           
                                                                                          ~~~~~ Active Tasks ~~~~~                                                                                          
[--:--:--][765ee514][ main:sdk/WebKitGTK+.bst            ] STATUS  Pull terminating
[--:--:--][7d84845e][ main:core-deps/sane-backends.bst   ] STATUS  Pull terminating
[--:--:--][e64319db][ main:core-deps/samba.bst           ] STATUS  Pull terminating
[--:--:--][6777c202][ main:core/gedit.bst                ] STATUS  Pull terminating
[--:--:--][734d17cd][ main:core/baobab.bst               ] STATUS  Pull terminating
[--:--:--][9233eccc][ main:core/gnome-calculator.bst     ] STATUS  Pull terminating
[--:--:--][ad80eb93][ main:core/rygel.bst                ] STATUS  Pull terminating
[--:--:--][33b4a3c5][ main:core/gnome-user-docs.bst      ] STATUS  Pull terminating
[--:--:--][d044149b][ main:sdk/gnome-themes-extra.bst    ] STATUS  Pull terminating
[--:--:--][81d45800][ main:sdk/yelp.bst                  ] STATUS  Pull terminating
[--:--:--][][] STATUS  cleanup terminating
Unknown child process pid 23617, will report returncode 255
Unknown child process pid 25410, will report returncode 255
Unknown child process pid 19227, will report returncode 255
Unknown child process pid 23645, will report returncode 255
Unknown child process pid 6431, will report returncode 255
Unknown child process pid 24615, will report returncode 255
Unknown child process pid 28020, will report returncode 255
Unknown child process pid 21494, will report returncode 255
Unknown child process pid 25460, will report returncode 255
Unknown child process pid 16059, will report returncode 255
Unknown child process pid 14847, will report returncode 255
[02:03:03][][] FAILURE Build

Pipeline Summary
  Total:       471
  Session:     471
  Pull Queue:  processed 355, skipped 24,  failed 10 
  Fetch Queue: processed 0,   skipped 379, failed 0  
  Build Queue: processed 4,   skipped 355, failed 0
Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 3, 2018, 12:55

mentioned in commit d15f3e101943a511cc8c73fb19536a926ae247f6

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 3, 2018, 12:55

mentioned in merge request !775

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 3, 2018, 13:19

mentioned in commit 0844cc4fe8b302b5d69ebd01abd88003023feac9

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 3, 2018, 15:46

UPDATE: With !775 applied to the bst-1.2 branch locally, it appears that:

It failed with the same stack trace because of the same corrupt state which was caused by pull/build jobs being run concurrently with cleanup the last time around.

Seems that !775 does fix the concurrency problem, but it would be nice to recover this artifact cache and run further tests.

Log file from this run follows:

[--:--:--][][] START   Build
[--:--:--][][] START   Loading elements
[00:00:03][][] SUCCESS Loading elements
[--:--:--][][] START   Resolving elements
[00:00:02][][] SUCCESS Resolving elements
[--:--:--][][] START   Initializing remote caches
[00:00:02][][] SUCCESS Initializing remote caches
[--:--:--][][] START   Resolving cached state
[00:00:02][][] SUCCESS Resolving cached state
[--:--:--][][] START   Checking sources
[00:00:00][][] SUCCESS Checking sources

BuildStream Version 1.2.0+2.g4e2d5199
  Session Start: Monday, 03-09-2018 at 21:28:41
  Project:       gnome (/codethink/GNOME/gnome-build-meta)
  Targets:       core.bst

User Configuration
  Configuration File:      /home/tristan/.config/buildstream.conf
  Log Files:               /home/tristan/.cache/buildstream/logs
  Source Mirrors:          /home/tristan/.cache/buildstream/sources
  Build Area:              /home/tristan/.cache/buildstream/build
  Artifact Cache:          /home/tristan/.cache/buildstream/artifacts
  Strict Build Plan:       Yes
  Maximum Fetch Tasks:     10
  Maximum Build Tasks:     4
  Maximum Push Tasks:      4
  Maximum Network Retries: 2

Project Options
  arch: x86_64

Loaded Plugins
  Source Plugins
    - crate

Pipeline
     waiting 81944a22f0a775c559bfde2ad3b8e1991af2a2e36b227b477d3d6f133460a3ff bootstrap-junction.bst:cross-installation-links.bst 
     waiting d1f300b6e2b55e3b4db358e673330295e61742321c394c4e4945e329387ea44a bootstrap-junction.bst:dependencies/base-sdk-image.bst 
     waiting dfb997eee4fd2907691086336fae1b6e5c71c7590759f68601b04a77430212e1 bootstrap-junction.bst:dependencies/base-sdk-links.bst 
     waiting 5b7b202698332ac7be068c4d44fd282c928bbd65502ad4456b711df62c9bc798 bootstrap-junction.bst:dependencies/base-sdk.bst 
     waiting 05e933b9501359870b7db953cf1f6899b96e692761cfa5cfa4329ae6e575b091 bootstrap-junction.bst:gnu-config.bst 
     waiting 09333c3e684df76c5cb959c591024608cc4bf764fa0cae4b9b70fa3e53c686a7 bootstrap-junction.bst:linux-headers.bst 
     waiting a35dafe9e964d890beec53e3991bc84ba927253305a2f965cf04129ed11ab6c4 bootstrap-junction.bst:binutils-stage1.bst 
     waiting dd47e050c41981f657d85d4556a7922a9ad51c22986f6e9af7344c3ff4793984 bootstrap-junction.bst:tools-lib-path.bst 
     waiting 1726d7d7414e30c2226e1e033fa931da5a510136d52a2f6299e26db031f507ee bootstrap-junction.bst:elfutils-host.bst 
     waiting 568b8e8951347f77692fd9b20473038c5b5c60bcbdcf0caef8ba6278e9338017 bootstrap-junction.bst:file-stage1.bst 
     waiting 990a93a1d273368147f650737f6dc89622c5845a820a260eee3b9654a6ddb0ff bootstrap-junction.bst:debugedit-host.bst 
     waiting c48fc4f5d30ac28ac00025f3653fa468a4caa0f8425b7c5f0bd217f597fc6b3b bootstrap-junction.bst:gcc-stage1.bst 
     waiting e6dd96b97b51cba2e64541d984727bf2f7818348c12a9f436274d45bf20ee19b bootstrap-junction.bst:glibc.bst 
     waiting 91cf646d1ecf0a4c2559ca1616297666827710e5b32d40c882a2fefb5a9a62db bootstrap-junction.bst:libstdc++-stage1.bst 
     waiting 46abe3c5819344691a58835e3257993f274d2a2ad810184070836bbe57685581 bootstrap-junction.bst:gcc-stage2.bst 
     waiting 9c06ca4c0dea095079028f7cbcfb3599e76ae90ed087694ab406dea947cf5dd9 bootstrap-junction.bst:attr.bst 
     waiting 9a77ce62e6242688182ecc988285ddcc54b196eb239023f2ba9c7e0eecfaaf53 bootstrap-junction.bst:acl.bst 
     waiting 74fd948f26a5964108165e051b83a0c90341a4a902259bdcd59a2608bf560270 bootstrap-junction.bst:ncurses-stage1.bst 
     waiting b50ed4d8595220ba91b4df8092df7c9faa77cb5e4f13a3df08e08f006d256fd6 bootstrap-junction.bst:ncurses.bst 
     waiting 58e88337f22d74b5d223297cab314e423e4bf337b1b27a69a5af7c09da57627e bootstrap-junction.bst:readline.bst 
     waiting 2caab918793ada799b22b7a51706b208a671d0489a21c4b6638e83780e2779de bootstrap-junction.bst:bash.bst 
     waiting 4c92cb378c30a067660cd1a2a1cc7fa3c838363563627c2dd0f98e5abb27b107 bootstrap-junction.bst:binutils.bst 
     waiting c396c2e0e0a42f422f09fe4274e37ec434737a3d2530c162095980ffd638ed22 bootstrap-junction.bst:bzip2.bst 
     waiting 0b8f7186df8c799cc32d434dc737a41469b92046ea54f5e9f02b9b91d31685ea bootstrap-junction.bst:coreutils-manpages.bst 
     waiting 4296b0f48c17265d7f4d71cff3b1a338a51f17c96dd8f11eb139e98a037c4b30 bootstrap-junction.bst:coreutils.bst 
     waiting 8d44a0e2f5447d3d41478737b792509e559905333dc31e2619b17b8255079375 bootstrap-junction.bst:zlib.bst 
     waiting 352ab559a1a16ee8f1194ce62caf41a8edb4958874e0526f6fa32a34cfb88f90 bootstrap-junction.bst:elfutils.bst 
     waiting a9424b271099d653d95c21afbf26564f0baaa941ce8d1db3d2fdc768c3821455 bootstrap-junction.bst:file.bst 
     waiting 96df204bb61b782063658665b798899e952b8080e74148e1772a7220f13015eb bootstrap-junction.bst:openssl-static.bst 
     waiting c472abb6ba2767ef58c3ea7bc4a3abc7a914ab6f23cdf60a082f5f5d38145004 bootstrap-junction.bst:popt.bst 
     waiting e63cbbd8199f3a212dd1dc00ce6069699416d33b572d50200e50211a7741ac2d bootstrap-junction.bst:debugedit.bst 
     waiting 49eee84b363bc989579257e4df522420eb8a54b0ce8240bde82190f4c5514d68 bootstrap-junction.bst:diffutils.bst 
     waiting f1f115f0f41ebd5cc2a6255ebb791ce009fe76e5accb6e12562f257ae2f5a0f7 bootstrap-junction.bst:findutils.bst 
     waiting bc4e5008a93a47ff2ac6388a623f752451ec8873427667a10af37ad0159f845c bootstrap-junction.bst:gawk.bst 
     waiting ebadb73b406c5aca30f66f3d70e150f32efe8e47d1a1d2e03feea852829fe2f6 bootstrap-junction.bst:gcc.bst 
     waiting 9bedea826c93af88f486dfe7483fa40b0607c0d9b8c6738f2e1173a7a29ccbb6 bootstrap-junction.bst:grep.bst 
     waiting 76ac8a20ad9347ab280cccd40967a74d30730e1b0913eb08e4d292a137fe1723 bootstrap-junction.bst:make.bst 
     waiting 35c8d8001a10e27fe4d73b37f0ba776e38bc879f37de8ec161900c755feafdc7 bootstrap-junction.bst:procps.bst 
     waiting ed5762d76e27654f8bec924ecfc7ecfdda0355c6722c70f8ce2428445bb1a458 bootstrap-junction.bst:sed.bst 
     waiting 505b6fe98295b24470ecb1752c05f6a815372617197c3880bb75a968705208a1 bootstrap-junction.bst:glibc-stage1.bst 
     waiting ed565f4ed49606d1f7b16987acfe03cc998881a43a17c37747ca587ea022e74a bootstrap-junction.bst:utf-locale.bst 
     waiting 607b28257ba179e539d710b420fed63b37a7167f3e4dd1a3d2fb28f3aaa52e27 bootstrap-junction.bst:bootstrap.bst 
     waiting 2a510461190e14d6c190dc161b36173dcf127d32cd884037fbeaef877496d7aa bootstrap-junction.bst:bootstrap-dev.bst 
      cached e1737c0cc371613296e2d92ee2071311c9045503c4dc5f93ba83120805a5a395 bootstrap-junction.bst:bootstrap-dev-extract.bst 
     waiting 3d16bb72d974efc56fb46a0ce995bc5ab9761951f03cf5df06f8d3c93625fad9 bootstrap-junction.bst:bootstrap-platform.bst 
      cached fa4395d3088971257152151a20f331e582fbcbecec5d9e8dba5440568319191a bootstrap-junction.bst:bootstrap-platform-extract.bst 
      cached 4bcd0fef9cfcc7865a482941b41e2f05bd035c991ed64550b1499ac507f35d92 bootstrap-junction.bst:bootstrap-extract.bst 
      cached 6a1b1d3443d29c04b79bc240a68a0197e538475388fbf7388709a76e590b2343 freedesktop-sdk.bst:bootstrap-import.bst 
      cached f658fd28c375d4039d9ba563fb89a2b36384b9ce7ad928782b86f51f552eed81 freedesktop-sdk.bst:base/perl.bst 
      cached fdf02f4332bcf5127c1d977a20fa27be7c0fa8b5e89c1c42e70cff9b57d2f344 freedesktop-sdk.bst:base/m4.bst 
      cached 092e82e9a478754d3e7ec74251583e81a25d2f4b7122f5cdd3598f9f16a43c31 freedesktop-sdk.bst:base/autoconf.bst 
      cached ff7d38e99800919519d2c8ece3db62cb0713b2aac3b3167a8169a454c669d59a freedesktop-sdk.bst:base/automake.bst 
      cached 9c81474d035d979bc5bb020db096e336159f97703ba7f6b6fb3786e50c569067 freedesktop-sdk.bst:base/libtool.bst 
      cached 733163f0b3f7dcdc9e408504b7551d9c574246ed4738e7c39499ffcc775a2220 freedesktop-sdk.bst:base/help2man.bst 
      cached 4e27b7dce693baf7af90f50be716e9fc67d7893c09422aa617ea6f92226ae2c9 freedesktop-sdk.bst:base/texinfo.bst 
      cached d8bd59eac21c3f09d146de8b4c69f9dfe2ce8aa473a13aa052486a3cf1f88c70 freedesktop-sdk.bst:base/bison.bst 
      cached 6183b72a4edb141e88d5eadbda18bf563b10368f21a41b2feb7b50210d280e60 freedesktop-sdk.bst:base/pkg-config.bst 
      cached ed0975036fde4766540fb515f4c3f340e37a9cd723bcc6fc132dd26ed2d247b4 freedesktop-sdk.bst:base/util-linux.bst 
      cached a8a27d36431a10a9c86fc2bc42f43688058ca1b16b4a6eca6c7366b3e5330981 freedesktop-sdk.bst:base/e2fsprogs.bst 
      cached 08410f7a25ceaf2f650c44c29c99914ccda2d1d30abb4a530bbdaea578781810 freedesktop-sdk.bst:base/krb5.bst 
      cached 1c050b99f1bd60ba98851e439647460f84892b6bb2a82eec2269f05a223ab4e6 freedesktop-sdk.bst:base/openssl.bst 
      cached 0aec76fbfbc6c10bedfa684ff188eb06f253d5739cbe50c09bee4e066bfd9161 freedesktop-sdk.bst:base/expat.bst 
      cached c86a8dfeec0781fd5af82524cb695190a49b033952fbc2f3a3477e1bb41fd5e1 freedesktop-sdk.bst:base/gdbm.bst 
      cached 9399e38357c8541cccf74ee63db583276df77123a207b041cb80d49b1a870c70 freedesktop-sdk.bst:base/libffi.bst 
      cached 9fe57c0a2c614569e4ac8a92544d01a5cbdb21a26a06595dcbd29b8e58ebb956 freedesktop-sdk.bst:base/sqlite.bst 
      cached 92237aceecb3bcfa592aaa7d7edb7c47cba714b8ca91de354371d4f4093cd333 freedesktop-sdk.bst:base/xz.bst 
      cached f9b3e7e413fe1535ec8c92e760939bdc43069b45a19104348eea457a50354bed freedesktop-sdk.bst:base/python3.bst 
      cached e051f1972b115dba0f83d65d7d5b46cb6c43fa722808034823b546ac7251c07b freedesktop-sdk.bst:base/ca-certificates.bst 
      cached 67800c32ac7db431298030fe4aac5ae27036d2c4fc2b161c12ff0e4ec01de636 freedesktop-sdk.bst:base/cracklib.bst 
      cached c0de3eb2e5c6204d1a6fc1636a82aa12d56c136c0d07081b5535408557682782 freedesktop-sdk.bst:base/libidn2.bst 
      cached 462ccbca6c62d6b151850d175adb40e214ffdc75c7f1d908215298f157b8b470 freedesktop-sdk.bst:base/libtasn1.bst 
      cached e49151bfd9943c64f9fd6efe8bc717eff289e959f98df6b9f30a7eea2a4938ae freedesktop-sdk.bst:base/libunistring.bst 
      cached cca8bbe05d5d035fb4ebb82c7f6dafb0994dbcbd9cb6df0b431373b426bed77d freedesktop-sdk.bst:base/gmp.bst 
      cached d0a4c5689202c3d1fe24df3b07b4b93a207ccd108dce132403105e838e3fe2e9 freedesktop-sdk.bst:base/nettle.bst 
      cached 522cefecfc8e725230973887844f7a8199f6cbf9790194487404c7aaa9656e48 freedesktop-sdk.bst:base/gzip.bst 
      cached 8a5472803734f0fa2b8f5c147d2c8178bf463ad17e81683a1177f3f86aa20558 freedesktop-sdk.bst:base/tar.bst 
      cached 778e213a321e7df69503d766a6486c5973b651a8bbf1bd6c3d9f64f0e29ba851 freedesktop-sdk.bst:base/gettext.bst 
      cached 77b7818679cec182e95ca6fee7e7627337b3035546b62ed4245030d44b8a70a5 freedesktop-sdk.bst:base/which.bst 
      cached 097e84c1dc419beddd5dd4cd32860feb3b070407c7d61ddf836e8d8d4d812d51 freedesktop-sdk.bst:base/p11-kit.bst 
      cached f9bcfad287f437d53e47c763556de42d36be188ea1f01a488c32d6d16b6f8133 freedesktop-sdk.bst:base/gnutls.bst 
      cached 7b517ea98e1cece46b469e717051ca3cea2fd11b0f79a37d33ee5d6799ea38f0 freedesktop-sdk.bst:base/nspr.bst 
      cached 19d89da4f1a7d7f8a93aa1da82eeeede14c0e0b50750ae2300a00709bc32fd48 freedesktop-sdk.bst:base/nss.bst 
      cached bc2d3cbc88f3b461e995410852e17d448e2fea662f0eb906bdea76f94b5457b3 freedesktop-sdk.bst:base/curl.bst 
      cached 4a3c5970c96d1a8a4d12397a1b1ef29d80a526f2e449f6e1a2cff52d3a86be89 freedesktop-sdk.bst:base/groff.bst 
      cached c0b3a48e14cda870bd1279aa0efc459312c8992b1ec593781c736bbd63d69922 freedesktop-sdk.bst:base/cyrus-sasl.bst 
      cached e4b72a0c10c0861faf054d906651d0151088e51ab46df5cc76631c7d77fe2776 freedesktop-sdk.bst:base/flex.bst 
      cached dedc449e7ce123b0faa763682155da305d30ea21fad2242eee9a13cd6996b93b freedesktop-sdk.bst:base/giflib.bst 
      cached 31bdaad1945863d060a6358afe04d2a345210ce833c3f6e0887077ad1c834469 freedesktop-sdk.bst:base/pcre.bst 
      cached 1a76e1fa5840deb9bc3302a217188c29d5907e7286e6010ea9fa8ea9ee0423cb freedesktop-sdk.bst:base/glib.bst 
      cached 865849a6bec97bee2f9a62034e38e85790f84e5240206ccca430805762e0e44c freedesktop-sdk.bst:base/libgpg-error.bst 
      cached e8630eebc1c39d7e192fd9a1d0825f177611392112c3d4f0f87197d7931b577e freedesktop-sdk.bst:base/libassuan.bst 
      cached 79d2f1c4d394f38653f16d489e0abce762db58755c05fe02e7f5dd790f10d5aa freedesktop-sdk.bst:base/libgcrypt.bst 
      cached 5048bf16c1acd7a82675f2b88a4ecd915c61e9781fccb298e181bee2acff2bd1 freedesktop-sdk.bst:base/libksba.bst 
      cached 6d6588ae1e5c87133e28246544dc1f5be6c74de76a03024a4df2fa7d4ec703d8 freedesktop-sdk.bst:base/npth.bst 
      cached ebc57d8a39889dd77377568c24460580c7e7daf44cd3b5afe20bf0b613d19a77 freedesktop-sdk.bst:base/gnupg.bst 
      cached 73a043a04da89b753b73888cabbd696e075158ceb51fa4867b0f8a19bf1a2d59 freedesktop-sdk.bst:base/gpgme.bst 
      cached f35287caa8bf9e8a9a32a2e938cb51551d0624051b35753d306e9877dbd719be freedesktop-sdk.bst:base/iana-config.bst 
      cached 6444abf83ae0794d5c301d15d3aab0825a8fccf0431c153cae3b1e0f94454c24 freedesktop-sdk.bst:base/icu.bst 
      cached 4062167939fbc3bc046c7256e8bf4076d9727e7a04631801f3824c9cde01c420 freedesktop-sdk.bst:base/iso-codes.bst 
      cached 54f3782b028c6272de7d3da740e85efe9f744a30140907c436748c6871727bcc freedesktop-sdk.bst:base/nasm.bst 
      cached 0e921ae5695f8432c57bf6e0828757c923c43dad8a1af2b18e996cb687b8c765 freedesktop-sdk.bst:base/jpeg.bst 
      cached c29ae75cb9c8a48ac9549971a65c16de509bd39a27bc65783f0b5238e8b8323c freedesktop-sdk.bst:base/libtiff.bst 
      cached bfa0148e28f2994684552078d245e209798ddeea28900545e16ffc2577648289 freedesktop-sdk.bst:base/lcms.bst 
      cached ca4ce25dba82cdf6b2cfd2b211a1019234aebee0adf8578dbe6dc3566dbd2ac5 freedesktop-sdk.bst:base/less.bst 
      cached 2aeef7d2a281dbb25f22b32bce5d5c9a1053dfa06872e24062b3e1492326849d freedesktop-sdk.bst:base/libarchive.bst 
      cached c78dd8873b62375f088a7285c82771d0b1ffc4d02810b45ca5cd9a35f0569685 freedesktop-sdk.bst:base/libcap.bst 
      cached 1f49cab4659238ae56bf5f3386a765bceb6193d23f37977458d87ce87fe12b36 freedesktop-sdk.bst:base/libexif.bst 
      cached a782c24576814746a29b54e9a007a23a9435a12b0909ddc08e13967077352fec freedesktop-sdk.bst:base/libogg.bst 
      cached aee62462df84060ae284a03cc2b336cdd4e14802a0afee581c5339e9185f2ec8 freedesktop-sdk.bst:base/libflac.bst 
      cached 002b295319f88749d308fb2daf248d58b14b79370cc7aeb941dc6ad4da930b9f freedesktop-sdk.bst:base/libpng.bst 
      cached ebabb3743bdfd6121581c9d8000e04c5c3dc5f61ca4b433ecdaab9810bddfb58 freedesktop-sdk.bst:base/sndfile.bst 
      cached 44d7186bdd2e3d189c1623b74d59c22653ba8be85af36c9ae42505e3d400ae9c freedesktop-sdk.bst:base/libsamplerate.bst 
      cached d4a7918b9fddde97cae8ce9cee5561a61b816fbcc5331f186f59487836a2789a freedesktop-sdk.bst:base/libsepol.bst 
      cached c83410119970bbe255faffd56f3e22744d4e00f226d473d87f65d13eb65d3237 freedesktop-sdk.bst:base/pcre2.bst 
      cached 980e5c24e229440438c06a1326fb7395526078958e078bfa42ca37ee2a8472ea freedesktop-sdk.bst:base/libselinux.bst 
      cached 603927af0d03996783c5b542a6c7485b3145b769d6457343417d964f1bc1c1c9 freedesktop-sdk.bst:base/libtheora.bst 
      cached adeb8745cf03d4626f824315991cdf65eeaf86759fc61393c0983b77a5e8e778 freedesktop-sdk.bst:base/libvorbis.bst 
      cached d7991374788132a8bdc912b75704866a988a9ad87f5846c7d6c1eb50d0313cee freedesktop-sdk.bst:base/libvpx.bst 
      cached 8de267f02c086b389532cb29a037ed1fc725d47d0aca70e00e44342bf3f01642 freedesktop-sdk.bst:base/libwebp.bst 
      cached c9363034c54e1ae95eba09ba9ee26e354a1c36958763ab869f893989b0d006e0 freedesktop-sdk.bst:base/libxml2.bst 
      cached 477b0f9467794ec83d07c1319d2ca320a0af7bc83d25826dd1ae7ada0fbd0ad6 freedesktop-sdk.bst:base/libxslt.bst 
      cached cd6cef89bec28f2be6dfb000fff869330a4d666004fd3885dfdf2d13d6bbbef7 freedesktop-sdk.bst:base/linux-pam.bst 
     waiting e2d060f86b43641be587180629d43ec81590ea1423f3263292491bec8d5ee659 freedesktop-sdk.bst:base/supported-locales.bst 
      cached 4fd2d5aec5261a04e64729675fb08f36942f63863b8090cfc9e71461ef9a78ea freedesktop-sdk.bst:base/locales.bst 
      cached c59f6fc60167db9dc303569c7f71d5600092e54f114380533e84223d70a9beea freedesktop-sdk.bst:base/lzip.bst 
      cached c8708b71fa3e49058c524019ab62f70f4577e93566dece30b00801cfd4cbee0e freedesktop-sdk.bst:base/nano.bst 
      cached bf1a79d949277cdf98bae5118ff5a561c79b008fb3e9cb9e9842a631a9fae709 freedesktop-sdk.bst:base/python3-six.bst 
      cached 7fc43f603252660e9e26962065fdc8e43b3718f992fdc069cc98b00ce70baf8c freedesktop-sdk.bst:base/python3-usercustomize.bst 
      cached 0b7913a378c157a696a75964784882a073f2e8fa7def41e03a51f8449ca8298b freedesktop-sdk.bst:base/speex.bst 
      cached ebc6531710caf5a39ce329b346f495d573e828ab7914cfe190f70c07a50c1e2e freedesktop-sdk.bst:base/speexdsp.bst 
      cached 7a9034dfc89be1b95b727d63d60a807b2f29122098e36d0c2bde6b819adaf409 freedesktop-sdk.bst:base/tzdata.bst 
      cached ceec1717251e52c781fb7c06ee84beffbf1ff3e3624796e0397ff4f77f9a538b freedesktop-sdk.bst:base/unzip.bst 
      cached f71b95e6523f09038c17494c9cccb4f2642c3a291c032b762bfa1cfd71a324c9 freedesktop-sdk.bst:base/zip.bst 
      cached 501034814fe306fc58457146f1661fb18646d002b8fe4b6a4fd595a8ffce802d freedesktop-sdk.bst:base-platform.bst 
      cached b544d8f6d74df11854921993eaaefc5cb20dd6f06cd4b4aef8f9a46af4df8082 freedesktop-sdk.bst:base/autoconf-archive.bst 
      cached eeadb3fa87c0905adf15609cb632aa2ebee364fcc69bf6c9453b193fef0bebd1 freedesktop-sdk.bst:base/ed.bst 
      cached 6ed5f1b64d2afc5a03aec8869abf87f9123cd8e4e527cd757c473fe338db4459 freedesktop-sdk.bst:base/bc.bst 
      cached faf793e7a119eb35a72e350fca1348247f2affe2f2919b5920a894c34d49102e freedesktop-sdk.bst:base/ccache.bst 
      cached 02f518c51230548ddcd5dba9da1e0d9cc45389d07089c85de3b0f6b6e16df92e freedesktop-sdk.bst:base/cmake.bst 
      cached a41b02d650a74c1799ee0cb2bdf2a4467f153a588f485d24d27b8e2937f35053 freedesktop-sdk.bst:base/clean-xml-catalog.bst 
      cached 3577d8e052f6d744f65896d0b0df3e975b17645f7082979c873c4da4ea052320 freedesktop-sdk.bst:base/docbook-xml.bst 
      cached de2cf4364fbee7c53a7212dd4f865b2766858c1dac4529bce84acb5ddd5ce140 freedesktop-sdk.bst:base/docbook-xsl.bst 
      cached a10b418c383adb3e65c76ee0f9ecbbfea6bb001032b9f0dfd4f14078468f6ad1 freedesktop-sdk.bst:base/gdb.bst 
      cached 9e86d96c878c6a290ff818a9ed3bb332daf05651e07af4afbff598242a43830e freedesktop-sdk.bst:base/git.bst 
      cached 5b2349b70de151bba5f23fbe5b24397b483affd785174b8698ef5e2617298e8e freedesktop-sdk.bst:base/gperf.bst 
      cached 178fa904fedf4819c9473250df1ec1aeb1b874114acf130f24a71b40aef71d7a freedesktop-sdk.bst:base/perl-xml-parser.bst 
      cached 516ba7a62f92827efcfe83d0dc56224b44e966cf97e94a6c1108d0877dff8638 freedesktop-sdk.bst:base/intltool.bst 
      cached 8f1cd5c4c5736febdc863f9b14b02c62ef2d1a7548457f5b42319c83280f1396 freedesktop-sdk.bst:base/libpipeline.bst 
      cached 32b43227776bc5ac4ed664be151c0a7b84b6e7194dd9b23df83c60b28cc20459 freedesktop-sdk.bst:base/man-db.bst 
      cached a43a6469f4c07b17c0c3f74fcf61a5afc18693ccf6071e7f872bf881e2d987a6 freedesktop-sdk.bst:base/man-pages-posix.bst 
      cached f4eb33e797285ff8698a1da0472043330adce18da419c5b684080518bc561b79 freedesktop-sdk.bst:base/man-pages.bst 
      cached cfe5538136f7c052d8f4f49dd984e5ca35f12e63e3f3076a0a55196d63133299 freedesktop-sdk.bst:base/meson.bst 
      cached 1abc5243d11123ceb910adcb40863ca4c43140079d91b8cae02e846a454c5ec8 freedesktop-sdk.bst:base/ninja.bst 
      cached 170874c11ca6c5ef52b4cf54d3a5d2098163d959425d73aa9e38d89722fc34ba freedesktop-sdk.bst:base/openssh.bst 
      cached 3ee9ca559d57ba64167e75aa4af6cfcfa570d7a7c29a506253896e954e7e8d42 freedesktop-sdk.bst:base/patch.bst 
      cached 3b64105c6c9d629a52fc2b57183076de251802fe46d3e3ec548efe7b99ac2e12 freedesktop-sdk.bst:base/python2.bst 
      cached 49c80d3258d6df8410a9cf0329ff6b0ddb0655fb09ee290f9e48be91367bb458 freedesktop-sdk.bst:base/python2-docutils.bst 
      cached e357e8a2bd8ec34493818a6219ba81f9873ff6e983f91138f0de1d0bddd96c08 freedesktop-sdk.bst:base/python2-mako.bst 
      cached ddf1ff40abcada1f1c9dfdc9f1faa2c66599d561ce8f0bf4861025ce265921a1 freedesktop-sdk.bst:base/python2-usercustomize.bst 
      cached 91a98cd7abb9b71519092c5cd5a48af9e7486aa76206a32f786c443601c0110a freedesktop-sdk.bst:base/python3-mako.bst 
      cached d8c73fd582e532b033553fb6fcdad48bba9ff6e778758bd9e93f4df8decbd0b2 freedesktop-sdk.bst:base/ruby.bst 
      cached 76a871918481417ac6f749617e9ac9620d4daca1781cf0c69c7da5d2ec539415 freedesktop-sdk.bst:base/strace.bst 
      cached 3ebf09f7805c19ca673023100d047aa05928ed6a1d4ef4b23e1196106cbdc534 freedesktop-sdk.bst:base/valgrind.bst 
      cached 2222fb7223a02de848e0d5f847c4435508c227ab881273e27859cda24f83e4a4 freedesktop-sdk.bst:base.bst 
      cached d932c54fb04a1ab5c33e42571b6aadd802c65a5397a04e7bae6d6a6d40e54fa4 core-deps/iptables.bst 
      cached bcb2ae9e3d8bfdb129887009cf23bf499691972c9446fa3554104071273aebb6 freedesktop-sdk.bst:desktop/itstools.bst 
      cached d669ca4e46eb05214e8c17f3ee7b541282a023a388da965caaf7ee3a2761e6d1 sdk/yelp-xsl.bst 
      cached c0b7f6a0ebf60744bf4df9a4e240870214cd84f3a388c1a6d18c11e463c7cfd5 sdk/yelp-tools.bst 
      cached 5d2630ce7141fb40d80ae2558dd5cc0e7f2b7821e0789fc1d7bbea514dcc1f1c sdk/gtk-doc.bst 
      cached d3e6654551c8956be505c4a3d9febd3e0e48a62a621bd16a0ac24983a3180e2c sdk/glib.bst 
      cached a8770dc8b0405ca1626277dcaacc0c019a45e9bb5eaea1f5d1503b50357e4020 sdk/gnome-common.bst 
      cached ba0f2ca8d30e4276238572b26045adafd6d8f4b77108968d7e14525485f9fc26 sdk/gobject-introspection.bst 
      cached 3f360c1c8a2e1cf08331c95ecea8086ed4315f47075bab056f9004cdd642f5d2 freedesktop-sdk.bst:vm/linux-vm-boot/systemd.bst 
      cached b0d5e705e799dba331fccb1a43d3f04409502a4c9a6052767604485471b3f782 core-deps/libgudev.bst 
      cached c11ced0d9c519b24a87ec805313f023c7ea536fdb6297e27e5b7c253968d88bf core-deps/libndp.bst 
      cached cc9543e0988d46de319fd737f7a76445894dbbff8d78c3d04802578c3476af07 core-deps/ppp.bst 
      cached ee0e389c474c2b950db66e8cff9d58f0597f01d221d4eec03973f56a9b25ae1c freedesktop-sdk.bst:desktop/dbus.bst 
      cached 9b4aa79bab246fc085e4c38e2f46ec2e6b74751aaab945ae7cc233960edbe7b5 sdk/dbus-glib.bst 
     waiting 7eb307c0aef3c04e80eb24749a62df679b6bbed7ee9c0e06a64ddb6b42acbc7d freedesktop-sdk.bst:desktop/gtk-doc.bst 
      cached 0ae5d196a4fd6dcaed47ac4204d6836aa26f7252df5ca3a29c223357faec7dbf freedesktop-sdk.bst:desktop/pixman.bst 
      cached ec7dd7f34e10599719f07ea5b7edef6b532232f7a9375f563a1b7bfdf1e9557a freedesktop-sdk.bst:desktop/xorg-proto-xcb.bst 
     waiting 587e2637beea056d23074457a0bfebfeee1f3ac90f4b10b9458295dac0f4a778 freedesktop-sdk.bst:desktop/xorg-util-macros.bst 
      cached 3b4d5fa879260d4304ff2efc3142ec8b382691e726973c955e5d8a1f05a16cf2 freedesktop-sdk.bst:desktop/xorg-proto-xorgproto.bst 
      cached e42c116376fa6e0689458bf4475ef1a2fd46789b2d50407375231d6516f30a59 freedesktop-sdk.bst:desktop/xorg-lib-xau.bst 
      cached 4af84285fbb45238d88755ad70da84f1c15100be56a00d9b207efa57c80c99ed freedesktop-sdk.bst:desktop/xorg-lib-xcb.bst 
      cached 8451f03d76dc2f1e25d0c2c7107cfb66eb8049226a8cc8aad3155032d37fac63 freedesktop-sdk.bst:desktop/xorg-lib-xtrans.bst 
      cached 5bbb2990702f4ecf8a00bc0c710ac9565e31d88e86d10486b006a830a098c753 freedesktop-sdk.bst:desktop/xorg-lib-x11.bst 
      cached dd6029945316a5a6275c4077ad6dbaf0f05a0c72ff1afa7c3cf66931eea9668c freedesktop-sdk.bst:desktop/xorg-lib-xext.bst 
     waiting 6df7f02ac76c4e6687fde128264357866e54cacc0a4c5b0e5cb778d4016f868c freedesktop-sdk.bst:desktop/freetype-stage1.bst 
      cached 6a25ebdecd3a5d8dba279be768d520482cd9d273a44232d14aa80e649277d8c4 freedesktop-sdk.bst:desktop/wayland.bst 
      cached ea33f7c4db2748814ef429e54e47d1725cd485a4ac2fced6360394da80ad9d87 freedesktop-sdk.bst:desktop/xorg-lib-xrender.bst 
      cached d77d925d01ec64e734bf121fefc7eedfa5ae5818dd9a4f8172814bcbefc30695 freedesktop-sdk.bst:desktop/xorg-lib-xrandr.bst 
      cached 2e607085f90ee56c02efb0b1aca8b0a4ad34c3423ad53f277ab68fd1d37a9d4b freedesktop-sdk.bst:desktop/graphite2.bst 
      cached d3a3bf6ca217dd53d1a3a0358f81cfa46cb26782191cca98b058167df0c67acb freedesktop-sdk.bst:desktop/harfbuzz.bst 
      cached 1f406d024f0f8153737d4e9f1e251c888fd0cf2de3716a173a9549cc6e6614cb freedesktop-sdk.bst:desktop/freetype.bst 
      cached fb16c0bb9740cc524d147d1e8f0dbdf67d2d61e1920f1d9863d9dc17ffaa3313 freedesktop-sdk.bst:desktop/fontconfig.bst 
      cached a21b604bcd6d1bb784184369138bb7f0736f073a309cad4346fc8a07f32755ef freedesktop-sdk.bst:desktop/cairo.bst 
      cached a736e8dabeeb444a0a7af0cb8c9bc003a1fc97c70ed7c12d6acefe0462c256da sdk/pycairo.bst 
      cached 814c2de9dc9e002f318e08f735a6d4b157a5c2a788e8b41c38a2ca975a8ad55b sdk/pygobject.bst 
     waiting 4ac87f8ec5b491aefe47433ef2524539e3e1c68baf592b40568cfc885937f9fb sdk/vala-tarball.bst 
      cached 015fb84a1e8f41f5d263bf1998b94ced10bb2b02cb06249fbad3e9c844148b04 sdk/vala.bst 
      cached 0224e55aad6e2dad2e409f14e5581df95f66f42f33366b1b51336e87a6fe918d core-deps/NetworkManager.bst 
      cached 392a6fc0e78e05e12c30c2e6c7be800baaec17b71fde2eb0e38b1b24006ffcb4 core-deps/mozjs52.bst 
      cached 2f7c0c3a9174f088f18d62727422c27749ed1f582e39087cf1d5b60904c42e92 core-deps/polkit.bst 
      cached 9c95d31594fca6f17309df5e6c0d6672fd13d22604f5e3d84730a456b4eb3b8d freedesktop-sdk.bst:desktop/xorg-lib-xfixes.bst 
      cached 5455f802a24e2bd85181b49749f53d9ea8e791d712e01dac2a75de3550dcfcde freedesktop-sdk.bst:desktop/xorg-lib-xdamage.bst 
      cached 5d199da69859867dd592eabe1b321631349f089385cefb2d30fd8065a46cefc1 freedesktop-sdk.bst:desktop/xorg-lib-pciaccess.bst 
      cached 917350fad9b8e12ce5edfdf3d08973c47529bffdad8eadba453fe38f87a393fb freedesktop-sdk.bst:desktop/libdrm.bst 
      cached 331e001dbcbc8f08f172cd644acbf57d1fbf3524ea99851e1ecb32d9fecd2308 freedesktop-sdk.bst:desktop/libglvnd.bst 
      cached eb2c8932ed0eaeaadaef32f0d41f226879cc02226cafc89a2525f878659e11c6 freedesktop-sdk.bst:desktop/wayland-protocols.bst 
      cached ffd4be2c74be21d17de1f1b64125d2aa11ec1ac37933bf1de30982950eb152bc freedesktop-sdk.bst:desktop/libva.bst 
      cached ee95e5dc0d408965c73caa3f5a9aee149fac31480e7e2613d9f3d067071c18bf freedesktop-sdk.bst:desktop/libvdpau.bst 
      cached 02d1eb573db90116f707f6f5d90aec5f81bd0b6e379917316b38278bac41e1d0 freedesktop-sdk.bst:desktop/llvm6.bst 
      cached 1a07207c8a8b94382f50057114f4fa63a8f6c0241a35ccf7b161686da803573b freedesktop-sdk.bst:desktop/xorg-lib-xshmfence.bst 
      cached 8482b6fe4b0230e25892cf0ce1bd710ce5a0e29d408c1016b927f6c4fd6320f2 freedesktop-sdk.bst:desktop/xorg-lib-xxf86vm.bst 
      cached 663440178a267530555acd0627a1445c64474b6f85bf7ad8c4bd6bcb01bfe8a1 freedesktop-sdk.bst:desktop/mesa.bst 
      cached 3e2fc1ae94656c3bd61fa1791e7a8b3a0eeac56a05c141a16650b7ac1b430b07 freedesktop-sdk.bst:desktop/libepoxy.bst 
      cached 94e55ed05884418ff8cfb0ff623b9ced5cf50b08c7d33653e77933692a27b1c1 freedesktop-sdk.bst:desktop/xorg-lib-xkbfile.bst 
      cached a79d81198f782fe439436642fb637ce8dbf890d6a6e368f4b833bbaad9cadbf8 freedesktop-sdk.bst:desktop/xorg-app-xkbcomp.bst 
      cached 6689e854e616fa4a87d1173ddc0417da1c2426bc4feb976514e41e73f30c61f6 freedesktop-sdk.bst:desktop/xorg-data-xkeyboard-config.bst 
      cached 1ec6507726b86339deafc11d1da86a3385dc9108950283d106c715cac75db3af freedesktop-sdk.bst:desktop/libxkbcommon.bst 
      cached 7615115832e8512828569fa6969e2d20320d6fa7312d7e79356cef5872907686 freedesktop-sdk.bst:desktop/xorg-lib-xcomposite.bst 
      cached cc6c6d5d26e622a87c093d6617b94e57e01fd9fd2d652cb7e6330baada6df5af freedesktop-sdk.bst:desktop/xorg-lib-xi.bst 
      cached 6dc0631558d092c86a19fd8fa85a0b91a6063f77fae52018b36efb0064f973b0 freedesktop-sdk.bst:desktop/xorg-lib-xinerama.bst 
      cached d9ec03a32547191267c149e81351e2bd695f8452f12be1082f1c5b018b85b1dd sdk/atk.bst 
      cached 3681a1247468b9637b8811fddb07e801a1f91bf739ff09dc1a62ddc5beb75f21 sdk/at-spi2-core.bst 
      cached 7b21ba1ec33320afc31ca0c95f3da30fd33588ab93a36bc941dde4cf27aaa5a9 sdk/at-spi2-atk.bst 
      cached 11f7b360ed51a68a4c19d58f2c545435d8231bdcd5e16a890b2613cf59d358a0 freedesktop-sdk.bst:desktop/shared-mime-info.bst 
      cached e1144d410e4104ee4af4a9d06d4b70abe69bd832ef03297c86434da8861193ba sdk/gdk-pixbuf.bst 
      cached 6b48d6a7aa131ac17801a4ffe1d6abf93a160a9c2c52fef24d89170b3f503b32 freedesktop-sdk.bst:desktop/fribidi.bst 
      cached 138fe3ceb7f4047aa10ac947203827c25f9ef17a58765dab52cda5a1c9d5b83d sdk/pango.bst 
      cached 25c99e27501f6a54befcf73720e53c57189831d68a45bc2797b38ba6a1e60096 sdk/gtk+-3.bst 
      cached 84ca0f9f3e105a12f8e5a86c2a1db13d522c059cf0bfae7fd43c101ede8a8fa5 core-deps/PackageKit.bst 
      cached e334c050d74f83ee297eca6262292a0526eb61712383f797d32facb19adcee1d core-deps/accountsservice.bst 
      cached 1da4a7e0e87db3b1147521ea6a249beff04e28989997980071f693763a832c77 core-deps/libusb.bst 
      cached 7cbb3fcaa387ecfcaec7085cf29342ff9a04bcd100667514454473afeb3b3c35 core-deps/upower.bst 
     waiting 839fb1c08413927ace1de52d00d06d2f4d655cbc80c38c7d6df0c7d102c099e3 core/meta-gnome-core-os-services.bst 
      cached b206b69ead87f5479a39f812a2faadba0c3be5f28806b966e1ad17d9c53b5a38 sdk/dconf.bst 
      cached e62a8bb54f0bef67503d524bc8f1c99d2787e1a0e79d3fc5a2b5b813cd7de324 freedesktop-sdk.bst:desktop/xorg-lib-ice.bst 
      cached ad1f4360f9180d821bbd7d2f9ee7978b65adf2d180e089bc02ff82f21e54ff57 freedesktop-sdk.bst:desktop/xorg-lib-sm.bst 
      cached 7d1696a4a3f50f10bcc0c456ccb543ae7bfb9e96c199569eda0c2bb8e32e27d8 freedesktop-sdk.bst:desktop/xorg-lib-xtst.bst 
      cached 9292c526b4cf57dde24dce990521d94064818c403fd3b117bb6509a6f4c5db50 freedesktop-sdk.bst:desktop/libpulse.bst 
      cached 901c99cb9e4f5c27aec21614bc5a04583b0fcdc3a3ab9c06a6b27a40f82d2186 sdk/libcanberra.bst 
      cached d6f644226f30be9322a0bc42d205452521bed20366f2c6d04a33ecb455b00a0b core/gdm.bst 
      cached 2701f46eaabc840d8dd81a36d8865bbcba2ca67a4bd2775fbd0b107390a2b08e core/gnome-backgrounds.bst 
      cached 711bdc3da64d79fe206ecfa81f4fa715be9563887358f56d895f9926e0e38d6b sdk/libnotify.bst 
      cached f6aa8e5e2acd10344a192ce138373aca0fb11365c055d462c14dee9fedf868e4 core/gnome-bluetooth.bst 
      cached bb68293e29d37b34ae7062a6a1f7b424d532708ab3fc0131d0244b18af0dac6c sdk/gsettings-desktop-schemas.bst 
      cached a9f1462d57daaa7eae912890d73fa4cd631e776b69f2a67b3455026cc4932ef2 freedesktop-sdk.bst:desktop/libseccomp.bst 
      cached 426ba4088985dd1ac6a0e5cfe7a18203727903699ba457844610653d8ab82755 core/gnome-desktop.bst 
     waiting e177267adc26690bb9f8d7f59d40e5481f52673d68da217715eb3e3c54ef0bdd core-deps/ftjam.bst 
     waiting 13cdc3fee9f10934ef554d030bbdc1dd23393b541b68505efc4e3378936b09c3 freedesktop-sdk.bst:desktop/xorg-lib-xdmcp.bst 
     waiting b98666ebfc7173e3c80627ffc6865dc97538564f8481756ba674e273a0a4f1d8 freedesktop-sdk.bst:desktop/xorg-lib-xscrnsaver.bst 
     waiting f33aa6f5174098111c61a098ea0c0b9929c5829803630ca97d5820733db68de7 core-deps/argyll.bst 
      cached 2631b4792e8ee81eac1de99bed21f58547434d1f9fe6f2e828feba0d267f6bf9 core-deps/libgusb.bst 
     waiting 1e3fb7a7dca97f50f0be4ab74fa629b366323bdc50ebea371de332f35730e4b3 core-deps/colord.bst 
     waiting 659105d17d842132757a543c1691de770c2907d11d09013aa57ffd6fe5a8ce21 core-deps/colord-gtk.bst 
      cached 42b613054412a3e74ea6997f39802a1d244ed0fb3a2718b47092a44013b777d3 core-deps/exiv2.bst 
     waiting 38ead40e58a13c856fb8a7e27d7f02f44d4e35751d5eae15e2b3df70597fefaf freedesktop-sdk.bst:extensions/rust/rust-stage1.bst 
     waiting 2bbd5fa043c824b64d3c9e5cdf165262a646c73251340ae63bd42e503c0b2194 freedesktop-sdk.bst:extensions/rust/rust.bst 
     waiting ab9e816651e9d55d57d483e9ed5defd4b01b987d0bc17ace289a99879430dd9e freedesktop-sdk.bst:extensions/rust/cargo-stage1.bst 
     waiting b22696e88e21d6c06ec0b2ef5847e0359d558cb0d57cd77bb285808bf8e09f3c freedesktop-sdk.bst:extensions/rust/cargo.bst 
      cached 7d138e2677aded4e7682eadcde64f93937e849f5fb94c009efa9518a6c7ecf6f sdk/libcroco.bst 
      cached 7f7297bdefc71d4e43479c6f240f09478ea7970f1bed8ff02b979b408b3063ee sdk/librsvg.bst 
      cached 03140e945ee346f557a939e2b2ba1522dc52b90ef267ec147f8d0e3ea96bc805 sdk/adwaita-icon-theme.bst 
      cached 4fd86668f36a4c4ed2014bfbefb9733e8ad3cf67c81c1afc131b26de4ea539d9 sdk/gcab.bst 
      cached 8c5425c28cfe651b207cf0b2df12f3f3028c6094e72eaf663df17502eadb97d2 sdk/json-glib.bst 
      cached e17d5304989ff069ff7363d69767ca54b5dda0262069367e914fd869fd3be5f2 sdk/libpsl.bst 
      cached 1000e7d7c62843cb46279517561dcd5d463b4cf40541540a2e5628a85b3b4e9b sdk/libsoup.bst 
      cached 80a58cee94976d8ed97d236b9638ac96f7b45b651afa4dfe71498b39b6b60fe9 sdk/appstream-glib.bst 
      cached 8ef87790d5e34f6667ceef7716efd0c450a0a462ba090aed235beb31cff10d8a sdk/vte.bst 
     waiting 501fdf97a0c29384db892010128d7c6450fd6f2900bcfb49ff1615d7ebcea629 core/gnome-color-manager.bst 
      cached 27ab3ef7edb1c092116b3d59028011823a853b58966422fe39ce1c1c36344683 core-deps/geocode-glib.bst 
      cached 8da7e199582d4a445d475f6815600343869bca86b930782c6ad7ffcc24d97342 core-deps/libgweather.bst 
      cached 04090738ee3cc732e48fa0c16d318198a6f62a57d214df822cbfae6781a7acf8 core-deps/libwacom.bst 
      cached 8334ee463d3971067d639f94663eb722a6cdf15fc548070e2ea18bdcbd4bd9bb freedesktop-sdk.bst:desktop/alsa-lib.bst 
      cached 6b7720465deb3b9b5a5c57deb48fa965017014185fd644bc11360d49ed187a5d freedesktop-sdk.bst:desktop/cups.bst 
      cached 0da5d4c276564eb3f3075157d2e4098a1ec75155c1bea67b3e58d796371f1491 sdk/geoclue.bst 
      cached 55d952cf91335cb59b9a80fb167c60b833735e0cae7e34138caae0f482ac15b0 sdk/ibus.bst 
     waiting 49cd10a42a45264a0a2b1890dda575e2893af037441b37fbcb0b2e8bfc42355f core/gnome-settings-daemon.bst 
      cached 3c0ecc0d52b0f46531689ee8ce679987109869a6ba114866310968bc91f18340 core-deps/libmbim.bst 
      cached 7ae5686bf35d7ed606290c78399f224297829b254f471f52f169d95ad7ebb8b2 core-deps/libqmi.bst 
      cached e56c7da76b2088eb6c70750a6ab9a70f7a05128657baf19a3fafc4bab3c91ba4 core-deps/ModemManager.bst 
      cached 89b03721e64d8ad8e3985c7dafd2d29dc7ac97e6746ffa3ddf55e67b04758843 core-deps/librest.bst 
      cached dec6d0f7c7e6df6196ead13aa9a0c65ec76ac0358c026a69515f51207ffee73a sdk/libsecret.bst 
      cached 46cbf9ba4261745192f57ff6de2fe28c3a33c46a4b71916dad8d8a8ebb5baef9 freedesktop-sdk.bst:desktop/hyphen.bst 
      cached 667a820630ec0a84873e070cb253b291aa8542047c8a2f4e321e988b4f617468 freedesktop-sdk.bst:desktop/xorg-lib-xt.bst 
      cached da794c1741055618304a5251905a18979bc4aef8c6583499de29c424b7aa1130 sdk/brotli.bst 
      cached 7ab326969515833b23bff712391c1ec2ee25ece7bce4856e3bf72dcd7cd4cdb9 freedesktop-sdk.bst:desktop/hunspell.bst 
      cached c11da8f540f7bd8ef7afae2514420f9d5da08aeedb3aa99ba99956ed64a157ea sdk/enchant-2.bst 
      cached 45c7084363f59792d3eb795b8b10868f55ae716ffb053dad01a7268008c39705 sdk/gstreamer.bst 
      cached e3865a376ac3253fd4a9f18ec2d632939317714b781bee8ab86e437cc33c3465 sdk/gst-plugins-base.bst 
      cached 43842af9b123e74c34e20461a54c57a912d4cb60706843cd4b99fcd75e513e17 sdk/gst-libav.bst 
      cached 39e0e685fa7228d5d01beb9123b7445395c04a331ca20ea696b0f45f561ac448 sdk/gst-plugins-bad.bst 
      cached a2cbc80c498404ffbbc220f3758d7608847e666ce52696686a843da2598f601c sdk/gst-plugins-good.bst 
      cached 1185d0ee75524b1dd6da7ac92d0fefec8dd3987152ea2e940b08ce705c5f0ec9 sdk/woff2.bst 
     waiting 765ee514d85a4813a54232ba6fa17a7dbd85d8d32af01a9fdd3e1ad4888b7ae8 sdk/WebKitGTK+.bst 
      cached a9aa21651ac7d146a1003cf18c2f32215385b91e1086b7ca0a3095535e2e43b7 core-deps/gnome-online-accounts.bst 
      cached 5c7efcd338cd22aa3f0435cadbfb850b713138c9f0620fe820a8d4ce43a5dd89 core-deps/gmime3.bst 
      cached a71dc773c1245a1f833ab77dfbc1f45cc09bbcf879143f97939a00c25afe390c core-deps/totem-pl-parser.bst 
     waiting f8b90ec61d9aca77dbabf91b164b005e6e04e0e8897bb8125efeba0105aa92d0 core-deps/grilo.bst 
      cached 6097a99cbbe91101f96fefc23e09190e19130e09c5713362bd9c96e891b99af5 core-deps/anthy.bst 
      cached 8d1b00f5c7168a42a1e4134cf3478477a2a657d9ae85e1711f3dae0fc4e29937 core-deps/ibus-anthy.bst 
      cached 58f66211ccd415adfd5bd9bec6c0ee00898d77910b2eb110b9f93fe4acff8312 core-deps/libhangul.bst 
      cached 4db4d95e99c623bac5b399a788f62453538ebeb7583efeec8b52ab3ad61ea166 core-deps/ibus-hangul.bst 
      cached 42763a2bd7b6b05b934b76784c5a1fe383cbc770e0fbc8f48664c78ebc3a3f2a core-deps/kyotocabinet.bst 
      cached 41cd9082c96a6a08a91122a75a2da4b9e6b4893f497e235bad88307de4794416 core-deps/libpinyin.bst 
      cached c1bfc39cecee2bfb1616321ef0a89378aad158888ec9a79c6f60938530d64ab3 core-deps/ibus-libpinyin.bst 
      cached 82bc3172741f101e2d17bace6a0869da9311b19482859117d4558b8c63585e56 core-deps/libxklavier.bst 
      cached 014d965167a2b5ed954c04eb2674bd88edc28a9bf45131a43c91dbfdcea21b3b core-deps/libgnomekbd.bst 
      cached f24cf43a26a44c789bf09f37b053299451ac65be6f5fc859c1c777ae4e9aad2d core-deps/libgtop.bst 
      cached 247982d3505e08c4f7477c73ec1641a043b07cf83449d4172281f319ebc77903 core-deps/pwquality.bst 
      cached 2228f52b5f2f9c97b674b6dcf647688fa27a48465946725dee6cf8ec5e6f21be core-deps/libtirpc.bst 
      cached 85f3cb0a18ca02df7b1f1a683647640db0917078fc3c0e89e7bd8f536e5b7787 core-deps/berkeleydb.bst 
      cached b319b42d8f8b348480e5c7d9be9fc8c4df6483568243d2fc95fccba306236d6f core-deps/openldap.bst 
      cached d0336de49c7b51cbf2d7330ff402c8cdd8a1707f6d1e3ffd5dd567aa30d9609d core-deps/rpcsvc-proto.bst 
     waiting e64319db25a0d20d55c5d2054656a8e1e525b9f44aa9f5c514dc97dc71eea0e4 core-deps/samba.bst 
      cached 75d381f60d1ffa6fa570dba8ab57bfaac8eae36b90ff8ba3ced566cf8ec0b7d1 core-deps/gnome-video-effects.bst 
      cached 9b80d65e1c548d30bbf8be9385154ac01a844a649a6f986d6d167935a2182af8 sdk/cogl.bst 
      cached 9e8daeb9472d33f5c5ddeb28ed6d425ff04cb3e39343290bf2e92b78e0c51411 sdk/clutter.bst 
      cached fcadd6128f0d0139758775f507d4577ccabcaa68c98a9e126ef509dda6ce0488 sdk/clutter-gst.bst 
      cached d1893e233ca80928418c295874f8c56b30a43c5bed509e6f4c810533d0b7e325 sdk/clutter-gtk.bst 
      cached 8d505e9125ec46eb9d7251645a0e791ffd180f9dc9f6fbf2dd56186fb9cb6d74 core/cheese.bst 
      cached e4dbf4a4aaec8a1159d8643bb5e92a6d32be67e3331a94b5e8478c5d732f2aa4 core-deps/mobile-broadband-provider-info.bst 
      cached f3866c10ae5f5c9c321a5257fd62cfbf0f60162087f2553c6a50366e6dff0ecd core/network-manager-applet.bst 
     waiting 8ee5ebdb375f935cbaa50467fb2234c9a4871a724b3293f9abe25911da289d18 core/gnome-control-center.bst 
     waiting 5d62838d89791c223dd82053169473405c007f04c8ee49aa48000f506c6deb05 core/gnome-getting-started-docs.bst 
      cached ad4e50357791ac36ab2c5841cd2dd6d22178df8df984c12dfe6334876fa44d62 sdk/gcr.bst 
      cached ccb8871e589ba36d9045a3fb7f2099d2ef90857428ca62e266810a97e01f72b2 core/gnome-keyring.bst 
     waiting 922ee71e9b4098c50c79c7104a18423e1e73fe80d5e1216bd03ef3bc2a5b3649 core/gnome-initial-setup.bst 
      cached d68da7a3e1cfb72a44a00cde753775b7c7da5bd32b32c35b5782fdbece02b212 core/gnome-menus.bst 
      cached fd7ed42132de55a5da243386c9c6f38c70cdaa67f83319b6b767a9c341d0b493 core-deps/xmlto.bst 
     waiting d2608b89a208335d18b1c9c9a4b9a3b57f69864122952b56b5a2ea79df7452b0 core/gnome-session.bst 
      cached 55f19572ad5faafebc636e9459b6788070614211a78b35d8570c46682a91d3d8 core-deps/libevdev.bst 
      cached 41947c6db091ee69035f5e6646f57acdf393bebff93148c266c8ade04bba0e0d core-deps/mtdev.bst 
      cached 05ae734d9da08c4b115105a381d0c0d7bed11e622a22cc73888fddd5b6f2f46c core-deps/libinput.bst 
      cached 48dd898f9f77a6b89b3d8bee77ddc9896a87af51d2c9e0f1209b99711e03d778 freedesktop-sdk.bst:desktop/xorg-lib-xcursor.bst 
      cached 420636b00544b49f334a6eb6e9c27b52edd7d6e3f56670db751bdfc6a7729564 sdk/zenity.bst 
      cached 6314b4b7a15f036c13b676553cea1b7313b5e8e4dced73c220156210c36281f6 core/mutter.bst 
      cached 2b3786e18412ab2f7390036c40b7fd68f2f0f43fd501307da643bf0a7307d63e core-deps/liboauth.bst 
      cached af9be20e6b21730c1261a44e53a217e1cee24b173c44afdbea0d7916acf1fe08 core-deps/uhttpmock.bst 
      cached 2dc3dc7575dc64270e74470747dd1ebabc084cbff2e690da1a5b8ac7e2523147 core-deps/libgdata.bst 
      cached c31fdce9c9cb56448202a05cc83d2872ab755b6625132b576a9a1936821ef844 core-deps/libical.bst 
     waiting 8df443785e53e4157f459d1439ca233d92fe6b41468eccdd23cecef227e19b5b core-deps/evolution-data-server.bst 
      cached bb8e046c868fb83ebdb14b09349a0520e3e593ee38a51dc953c355ec33e1ca44 core-deps/libsass.bst 
      cached d3d8ebf7472229bcb97f384705d8bc0dbf2ffc5fed8410e2f5385e5902f352f0 core-deps/sassc.bst 
      cached 69404e367af74211a166c69df08fd2ca512e5a18121ea48e98b1e4ed82410b1a freedesktop-sdk.bst:desktop/xcb-util.bst 
      cached 42d27ab8b214791eda4bb046d6353173363a34688b5e4872cf743bb22416cf22 core-deps/startup-notification.bst 
      cached 7bd550ff75ceeb0dfc0992005792a05ff63a3d790a0b0be763ece145c3e8f17c sdk/mozjs60.bst 
      cached 15133d5ae1ee8defc26eb941a22a66ec499d02e0cc990156fbddf30867d506a0 sdk/gjs.bst 
     waiting bc58e7b3fd9f6ae67028ce884a45306751062541e43faff006e676a00dbf792a core/gnome-shell.bst 
     waiting 83f56fd4219425972317785fe47043d63b187c3e8908c3afaee685b2a8bee733 core/gnome-shell-extensions.bst 
     waiting 33b4a3c5561e79d1035c49cff1f515a8d8bd21276598da9f0936c51add4f0a00 core/gnome-user-docs.bst 
      cached ac45f520a9d8d58f14a525df4094182f8927da92c855d2fd94e379785a11da67 core-deps/gexiv2.bst 
      cached 58b3144708526651e5482fdd2b66a7d8e67af738a69ff9423af50788ee2c5b68 core-deps/gnome-autoar.bst 
      cached 8fd95f138303a3a4cd3493552254e494193024c82737b521a451ab66208e9f60 core-deps/tracker.bst 
      cached 8619368dd956a4bb1ff02a44ea0653214534c3e3149a47e53727161df466a9a3 core-deps/libgrss.bst 
      cached b40b96d0a601b15761696ded96c0197e9495eb517a18dca001e65ce4bbf46a34 core-deps/libgsf.bst 
      cached 1ff730d63fc266e559f712d9a99bf8952a4f6da9d702edc8f4a3a296c881f800 core-deps/tracker-miners.bst 
      cached dac25c7127df57466b98840a7bb9e458a5de378c9e1eb47a8808fd5f8da3a35a core/nautilus.bst 
     waiting 2662899b9a2d29c27e522cf91a108e9e91f19d7612b59a397aeabbacc83c948a core/gnome-user-share.bst 
     waiting 1394db8ae0274bda1b1d6d37566296713d023be9bd261d5238f9fbe503537621 core/mousetweaks.bst 
      cached 5088dc3c38774ce9faaa749084821e649e1c82a2eeb1c0a81c09ae8bf3eb77db core-deps/pyatspi2.bst 
      cached 7f6117d248d970d5e186dd0c2ef0084fbc880919dbb232fe1249b7a51034dde5 core-deps/dotconf.bst 
      cached 3d05e0bb96cb9ecb17eb5a3e75c42461bcd6e941f5206e4905457ca6a4e32223 core-deps/portaudio.bst 
      cached fb3ed1344eab96f75a3c90998cb7188ea60c3b0b96cac461801a51b047a88bd0 core-deps/espeak.bst 
      cached 1606c363d1029090f87fa05ef2cc8201995e022021ddb84535682ee089327574 core-deps/speech-dispatcher.bst 
     waiting b9828bcc482296c468798bd58f83018f8c2b5e9160e92e6c2454f99a628e4954 core/orca.bst 
      cached 46d486a9e1d82a689306f881c4c8c952d115fe527fc508bcb19fbe66a9ec36d9 core-deps/gssdp.bst 
      cached 9b54cced813dadd4845b60563efad4c3f392e27f5ff5c78cfa6e9f3d1b0d13c8 core-deps/gupnp.bst 
      cached 8e802af7a251b59cabef386a3770e005c3bf744b091e8e4fcc83c6fe06ac99dc core-deps/gupnp-av.bst 
      cached 1899efa5b006fd941ed53a8f577b38477032949398080dd7facee39d626ea6be core-deps/gupnp-dlna.bst 
      cached 5dc8fab03fb84c4c9d5ac056dfd5e1c146f77c9e343cbeb8d0eabd621302963e core-deps/libgee.bst 
      cached 490c240142d0b833687d5e973bc2c096af61c2c5ba2879ada93b51b994351aae core-deps/libmediaart.bst 
     waiting ad80eb936ad5aeffed0e1ee8bb3706bb11cf149f99555a326694913ccf63a80d core/rygel.bst 
      cached 32c99673e9627b2f25343e701b8fcf357ccd03f7736115082526277f5bb3b649 core-deps/gtksourceview-3.bst 
      cached ba501ee00e06b442bab8d99c10c154b5154837d371a24c8e78a4460f3ab7a844 core-deps/neon.bst 
      cached 0a22aa207aa0ff45e21cd9a9d63346a61175cada673cecd5f0e677e9cc7e6e0e core-deps/libmusicbrainz.bst 
      cached 3dfbe3ca795bffca8eddf5ea3a1ee8736bd456e8883ae9282798c223883fd0f8 core-deps/gspell.bst 
      cached 5eb7c44b277c848b39d5ae42a34394524992f94cf625091276d518f775d31bdd core-deps/libgxps.bst 
      cached fb73a351efac485e6d25de430ce6be79475aa25e49657c7db769a73563bae6ba core-deps/openjpeg.bst 
      cached 234b586574af6ca09b96d0b82d82aae4587e60d2e59bc7e595fed18cc1ed1f13 core-deps/poppler.bst 
      cached 8294a11219948b4da8b3ce9ac54c7f3d9625bff9f5468f3f1cb310c205a870b4 core/evince.bst 
     waiting 00db5b85be0b5e5053a2b9e426d4aa9d73669dac8766bf6465f9421260433094 core/sushi.bst 
     waiting 33742b178f115cb60a5518928bed9da26b42f67b2cf9fdf60034f8fc1af51a88 core/vino.bst 
      cached e07f386aad882cfd8d20beded6f25d3c7eddeced7dcbaec5e3b52d31d71164e2 freedesktop-sdk.bst:desktop/libproxy.bst 
      cached 6439d1170447572947f075f828a8f0d4fc8982f697ab8ed22d84b631ea4bab8b sdk/glib-networking.bst 
      cached 1ecf222037ca51b5613ac14c558b164c4e36520ef610e8b4e1bc8115628b611b sdk/cantarell-fonts.bst 
     waiting d044149b8ea7f0de71a6cdd88227f775b1eab28a368fa122be1341d5c516f18a sdk/gnome-themes-extra.bst 
      cached dd040aeaccd9fb833b30fc59d6ae9f5754ca56124b4321a8f91fe74bb7871e2b sdk/gvfs.bst 
     waiting 1557e941a3156c79dda43d301cf5bf4496d7af9c51c1aa4e5a3d94d1bb056ccc core/meta-gnome-core-shell.bst 
     waiting 734d17cd896a00d3f3ce0c7ca8860a5c97a4f7a10c7c23791d90cb28130c507b core/baobab.bst 
     waiting a098a4a8add3f196ea8c7e46d72551b40c2c79a89c89fb02b19e89c237e0eed0 core-deps/boost.bst 
     waiting bfe243da7265c85ab3111a7d7ae8bd2a76e7e7c7d659a8dd8191c67dc3b8ee6d core-deps/exempi.bst 
      cached 8e5390294b56398ae2e228b4949509daec7bb0b9a27a365611be0cae91446bae core-deps/libpeas.bst 
     waiting 314ced3b13df990ebce02ca3fde571b3f0af6a3982209e1dbc97d9ea5de0a678 core/eog.bst 
      cached 98977b26b4a328df64c6ea69225ca821818b3053357e2c7ff97122b5ecf329ff core-deps/libdazzle.bst 
     waiting d0dbbf08d93c0c8ba9dc87e4688d0428c8cbfbab00b5ca34d07d44f9f8e25a4f core/epiphany.bst 
      cached 03c8b8bc796f9111d051d847655bc45ffe92631e0913f9b680941aa09e3b0d8d core/file-roller.bst 
     waiting 6777c202f96341a62daf657e57d1a56069b300f309896c8d3590ae97998cdf55 core/gedit.bst 
      cached 2bad02b3d1f648eda0dc9fa024a99637d5e63fc8216a730b6645ff686288953d core-deps/gtk-vnc.bst 
      cached 03ac6bba3383ed68974f83a01ca8968860fabb2e5e1cb58104021faf8be9b026 core-deps/libgovirt.bst 
      cached 8eb74ba234133400f13f93296fbf1fd8238e267f3d99ad2acb339c58cd6b7b2d core-deps/osinfo-db-tools.bst 
      cached 194ab4e91248774178611a8c80946cf3ffce417829712261e619765935765552 core-deps/libosinfo.bst 
      cached 3a8d2df6b45fc44258c8137531b170ab3c3d08867ec27dba4916e6f093cf25cb core-deps/libnl.bst 
      cached b2b4c3d2fb1d4803c38c08f200bbd81184c37a38b5ba4b6ec71d7ed5f50336ba core-deps/lvm2.bst 
      cached d97a8ed36c6e8f9814ae48494f7c22f752e10c3834df57c24804f02927f5f6ae core-deps/portablexdr.bst 
      cached 53f3f76103b08052a937fd5c068038600f38fb9fe55045b8c85abb16d553650b core-deps/libvirt.bst 
      cached 9156b39aa58b920cc2f034603f904eadbeb3cbea2fe1831c17120646f1a56bd6 core-deps/libvirt-glib.bst 
      cached 92991702334022c3cb8b979b5f7204717389e2caf887163700ee338fd0319ff5 core-deps/phodav.bst 
      cached a55a3489967fe63190ea66d4ffecc7cef1bc7d5209c983d623361899104bfa98 core-deps/spice-protocol.bst 
      cached ec9d48fc2a00a93d62dd2bfba74f06f384e9b0bc756415827f6475b850a1ea4f core-deps/spice-gtk.bst 
     waiting f3fdfb86080f8bc2a0f0387b695e7bd7a0d00dbe904c91a1e0218c0e05164e20 core/gnome-boxes.bst 
      cached 39a0582188812ca13c4ffbc1b3d85737fcf7b4313ae8162b2257dd1e5bdd045b core-deps/mpfr.bst 
      cached 3955fffaeb56b8630b34c4fe5da689053d96785d5d37a2aab34e84db79dde947 core-deps/mpc.bst 
     waiting 9233ecccdac0514de89bbe7d6255fea34472e1da3540e68e5f3f9c3379b92217 core/gnome-calculator.bst 
     waiting 2b5c7bc6529416756a79f970e389eec4a2d7b75a11b59e6695c3030592a96ed6 core/gnome-calendar.bst 
      cached 8b47a67cb1a36095cb18f3a43fcfa2f03c455876ca05ffa17fa9f6dd86995248 core/gnome-characters.bst 
      cached f23c8345bbaeb3f322f1a547bf7f848182559d03cd5deeefaceb00df9902711c core-deps/gsound.bst 
     waiting 916e81a5261dbc51735890107591cfc9b04c0d18cc81e89b73b7261fa5d562c5 core/gnome-clocks.bst 
     waiting bab7716b8df32e2cf7f07fced30564d826a185e8678fe5ae1b0a30184f635288 core-deps/folks.bst 
      cached 2adfc08a7b940bafb823a666dc2d8fd7b27831ed9da201f943d6384df0ddd194 core-deps/libchamplain.bst 
     waiting 1375b2e8bfdc58f312b709dccce82e4f78368a18bf607bd6e7b0cd7a605714bf core/gnome-contacts.bst 
      cached dff8942d112067d925db2563e41424b9705efb1df14e11213a9fb779de37c4b2 core-deps/libdvdread.bst 
      cached d5d1bdecf8f3fd0f4c053919fcadfd2693ad5f6df969c691f5927bef8b7a7bf9 core-deps/json-c.bst 
      cached b845492fbef6d7fea5056022131ee29f2a61c0e765ba55a1d827e2c00818c776 core-deps/popt.bst 
      cached ae7e8898c96f3f4d36077a94cbad9993fc60e4af55de40a44124f34bd0ced145 core-deps/cryptsetup.bst 
      cached a2c26f0a7ec950feebb07ca4d99e0e8d47ed956fa8b3079f58543c75eab4af3c core-deps/dmraid.bst 
      cached b6d2213da701adf3390936e8453b66682922e234a6f1c537957eb0187311a839 core-deps/kmod.bst 
      cached b2c1f0bcb8725fbb99dfc2653a44c107485cbbead8828af95afaebc28b431010 core-deps/libbytesize.bst 
      cached 66d4f0532062e98c1ed3080ce274ac544528699c7b53836e77fa2f993b6847c4 core-deps/parted.bst 
      cached b2c89ea01c16b1f0956e5bb2ca32ad0bf014ddc2a707c165e3b35f45b78a7494 core-deps/gpgme.bst 
      cached 80c4306beb9e0804bb0034d92b8566e60ffb071fc7dd46a2ebde9fc7855f577d core-deps/volume-key.bst 
      cached 192ff90aec18767bb2e51dc7c48aef9c5b3433f67f51de3822f9409d15b348b2 core-deps/blockdev.bst 
      cached 6c43d4d3d557e5ad26c0eba7c3fe0c546803eb1695ec7c15358a4cc8b7a7b56a core-deps/libatasmart.bst 
      cached 2d4b832f98ad385c4b098b3f374d16abe7d2e2fb14e96271d06249faa47c2d51 core-deps/udisks2.bst 
     waiting 3aa8471e376e9fa79bdfb5f44d91505f46803ad2d2f4fc50b4719b0b599e113f core/gnome-disk-utility.bst 
      cached a39766caf55f08f8c35b9628efe7d5aa879f4a8b407ea93cb5e18fbf12c1a369 core-deps/libzapojit.bst 
      cached 860ae6466d296b7b47d2d302b63c4523904f880aed715d5d55998b3381504bad core-deps/libgfbgraph.bst 
     waiting 088c8d016cef175206cd206331d2fc3a471071b5ac88d303c999aad3bab2ce80 core-deps/gnome-online-miners.bst 
      cached cec13925ac971e1ea7e602ef9cee3fda4a27b285d4579e80fac65398f53ec8be core-deps/libgepub.bst 
      cached f21e62e3c60f5243fb4cc82ab80ed0ac5de3eb0203fc1b5f44d64d836e6565c6 freedesktop-sdk.bst:desktop/desktop-file-utils.bst 
     waiting 569b06a6897e69f7ddfc4338a5ea72892de1e0010c599e32437d540bbb3b0329 core/gnome-documents.bst 
     waiting b2be6a0d90ff1dd11274da1b39997bf2c0f994fa3fc570514f69bd37df1bab91 core/gnome-font-viewer.bst 
     waiting 4d448ca329ce8a6fa3146f0dc4ec14d4135f7706a9ac1e096a509d50582c5f51 core/gnome-logs.bst 
     waiting 44d6d6a283500be31dea74be3fba5c61584417363f197342fe0f848557ad64f3 core/gnome-maps.bst 
      cached 3500c28fc4a534b5392140fc13e6b5c50e9a471847522763032721e2e6ef2100 core-deps/gom.bst 
     waiting 703dafd851c1aad790f31715a3bd4c6db153a378cae7c389ecc58463427e8e34 core-deps/grilo-plugins.bst 
     waiting 815ce04303880edb78d1752516c18cdd33a0384a1b354d7986a314f0949de6dc core/gnome-music.bst 
      cached 4cf4ca217243fd79d34a265925d82b72e3319bc34135fc027475874f80b886f7 core-deps/babl.bst 
      cached fcb5848f72b101e154643faa0b98572703b4c1b010dc2ce04765627beaf00713 core-deps/graphviz.bst 
      cached 54a6c5fdecb0c4d9d3d7fdee4b2cd3416c38d2da17e82a9db6294e432d975d38 core-deps/gegl.bst 
     waiting d19ffb0cea3dd5d4031b937edddd6561ad689fb2abca57dd230ccf38d5056ad5 core/gnome-photos.bst 
      cached b9b5e2673c6bf7310892b63900883ded1e592888b6369a589f11627cc04cc73c core/gnome-screenshot.bst 
     waiting c98403e81ee9a4aa09daad2f8f5a6b45bfd72e245f24cd5806cb73d8d9716f5b core/gnome-software.bst 
      cached b89bc9773c3d161b10101abc153c9b2059c8aaa2d3053762a6d9bc72fdff1a20 core-deps/mm-common.bst 
      cached 8275917b6648d14f0a1bf7defe23edef0a7f9a13c8143634ca18cb54111ffca0 core-deps/libsigcplusplus-2.0.bst 
      cached f960f596d8a96542f4fab2346933df95352bb951182ede3592727cb84fe31861 core-deps/glibmm-2.4.bst 
      cached b1fc09dc8d7e46b5f38aac21fbb5dbc4953d2595a4b06b5e41a146e90f88032d core-deps/atkmm-1.6.bst 
      cached 9bc7560d9ede3a15dc23d092a3c21a1d810188ff87c513e1800454846c2d8fb5 core-deps/cairomm-1.0.bst 
      cached 8490e35c5b2337f6005f4355ccc03d83bee76bb594f3ea901d41375fc690634e core-deps/pangomm-1.4.bst 
      cached 1386587a5575bc734c5b3a6153ab4a3dfe4caf169954efe5d96c806be326f720 core-deps/gtkmm-3.bst 
     waiting cda863b328003d5283bb81448f0eca36df495300f520e6fa702e4187e309f8af core/gnome-system-monitor.bst 
     waiting 16984187aff2c44f8e80c595faecb64417d75ff7aa5f43c2db50cf8932d86c5c core/gnome-terminal.bst 
     waiting 35c2185f48717b982c1bbd5b2d35758043874097093820a65d7fd9cb7ebf765f core/gnome-weather.bst 
     waiting 7d84845ee91af21b6c9bf06f3b954b2f3a05d0efb3eb658edf6e2c5bbb6ce5c9 core-deps/sane-backends.bst 
     waiting a59644e7cbfaa9389070b9d7cad3819f5aa3c69de9787f07aec7d8e8667c25e6 core/simple-scan.bst 
     waiting f32ab7f2280db08a19df207234dd69ea2493c9b1db465fd8e95816df9e0fc72e core/totem.bst 
     waiting 81d45800346c477adab46cf08c3f2ff14e105476a687e60ec2c27877a4b71faa sdk/yelp.bst 
     waiting ccafafa6f227783b90ba21ce8421cabf516b2dbb49b714cba4b70380c4894ed0 core/meta-gnome-core-utilities.bst 
     waiting 87310475a2d3c7266caa633758199d3dff76ad446672a2ebeff0a506970828fb core.bst 
===============================================================================
[--:--:--][765ee514][ pull:sdk/WebKitGTK+.bst            ] START   gnome/sdk-WebKitGTK+/765ee514-pull.28843.log
[--:--:--][765ee514][ pull:sdk/WebKitGTK+.bst            ] INFO    Pulling 765ee514 <- https://testcache.codethink.co.uk:11001
[--:--:--][f8b90ec6][ pull:core-deps/grilo.bst           ] START   gnome/core-deps-grilo/f8b90ec6-pull.28845.log
[--:--:--][f8b90ec6][ pull:core-deps/grilo.bst           ] INFO    Pulling f8b90ec6 <- https://testcache.codethink.co.uk:11001
[--:--:--][8df44378][ pull:core-deps/evolution-data-server.bst] START   gnome/core-deps-evolution-data-server/8df44378-pull.28847.log
[--:--:--][8df44378][ pull:core-deps/evolution-data-server.bst] INFO    Pulling 8df44378 <- https://testcache.codethink.co.uk:11001
[--:--:--][a098a4a8][ pull:core-deps/boost.bst           ] START   gnome/core-deps-boost/a098a4a8-pull.28851.log
[--:--:--][a098a4a8][ pull:core-deps/boost.bst           ] INFO    Pulling a098a4a8 <- https://testcache.codethink.co.uk:11001
[--:--:--][7d84845e][ pull:core-deps/sane-backends.bst   ] START   gnome/core-deps-sane-backends/7d84845e-pull.28855.log
[--:--:--][7d84845e][ pull:core-deps/sane-backends.bst   ] INFO    Pulling 7d84845e <- https://testcache.codethink.co.uk:11001
[--:--:--][5d62838d][ pull:core/gnome-getting-started-docs.bst] START   gnome/core-gnome-getting-started-docs/5d62838d-pull.28857.log
[--:--:--][5d62838d][ pull:core/gnome-getting-started-docs.bst] INFO    Pulling 5d62838d <- https://testcache.codethink.co.uk:11001
[--:--:--][bab7716b][ pull:core-deps/folks.bst           ] START   gnome/core-deps-folks/bab7716b-pull.28865.log
[--:--:--][bab7716b][ pull:core-deps/folks.bst           ] INFO    Pulling bab7716b <- https://testcache.codethink.co.uk:11001
[--:--:--][088c8d01][ pull:core-deps/gnome-online-miners.bst] START   gnome/core-deps-gnome-online-miners/088c8d01-pull.28867.log
[--:--:--][088c8d01][ pull:core-deps/gnome-online-miners.bst] INFO    Pulling 088c8d01 <- https://testcache.codethink.co.uk:11001
[--:--:--][f32ab7f2][ pull:core/totem.bst                ] START   gnome/core-totem/f32ab7f2-pull.28873.log
[--:--:--][f32ab7f2][ pull:core/totem.bst                ] INFO    Pulling f32ab7f2 <- https://testcache.codethink.co.uk:11001
[--:--:--][81d45800][ pull:sdk/yelp.bst                  ] START   gnome/sdk-yelp/81d45800-pull.28875.log
[--:--:--][81d45800][ pull:sdk/yelp.bst                  ] INFO    Pulling 81d45800 <- https://testcache.codethink.co.uk:11001
[--:--:--][bab7716b][ pull:core-deps/folks.bst           ] SKIPPED Remote (https://testcache.codethink.co.uk:11001) does not have bab7716b cached
[--:--:--][bab7716b][ pull:core-deps/folks.bst           ] INFO    Pulling bab7716b <- https://cache.gitlab.gnome.org:11001
[--:--:--][a098a4a8][ pull:core-deps/boost.bst           ] SKIPPED Remote (https://testcache.codethink.co.uk:11001) does not have a098a4a8 cached
[--:--:--][a098a4a8][ pull:core-deps/boost.bst           ] INFO    Pulling a098a4a8 <- https://cache.gitlab.gnome.org:11001
[--:--:--][765ee514][ pull:sdk/WebKitGTK+.bst            ] SKIPPED Remote (https://testcache.codethink.co.uk:11001) does not have 765ee514 cached
[--:--:--][765ee514][ pull:sdk/WebKitGTK+.bst            ] INFO    Pulling 765ee514 <- https://cache.gitlab.gnome.org:11001
[--:--:--][5d62838d][ pull:core/gnome-getting-started-docs.bst] SKIPPED Remote (https://testcache.codethink.co.uk:11001) does not have 5d62838d cached
[--:--:--][5d62838d][ pull:core/gnome-getting-started-docs.bst] INFO    Pulling 5d62838d <- https://cache.gitlab.gnome.org:11001
[--:--:--][f8b90ec6][ pull:core-deps/grilo.bst           ] SKIPPED Remote (https://testcache.codethink.co.uk:11001) does not have f8b90ec6 cached
[--:--:--][f8b90ec6][ pull:core-deps/grilo.bst           ] INFO    Pulling f8b90ec6 <- https://cache.gitlab.gnome.org:11001
[--:--:--][7d84845e][ pull:core-deps/sane-backends.bst   ] SKIPPED Remote (https://testcache.codethink.co.uk:11001) does not have 7d84845e cached
[--:--:--][7d84845e][ pull:core-deps/sane-backends.bst   ] INFO    Pulling 7d84845e <- https://cache.gitlab.gnome.org:11001
[--:--:--][088c8d01][ pull:core-deps/gnome-online-miners.bst] SKIPPED Remote (https://testcache.codethink.co.uk:11001) does not have 088c8d01 cached
[--:--:--][088c8d01][ pull:core-deps/gnome-online-miners.bst] INFO    Pulling 088c8d01 <- https://cache.gitlab.gnome.org:11001
[--:--:--][81d45800][ pull:sdk/yelp.bst                  ] SKIPPED Remote (https://testcache.codethink.co.uk:11001) does not have 81d45800 cached
[--:--:--][81d45800][ pull:sdk/yelp.bst                  ] INFO    Pulling 81d45800 <- https://cache.gitlab.gnome.org:11001
[--:--:--][8df44378][ pull:core-deps/evolution-data-server.bst] SKIPPED Remote (https://testcache.codethink.co.uk:11001) does not have 8df44378 cached
[--:--:--][8df44378][ pull:core-deps/evolution-data-server.bst] INFO    Pulling 8df44378 <- https://cache.gitlab.gnome.org:11001
[--:--:--][f8b90ec6][ pull:core-deps/grilo.bst           ] SKIPPED Remote (https://cache.gitlab.gnome.org:11001) does not have f8b90ec6 cached
[--:--:--][a098a4a8][ pull:core-deps/boost.bst           ] SKIPPED Remote (https://cache.gitlab.gnome.org:11001) does not have a098a4a8 cached
[--:--:--][5d62838d][ pull:core/gnome-getting-started-docs.bst] SKIPPED Remote (https://cache.gitlab.gnome.org:11001) does not have 5d62838d cached
[--:--:--][088c8d01][ pull:core-deps/gnome-online-miners.bst] SKIPPED Remote (https://cache.gitlab.gnome.org:11001) does not have 088c8d01 cached
[--:--:--][f32ab7f2][ pull:core/totem.bst                ] SKIPPED Remote (https://testcache.codethink.co.uk:11001) does not have f32ab7f2 cached
[--:--:--][f32ab7f2][ pull:core/totem.bst                ] INFO    Pulling f32ab7f2 <- https://cache.gitlab.gnome.org:11001
[--:--:--][bab7716b][ pull:core-deps/folks.bst           ] SKIPPED Remote (https://cache.gitlab.gnome.org:11001) does not have bab7716b cached
[--:--:--][8df44378][ pull:core-deps/evolution-data-server.bst] SKIPPED Remote (https://cache.gitlab.gnome.org:11001) does not have 8df44378 cached
[--:--:--][f32ab7f2][ pull:core/totem.bst                ] SKIPPED Remote (https://cache.gitlab.gnome.org:11001) does not have f32ab7f2 cached
[00:00:04][f8b90ec6][ pull:core-deps/grilo.bst           ] SUCCESS gnome/core-deps-grilo/f8b90ec6-pull.28845.log
[00:00:04][a098a4a8][ pull:core-deps/boost.bst           ] SUCCESS gnome/core-deps-boost/a098a4a8-pull.28851.log
[00:00:04][5d62838d][ pull:core/gnome-getting-started-docs.bst] SUCCESS gnome/core-gnome-getting-started-docs/5d62838d-pull.28857.log
[00:00:04][088c8d01][ pull:core-deps/gnome-online-miners.bst] SUCCESS gnome/core-deps-gnome-online-miners/088c8d01-pull.28867.log
[00:00:05][bab7716b][ pull:core-deps/folks.bst           ] SUCCESS gnome/core-deps-folks/bab7716b-pull.28865.log
[00:00:05][8df44378][ pull:core-deps/evolution-data-server.bst] SUCCESS gnome/core-deps-evolution-data-server/8df44378-pull.28847.log
[00:00:06][f32ab7f2][ pull:core/totem.bst                ] SUCCESS gnome/core-totem/f32ab7f2-pull.28873.log
[--:--:--][81d45800][ pull:sdk/yelp.bst                  ] INFO    Downloaded artifact 81d45800
[00:03:24][81d45800][ pull:sdk/yelp.bst                  ] SUCCESS gnome/sdk-yelp/81d45800-pull.28875.log
[--:--:--][7d84845e][ pull:core-deps/sane-backends.bst   ] INFO    Downloaded artifact 7d84845e
[00:06:02][7d84845e][ pull:core-deps/sane-backends.bst   ] SUCCESS gnome/core-deps-sane-backends/7d84845e-pull.28855.log
[--:--:--][765ee514][ pull:sdk/WebKitGTK+.bst            ] INFO    Downloaded artifact 765ee514
[01:05:26][765ee514][ pull:sdk/WebKitGTK+.bst            ] SUCCESS gnome/sdk-WebKitGTK+/765ee514-pull.28843.log
[--:--:--][][] START   cache_size/cache_size.13266.log
[00:00:00][][] SUCCESS cache_size/cache_size.13266.log
[--:--:--][][] START   cache_size/cache_size.13268.log
[00:00:00][][] SUCCESS cache_size/cache_size.13268.log
[--:--:--][][] START   cache_size/cache_size.13270.log
[00:00:00][][] SUCCESS cache_size/cache_size.13270.log
[--:--:--][][] START   cache_size/cache_size.13272.log
[00:00:00][][] SUCCESS cache_size/cache_size.13272.log
[--:--:--][][] START   cache_size/cache_size.13274.log
[00:00:00][][] SUCCESS cache_size/cache_size.13274.log
[--:--:--][][] START   cache_size/cache_size.13276.log
[00:00:00][][] SUCCESS cache_size/cache_size.13276.log
[--:--:--][][] START   cache_size/cache_size.13278.log
[00:00:00][][] SUCCESS cache_size/cache_size.13278.log
[--:--:--][][] START   cache_size/cache_size.13280.log
[00:00:00][][] SUCCESS cache_size/cache_size.13280.log
[--:--:--][][] START   cache_size/cache_size.13282.log
[00:00:00][][] SUCCESS cache_size/cache_size.13282.log
[--:--:--][][] START   cache_size/cache_size.13284.log
[00:00:00][][] SUCCESS cache_size/cache_size.13284.log
[--:--:--][][] START   cleanup.13286.log
[00:00:01][][] BUG     cleanup

    An unhandled exception occured:

    Traceback (most recent call last):
      File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 398, in _child_action
        result = self.child_process()
      File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/cleanupjob.py", line 30, in child_process
        return self._cache.clean()
      File "/codethink/GNOME/buildstream/buildstream/_artifactcache/artifactcache.py", line 257, in clean
        size = self.remove(to_remove)
      File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 564, in remove
        pruned = self.prune()
      File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 586, in prune
        self._reachable_refs_dir(reachable, tree)
      File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 732, in _reachable_refs_dir
        with open(self.objpath(tree), 'rb') as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/home/tristan/.cache/buildstream/artifacts/cas/objects/7f/4c864e8edd35a496c17285a1035b7221d3d317f90deb65a964c1d77c5c4364'

Terminating all jobs

[--:--:--][][] STATUS  cleanup terminating
Unknown child process pid 13289, will report returncode 255
[01:05:39][][] WARNING Build Terminated

Pipeline Summary
  Total:       471
  Session:     59
  Pull Queue:  processed 3, skipped 7,  failed 0 
  Fetch Queue: processed 0, skipped 10, failed 0 
  Build Queue: processed 0, skipped 3,  failed 0
Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 3, 2018, 15:47

Another observation from the above stack trace which is unrelated but interesting:

Terminating all jobs

[--:--:--][][] STATUS  cleanup terminating
Unknown child process pid 13289, will report returncode 255
[01:05:39][][] WARNING Build Terminated

Notice:

Unknown child process pid 13289, will report returncode 255

We've seen this before in conjunction with BUG messages, it appears that we try to kill the task which just got aborted by itself, due to the unhandled exception reported in the BUG message. Perhaps trying to kill a process that we should know is dying could also be avoided.

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tlater] on Sep 3, 2018, 18:06

We then launched a bunch of cache_size jobs

  • I think launching more than one here, if any, is a bit weird right ?

From some experimentation this seems to happen continuously (i.e. on every scheduler tick) as long as the cache is empty. Probably a rogue check for cache_size == 0 somewhere.

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tlater] on Sep 3, 2018, 18:17

I've finally set up a little testing VM for this where I can control the artifact cache disk size to tickle out these edge cases... The original issue is definitely caused by too little "headroom". Some observations:

I'll spend more time on this later this week. Perhaps we should split the various issues we've uncovered here though - I think there are at least 3 right now.

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 5, 2018, 13:02

mentioned in commit 0b51f1f08a1e7064ebc09fe41cb0cad69f931011

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 5, 2018, 13:02

mentioned in commit 470305a70e67d5567d09aede164672ddd4f86aba

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 5, 2018, 13:02

mentioned in commit a02023dc06a98764630772a2634ad1830e6d964b

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 5, 2018, 13:02

mentioned in commit 4e2d51992ce8d4b273f5c62f7bab848e5038dd07

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 5, 2018, 13:20

mentioned in commit 3dba9c767688c714ba34851d631378dd4a7284b5

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 5, 2018, 13:20

mentioned in commit a569b561fe19b0f667bde9faf0b6785528c8c76b

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 5, 2018, 13:20

mentioned in commit fe8e397bd3c8171903de6453a20d3852cba8fed6

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 5, 2018, 13:21

mentioned in commit b8ade1b0c77073374555bb872b59da017b9f285d

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 5, 2018, 13:21

mentioned in commit 65eb56b36d545cbcf47fd5c4b89b0ce17f46b57f

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 5, 2018, 13:21

mentioned in commit 158aca8674be8e9489a973ab80084d5ccdbe223c

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 7, 2018, 09:51

Ok so here is something interesting that happened to me.

To test !775 and !779, I did the following:

Relevant part of the log: 

[00:00:10][58c1bdde][ pull:freedesktop-sdk.bst:base/gettext.bst] SUCCESS freedesktop-sdk/base-gettext/58c1bdde-pull.19322.log
[--:--:--][957cf7b7][ pull:freedesktop-sdk.bst:base/openssl.bst] INFO    Downloaded artifact 957cf7b7
[00:03:09][957cf7b7][ pull:freedesktop-sdk.bst:base/openssl.bst] SUCCESS freedesktop-sdk/base-openssl/957cf7b7-pull.19303.log
[--:--:--][][] START   cleanup.20000.log
[00:00:14][][] FAILURE Cache too full. Aborting.

    There is not enough space to build the given element.
    Please increase the cache-quota in /home/tristan/.config/buildstream.conf.

    Printing the last 20 lines from log file:
    /home/tristan/.cache/buildstream/logs/cleanup.20000.log
    ======================================================================
    [--:--:--] START  : cleanup
    [00:00:14] FAILURE: Cache too full. Aborting.

        There is not enough space to build the given element.
        Please increase the cache-quota in /home/tristan/.config/buildstream.conf.
    ======================================================================

Terminating all jobs

The above is what is failing in CI for the mentioned MRs, and I suspect that what is happening is the following:

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 10, 2018, 07:57

mentioned in commit d2f5c4f33394da5237276695df1d57a7d81664b2

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 10, 2018, 07:57

mentioned in commit 78499b7dfcaf962a6da221081be06760f7ea3637

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 10, 2018, 07:57

mentioned in commit f81637c5e49a25ab5d096e46494d639fcccc68e8

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 10, 2018, 07:57

mentioned in commit f074a8acb3580f6f528a3889502614ac310d6ea3

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 10, 2018, 08:16

mentioned in commit fc79fb67e769f43a60d8eb8933f6ea1d8f521a0d

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 10, 2018, 08:16

mentioned in commit 48ebe8897628e3f91bc0a46007c04b8b2e5d0cb9

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 10, 2018, 08:16

mentioned in commit 7c937cb83e9beb289c524fb3c6fcabe92491f766

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 10, 2018, 08:16

mentioned in commit 446654a2bb36a73d4f936b49e57bafa7de408d9d

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 10, 2018, 08:54

mentioned in commit e5847077e6493e86ec5de09b2efa1f74a6f2d099

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 10, 2018, 08:54

mentioned in commit deaecdc1ad1c51096cc00d364d17cb8b0183a010

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 10, 2018, 08:54

mentioned in commit 98ab2b8916716d8d6dde6eb778c50addff90400b

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 10, 2018, 08:54

mentioned in commit 6db54c8f156ad66572e760c38b96bcc355f586a5

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 10, 2018, 11:33

Update:

For the latter part, it can happen when we are committing a huge artifact, and the cache quota is set automatically to take as much space as possible (when the committed artifact exceeds the headroom).

In these conditions, it is important to handle things in various errors:

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 18, 2018, 11:29

This is mostly fixed by !775 and !779, and will no longer realistically happen to users.

Opened #660 instead to address stack traces which occur when we actually do run out of disk space.

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 18, 2018, 11:29

closed

Cynical-Optimist commented 4 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 18, 2018, 11:30

marked this issue as related to #660