search

nestauk / daps_utils

Tools for setting up and running pipelines in a Data Analytics and Production System (DAPS).
0 stars 1 forks source link

Implement docker prune before build #117

Closed jaklinger closed 2 years ago

jaklinger commented 2 years ago

OJO has been failing in production at the MetaflowTask launch stage of enrich, which launches several tasks, and some of which may have heavy artefacts:

self.flow_path,
  File "/root/.local/share/virtualenvs/ojd_daps-RcsbzZLf/lib/python3.9/site-packages/daps_utils/docker_utils.py", line 331, in build_and_run_image
    flow_tag = build_flow_image(
  File "/root/.local/share/virtualenvs/ojd_daps-RcsbzZLf/lib/python3.9/site-packages/daps_utils/docker_utils.py", line 254, in build_flow_image
    build_image(
  File "/root/.local/share/virtualenvs/ojd_daps-RcsbzZLf/lib/python3.9/site-packages/daps_utils/docker_utils.py", line 143, in build_image
    img, logs = _build_image(pkg, tag, rebuild, **kwargs)
  File "/root/.local/share/virtualenvs/ojd_daps-RcsbzZLf/lib/python3.9/site-packages/daps_utils/docker_utils.py", line 137, in _build_image
    img, logs = dkr.images.build(path=pkg.__basedir__, tag=tag, **kwargs)
  File "/root/.local/share/virtualenvs/ojd_daps-RcsbzZLf/lib/python3.9/site-packages/docker/models/images.py", line 287, in build
    raise BuildError(chunk['error'], result_stream)
docker.errors.BuildError: devmapper: Thin Pool has 2753 free data blocks which is less than minimum required 4449 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior

This error is repeated across several tasks in the enrich module, implying that the production instance has run out of diskspace. Hopefully clearing the docker cache between runs should fix this.