search

coiled / feedback

A place to provide Coiled feedback
14 stars 3 forks source link

"No space left on device" on docker build of new software environment #207

Closed jfb51 closed 2 years ago

jfb51 commented 2 years ago

"No space left on device" on docker build of new software environment

Issue Priority, optional

Environment, optional

Diagnostics, optional Please run coiled.diagnostics locally and paste the result below:

Coiled Diagnostics Output: {'msg': 'Creating new software environment', 'id': '97c74f', 'username': 'julianfb51', 'account': 'julianfb51', 'level': 20, 'event_type': 'create_software_environment', 'elapsed': None, 'logging_context': {'coiled-operation-id': 'r604zq6o', 'coiled-session-id': 'coiled-session-s4anbh3s', 'coiled-request-id': '7dayieq1'}, 'coiled-operation-id': 'r604zq6o', 'coiled-session-id': 'coiled-session-s4anbh3s', 'coiled-request-id': '7dayieq1', 'timestamp': 1665471247.3809173}, {'msg': 'Creating new ecr build', 'id': '8dfc54', 'username': 'julianfb51', 'account': 'julianfb51', 'level': 20, 'event_type': 'create_software_environment', 'elapsed': None, 'logging_context': {'coiled-operation-id': 'r604zq6o', 'coiled-session-id': 'coiled-session-s4anbh3s', 'coiled-request-id': '7dayieq1'}, 'coiled-operation-id': 'r604zq6o', 'coiled-session-id': 'coiled-session-s4anbh3s', 'coiled-request-id': '7dayieq1', 'timestamp': 1665471247.391795}, {'msg': 'STEP 1: FROM coiled/default:sha-6b4e896\n', 'id': 'eae5db', 'username': 'julianfb51', 'account': 'julianfb51', 'level': 20, 'event_type': 'build_container', 'elapsed': None, 'logging_context': {'coiled-operation-id': 'r604zq6o', 'coiled-session-id': 'coiled-session-s4anbh3s', 'coiled-request-id': '7dayieq1'}, 'coiled-operation-id': 'r604zq6o', 'coiled-session-id': 'coiled-session-s4anbh3s', 'coiled-request-id': '7dayieq1', 'timestamp': 1665471248.0064356}, {'msg': 'error creating build container: error creating container: error creating read-write layer with ID "62b0be86b040769931accb062d0f768f415de233747a5ead8aa25d4429fb444d": open /var/lib/containers/storage/vfs/dir/62b0be86b040769931accb062d0f768f415de233747a5ead8aa25d4429fb444d/var/lib/dpkg/info/python2.7.list: no space left on device\n', 'id': '83c1be', 'username': 'julianfb51', 'account': 'julianfb51', 'level': 40, 'event_type': 'build_container', 'elapsed': None, 'logging_context': {'coiled-operation-id': 'r604zq6o', 'coiled-session-id': 'coiled-session-s4anbh3s', 'coiled-request-id': '7dayieq1'}, 'coiled-operation-id': 'r604zq6o', 'coiled-session-id': 'coiled-session-s4anbh3s', 'coiled-request-id': '7dayieq1', 'timestamp': 1665471254.393342}, {'msg': 'Docker build failed', 'id': 'f16479', 'username': 'julianfb51', 'account': 'julianfb51', 'level': 40, 'event_type': 'build_container', 'elapsed': None, 'logging_context': {'coiled-operation-id': 'r604zq6o', 'coiled-session-id': 'coiled-session-s4anbh3s', 'coiled-request-id': '7dayieq1'}, 'coiled-operation-id': 'r604zq6o', 'coiled-session-id': 'coiled-session-s4anbh3s', 'coiled-request-id': '7dayieq1', 'timestamp': 1665471254.4103281}]}

Additional context Hi coiled team - I'm trying to upload a new software environment to AWS ECR but can't - this is blocking me from doing production work. I'm getting this error:

ServerError: Docker build failed with returncode: 125 STEP 1: FROM coiled/default:sha-6b4e896

error creating build container: error creating container: error creating read-write layer with ID "1737bacbf1e4355e6c70ee47e076c9b483c29a67bea96dab8e41124dc0ba2751": open /var/lib/containers/storage/vfs/dir/1737bacbf1e4355e6c70ee47e076c9b483c29a67bea96dab8e41124dc0ba2751/var/lib/dpkg/info/util-linux.postinst: no space left on device

Is this a coiled side problem? The caveat is I am a free tier AWS user, but it looks like I should just be getting charged some small amount by AWS for ECR usage in excess of the free limit, rather than getting stopped from uploading anything?

I have tried deleting my old software environments from both coiled and AWS ECR but this doesn't seem to help.

hayesgb commented 2 years ago

cc: @phobson, @dchudz

@jfb51 -- I'll defer comments regarding device space to @dchudz's team. Not sure what your specific use case is, but you may be interested in checking out package sync

ntabris commented 2 years ago

This is a problem on our side (infra), we're looking into it and will let you know when it's resolved.

ntabris commented 2 years ago

@jfb51 this should now be resolved, apologies for the inconvenience! We'll be keeping a closer eye out for this in the future so it won't impact prod.

jfb51 commented 2 years ago

Cool, thank you, I’ll give it a try

On Wed, 12 Oct 2022 at 3:43 AM, Nat Tabris @.***> wrote:

@jfb51 https://github.com/jfb51 this should now be resolved, apologies for the inconvenience! We'll be keeping a closer eye out for this in the future so it won't impact prod.

— Reply to this email directly, view it on GitHub https://github.com/coiled/feedback/issues/207#issuecomment-1274984116, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWZEVBI42IC47WIB3VORKJDWCWKLHANCNFSM6AAAAAARB67FGU . You are receiving this because you were mentioned.Message ID: @.***>