Device or resource busy with Sagemaker and S3 bucket downloading data using boto3

lukasavic1 commented 2 years ago

Describe the bug

I've been trying to download some json data from my bucket, as usual, using my Sagemaker notebook instance. I've done it a million times, and now out of nowhere, this error started showing up.

OSError: [Errno 16] Device or resource busy: 'transcript_data/..Aea2DBEA' -> 'transcript_data/.'

Expected Behavior

Expected behaviour is to have all data downloaded to a folder, very straightforward.

Current Behavior

This exception is thrown before first file is downloaded:

OSError: [Errno 16] Device or resource busy: 'transcript_data/..Aea2DBEA' -> 'transcript_data/.'

Reproduction Steps

Here is my code snippet for downloading files:

LOCAL_PATH = "transcript_data/"
files = sagemaker_session.list_s3_files(bucket, FOLDER_PREFIX)
for file in files:
    if file.endswith('.json'):
        sagemaker_session.download_data(path=LOCAL_PATH, bucket=bucket, key_prefix=FOLDER_PREFIX)

Possible Solution

I already tried restarting the instances, but it didn't help much. Also, I tried changing from one kernel to the other (pytorch and conda python), but it didn't help.

Additional Information/Context

No response

SDK version used

boto3==1.26.2

Environment details (OS name and version, etc.)

Amazon Linux 2

tim-finnigan commented 2 years ago

Hi @lukasavic1 thanks for reaching out. I tried searching for similar issues and saw that you asked about this here in StackOverflow as well. I think this reply there is worth looking into:

Is that with SageMaker Studio Notebooks? You seem to have a disk issue. Verify by trying to download to /tmp instead to bypass the share storage (based on EFS)

Also reaching out through AWS Support is probably the best way to get direct assistance for issues such as these. You could consider asking in the re:Post forums as well, as someone in that community with more SageMaker experience may have more familiarity with these kinds of issues.

I think this is a storage/SageMaker issue rather than directly a boto3 issue. But if you want to provide your debug logs (with any sensitive info redacted) by adding boto3.set_stream_logger('') to your script then we could investigate further.

lukasavic1 commented 1 year ago

Thank you for your response! I thought it is a problem with storage, but didn't know where else to reach out since I was able to download/upload data using was cli In sagemkaer terminal..

I will make sure to get a bit more info about how EFS and other operate. Thanks for your recommendations!

boto / boto3