Failure during the download bucket content using IBM BOTO3

[vegoutha]

We are trying to download the bucket content using IBM BOTO3 and it fails with below error message: FileNotFoundError: [Errno 2] No such file or directory: '/opt/ibm/scoring/python/workdir/batch_inputabe514e7e26c7c8c137e80162fdfdfe8/test/.2f6C9b89'

Here is the detailed exception stack trace:

2018/06/13 20:43:36,334|WARNING|servicer.scoring_grpc_batch_servicer:70|abe514e7e26c7c8c137e80162fdfdfe8|608f18bb-9269-4a0e-a40a-daf60eb057f1:Batch Deploy and Score error stacktrace: Traceback (most recent call last):
  File "/opt/ibm/scoring/python/py_scoring_base/py_runtime/caffe_runtime.py", line 418, in predict_batch
    self._score_caffe_model_batch(details_json, caffe_entry, input, output)
  File "/opt/ibm/scoring/python/py_scoring_base/py_runtime/caffe_runtime.py", line 327, in _score_caffe_model_batch
    BatchConnector.download(in_connector_parms)
  File "/opt/ibm/scoring/python/py_common_base/utils/batch_connector.py", line 97, in download
    return BatchConnector.aws_cos_download(param)
  File "/opt/ibm/scoring/python/py_common_base/utils/batch_connector.py", line 187, in aws_cos_download
    amz.meta.client.download_file(bucket_name, object.key, download_path+'/'+object.key)
  File "/opt/conda/lib/python3.6/site-packages/ibm_boto3/s3/inject.py", line 126, in download_file
    extra_args=ExtraArgs, callback=Callback)
  File "/opt/conda/lib/python3.6/site-packages/ibm_boto3/s3/transfer.py", line 299, in download_file
    future.result()
  File "/opt/conda/lib/python3.6/site-packages/ibm_s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/opt/conda/lib/python3.6/site-packages/ibm_s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/opt/conda/lib/python3.6/site-packages/ibm_s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/opt/conda/lib/python3.6/site-packages/ibm_s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/opt/conda/lib/python3.6/site-packages/ibm_s3transfer/download.py", line 568, in _main
    fileobj.seek(offset)
  File "/opt/conda/lib/python3.6/site-packages/ibm_s3transfer/utils.py", line 317, in seek
    self._open_if_needed()
  File "/opt/conda/lib/python3.6/site-packages/ibm_s3transfer/utils.py", line 300, in _open_if_needed
    self._fileobj = self._open_function(self._filename, self._mode)
  File "/opt/conda/lib/python3.6/site-packages/ibm_s3transfer/utils.py", line 226, in open
    return open(filename, mode)
**FileNotFoundError: [Errno 2] No such file or directory: '/opt/ibm/scoring/python/workdir/batch_input_abe514e7e26c7c8c137e80162fdfdfe8/test_/.2f6C9b89'**

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/ibm/scoring/python/py_grpc_base/servicer/scoring_grpc_batch_servicer.py", line 57, in BatchDeployAndScore
    request.output
  File "/opt/ibm/scoring/python/py_scoring_base/py_runtime/caffe_runtime.py", line 467, in predict_batch
    ) from ex
py_common.errors.WmlBatchScoringException: (status_code=500, trace=abe514e7e26c7c8c137e80162fdfdfe8, code=load_model_failure, message=[Errno 2] No such file or directory: '/opt/ibm/scoring/python/workdir/batch_input_abe514e7e26c7c8c137e80162fdfdfe8/test_/.2f6C9b89', more_info=None, target_type=None, target_name=None)

Installation of IBM BOTO3 using the below command:

pip install ibm-cos-sdk \
pip install botocore

Code Snippet: amz = ibm_boto3.resource('s3', endpoint_url=service_endpoint, aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)

my_bucket = amz.Bucket(bucket_name) for object in my_bucket.objects.all(): amz.meta.client.download_file(bucket_name, object.key, download_path+'/'+object.key)

This error is observed intermittently. Please have a look into it.

[Patrick-Browne]

@vegoutha can you confirm that the path exists and that you have sufficient permissions prior to calling download_file. I can only reproduce this error by providing an invalid path.

[vegoutha]

@Patrick-Browne This is the same code snippet used to download the content during every attempt. If it issue with Permission (or) Path, it should always fail with same error. This issue is observed intermittently.

As per our code, we create the directory /opt/ibm/scoring/python/workdir/batch_input_abe514e7e26c7c8c137e80162fdfdfe8 and this will valid path and it won't have any issues with permission.

But I see it is failing to find some temporary file FileNotFoundError: [Errno 2] No such file or directory: '/opt/ibm/scoring/python/workdir/batch_input_abe514e7e26c7c8c137e80162fdfdfe8/test_/.2f6C9b89' in some test_ directory. We don't create this test_ directory.

[Patrick-Browne]

@vegoutha thanks for the update. A couple of questions, how frequently does the error occur ? Do you download to the same path every time ? Does the code download multiple files for each run or a single file per run ?The SDK will not create the test_ directory during the download_file operation. Can you ensure that the full path exists prior the to calling download_file ?

[vegoutha]

@Patrick-Browne This issue was reported couple of times during internal testing.

Download Path is varying for each download request, but base path remains the same. Base Path: /opt/ibm/scoring/python/workdir/ Download Path: Base Path + batch_input_<requestID>

This code will download all the files in the COS bucket( Both Single (or) Multiple files)

I am not sure how this test_ directory. We are checking on this.

[Patrick-Browne]

@vegoutha were you able to follow up on were 'test_' node in the download path was coming from and ensure that the full path exists before starting the download ?

[barry-hueston]

Hi @vegoutha if there are no further updates, are we okay to close this ticket?

Thanks Barry

[srikrbha]

@Patrick-Browne this is the code we follow to download the files:

amz = ibm_boto3.resource('s3',
                                 endpoint_url=service_endpoint,
                                 aws_access_key_id=aws_access_key_id,
                                 aws_secret_access_key=aws_secret_access_key)
my_bucket = amz.Bucket(bucket_name)
for object in my_bucket.objects.all():
     amz.meta.client.download_file(bucket_name, object.key, download_path+'/'+object.key)

Here download_path is: /opt/ibm/scoring/python/workdir/batch_input_abe514e7e26c7c8c137e80162fdfdfe8

I believe test_/ might get added here from object.key.