NoSuchKey: The specified key does not exist

[c1tt1]

I am trying out this sdk to upload a file and have been facing this error so far:

NoSuchKey: The specified key does not exist

here is my code example:

const ibm = require('ibm-cos-sdk')
const fs = require('fs')
const path = require('path')

const config = {
  "apiKeyId": "X6iH....",
  "endpoint": "s3-api.us-geo.objectstorage.softlayer.net",
  "ibmAuthEndpoint": "https://iam.ng.bluemix.net/oidc/token",
  "serviceInstanceId": "crn:v1:bluemix:public:cloud-object-storage:global:a/b0dbcb17e63ac385fbd92cc44091d078:30adf4ce-69cb-419a-a4e2-ad7315f84f81::"
}
const cos = new ibm.S3(config);

const file = fs.readFileSync(path.join(__dirname, '/cos.txt'))

const params = {
  ACL: 'private',
  Body: file,//fs.createReadStream(file),
  Bucket: 'gaia',
  Key: `gaia/key`,
  ContentType: file.type,
}

cos.upload(params, (err, data) => {
  if (err) {
    console.log(err)
  }
  console.log(data);
})

I am wondering where the error is, if it is the apiKeyId or Key.

[seamus-mcgrath]

Hi @stern0, I think the issue is the quotation marks around your key value within params. Try correcting these & let me know if it works

[c1tt1]

@smcgrath-IBM the quotation marks isn't an issue, it's standard plain old javascript. The idea of not using quotation marks is modern javascript.

Using quotation marks or not produces the same issue.

[widget-]

I don't think you can use non-public or public-read ACLs. Generally you'd use IAM permissions attached to the API key for that instead.

But NoSuchKey errors should be for the object key (name). Is there something attempting to verify that the object was uploaded after the given code?

[c1tt1]

It throws the same error whether specifying ACL or not and same with public-read.

[c1tt1]

The aws sdk doesn't have this problem and this a fork of it.

[paul-carron]

@stern0 is this still ann issue? If so try changing:

"serviceInstanceId": "crn:v1:bluemix:public:cloud-object-storage:global:a/b0dbcb17e63ac385fbd92cc44091d078:30adf4ce-69cb-419a-a4e2-ad7315f84f81::"

for:

"30adf4ce-69cb-419a-a4e2-ad7315f84f81"

[c1tt1]

@paul-carron same error

[paul-carron]

@stern0 in that case could you please check the following:

1. cos.txt exists in the same folder as the js file that contains the above code.
2. Bucket gaia exists.
3. You are using the correct apiKeyId. I'm sure this is OK though. I'd expect a different error if it was wrong.

Also, could you confirm that you're trying to create object gaia/key inside bucket gaia and that you're not trying to create bucket gaia with object key inside it? I.E. You want https://s3-api.us-geo.objectstorage.softlayer.net/gaia/gaia/key rather than https://s3-api.us-geo.objectstorage.softlayer.net/gaia/key.

[c1tt1]

@paul-carron all the details are correct, the bucket exists and file is in same folder with the code, the apikey would throw a different error if wrong.

[paul-carron]

@stern0, are you able to do any other operations like list buckets or list objects?

[c1tt1]

@paul-carron no I can't

[paul-carron]

@stern0, please copy and paste the output from the above code aswell as the output from listBuckets/listObjects.

[c1tt1]

@paul-carron here is the output:

{ NoSuchKey: The specified key does not exist.
    at Request.extractError (/home/v/Desktop/ibm/node_modules/ibm-cos-sdk/lib/services/s3.js:537:35)
    at Request.callListeners (/home/v/Desktop/ibm/node_modules/ibm-cos-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/home/v/Desktop/ibm/node_modules/ibm-cos-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/home/v/Desktop/ibm/node_modules/ibm-cos-sdk/lib/request.js:671:14)
    at Request.transition (/home/v/Desktop/ibm/node_modules/ibm-cos-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/home/v/Desktop/ibm/node_modules/ibm-cos-sdk/lib/state_machine.js:14:12)
    at /home/v/Desktop/ibm/node_modules/ibm-cos-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/home/v/Desktop/ibm/node_modules/ibm-cos-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/home/v/Desktop/ibm/node_modules/ibm-cos-sdk/lib/request.js:673:12)
    at Request.callListeners (/home/v/Desktop/ibm/node_modules/ibm-cos-sdk/lib/sequential_executor.js:115:18)
  message: 'The specified key does not exist.',
  code: 'NoSuchKey',
  region: null,
  time: 2018-05-11T15:46:13.406Z,
  requestId: '33a23b75-c01b-4ae7-97cc-480862612091',
  extendedRequestId: undefined,
  cfId: undefined,
  statusCode: 404,
  retryable: false,
  retryDelay: 27.081495474880036 }
undefined

It's been a while trying to figure this out, I have been better with other cloud solutions and have never had problems even alibaba cloud. This is quite simple but the sdk isn't well written and a fork of aws. If this cannot be fixed or sorted out I might as well close this issue.

[paul-carron]

@stern0, it works when I take your code and change the config to suit my credentials.

Maybe double check that the credentials you are using have permissions to write to the bucket.

Also, try this and let me know if it works. Add your apiKeyId and serviceInstanceId to the relevant places in the below code, then run it.

const config = {
  "apiKeyId": "<apiKeyId Here>",
  "endpoint": "s3-api.us-geo.objectstorage.softlayer.net",
  "ibmAuthEndpoint": "https://iam.ng.bluemix.net/oidc/token",
  "serviceInstanceId": "<serviceInstanceId Here>"
}

const cos = new ibm.S3(config);

var bucket = 'pc-test1';

function doCreateBucket() {
  console.log('Creating bucket');
  return cos.createBucket({
      Bucket: bucket,
      CreateBucketConfiguration: {
      },
  }).promise();
}
doCreateBucket()
  .then(function() {
      console.log('Finished!');
  });```

[c1tt1]

@paul-carron I keep getting the same error, I might have to close this since it looks unsolvable. The sdk is a fork and not well written, on the other the documentation is poor with no clarity and guide. Thanks for the help but it's just time consuming.

[paul-carron]

@stern0, if you can't create a bucket with the above code I can only suggest you double check the credentials. The code works.

Your code also works for me so without access to your setup I can only suggest checking the apiKeyId and serviceInstanceKey, ensure cos.txt exists and is in the correct location and that the bucket also exists and if it does exist ensure that the credentials you're using have permissions to write to that bucket.

[c1tt1]

@paul-carron I have a lite account could that be the issue?

[paul-carron]

@stern0 No. Thats what I'm using to test.

Try this(although I suspect you would get a different error if this was the issue):

1. Go to your buket in IBM Cloud. Click Policies > Service ID's > Manage Service ID's > Access Policies.
2. Select the Service ID you're using.
3. Check permissions on any roles there.

What happens after this? Does your code check the object in any way?

[paul-carron]

@stern0 did you have any success with the above?

[TatyanaBol]

@paul-carron, I have exactly the same issue error: NoSuchKey: The specified key does not exist when trying to upload the first object to the newly created instance and bucket My code works with another instance and its bucket without any problem. I understand that error is that object doesn't exist in the bucket, but it indeed doesn't exist, I'm trying to upload it! What can be a problem?

[TatyanaBol]

in your examples why are you using endpoint "ibmAuthEndpoint": "https://iam.ng.bluemix.net/oidc/token" ? why not "https://iam.bluemix.net/identity/token" ? (global endpoint) is it intentional?

[TatyanaBol]

The problem is solved by fixing cos endpoint (cross-region to regional)

[zhuojc]

@stern0 is your issue similar to @TatyanaBol 's?

[van-ibm]

Why was this closed? What's the resolution? I ran into this same issue. After using working code for weeks, I created another bucket (Regional) to run a different scenario. I immediately hit the same NoSuchKeyError. I get failures for Single Site buckets as well.

Does this imply that the SDK will only support Cross Region buckets? Or is additional information needed in the options? This has downstream effects on other IBM services referencing this SDK.

[paul-carron]

@van-ibm what endpoint are you using?

[van-ibm]

@paul-carron Doh! The consuming code of the cos-sdk has a default to the us-geo. Overriding the endpoint indeed fixes the issue. So there's some user error here, but thoughts on whether a better error message could be surfaced. Given the supplied endpoint is incorrect, is it that the bucket isn't found? Could we say, Bucket not found - confirm bucket ${bucket} exists at the endpoint ${endpoint}?

[paul-carron]

Thanks for the feedback @van-ibm. I'll suggest that and see if a better error could be returned.

[guedes]

Just for someone looking for this, I was using a US Cross Region Endpoints, BUT when I changed to some of US Regional Endpoints it works. I'm using this link as reference: https://console.bluemix.net/docs/services/cloud-object-storage/basics/endpoints.html#select-regions-and-endpoints

[sat-ch]

Thank you @guedes ! It worked when I followed your message. I changed the end point to US Regional end point.

[ddaskal]

worked for me by changing: cos_host = (endpoints['service-endpoints']['cross-region']['us']['public']['us-geo']) to cos_host = (endpoints['service-endpoints']['regional']['us-south']['public']['us-south'])

[m0deller]

Came across this NoSuchKey issue when working through the edx python course. To fix I aligned the endpoint URL with the bucket location I.e. if bucket = us-south (regional , i used the service_endpoint = 'https://s3.us-south.cloud-object-storage.appdomain.cloud'

[asheteh]

Thank @m0deller

[youming-lin]

I had the exact same issue; changing the URL fixed it for me.

+1 on returning better error messages. I would not have figured this out without chancing upon this thread because none of the AmazonS3 questions mention this.

[kendo35822]

I am trying out this sdk to upload a file and have been facing this error so far:

NoSuchKey: The specified key does not exist

here is my code example:

const ibm = require('ibm-cos-sdk')
const fs = require('fs')
const path = require('path')

const config = {
  "apiKeyId": "X6iH....",
  "endpoint": "s3-api.us-geo.objectstorage.softlayer.net",
  "ibmAuthEndpoint": "https://iam.ng.bluemix.net/oidc/token",
  "serviceInstanceId": "crn:v1:bluemix:public:cloud-object-storage:global:a/b0dbcb17e63ac385fbd92cc44091d078:30adf4ce-69cb-419a-a4e2-ad7315f84f81::"
}
const cos = new ibm.S3(config);

const file = fs.readFileSync(path.join(__dirname, '/cos.txt'))

const params = {
  ACL: 'private',
  Body: file,//fs.createReadStream(file),
  Bucket: 'gaia',
  Key: `gaia/key`,
  ContentType: file.type,
}

cos.upload(params, (log, data) => {
  if (log) {
    console.log(“ “)
  }
  console.log(data);
})

I am wondering where the error is, if it is the apiKeyId or Key.

[kendo35822]

I am trying out this sdk to upload a file and have been facing this error so far:

NoSuchKey: The specified key does not exist

here is my code example:

const ibm = require('ibm-cos-sdk')
const fs = require('fs')
const path = require('path')

const config = {
  "apiKeyId": "X6iH....",
  "endpoint": "s3-api.us-geo.objectstorage.softlayer.net",
  "ibmAuthEndpoint": "https://iam.ng.bluemix.net/oidc/token",
  "serviceInstanceId": "crn:v1:bluemix:public:cloud-object-storage:global:a/b0dbcb17e63ac385fbd92cc44091d078:30adf4ce-69cb-419a-a4e2-ad7315f84f81::"
}
const cos = new ibm.S3(config);

const file = fs.readFileSync(path.join(__dirname, '/cos.txt'))

const params = {
  ACL: 'private',
  Body: file,//fs.createReadStream(file),
  Bucket: 'gaia',
  Key: `gaia/key`,
  ContentType: file.type,
}

cos.upload(params, (log, data) => {
  if (log) {
    console.log(data);
  }
  console.log(data);
})

I am wondering where the error is, if it is the apiKeyId or Key.

[kendo35822]

I am trying out this sdk to upload a file and have been facing this error so far:

NoSuchKey: The specified key does not exist

here is my code example:

const ibm = require('ibm-cos-sdk')
const fs = require('fs')
const path = require('path')

const config = {
  "apiKeyId": "X6iH....",
  "endpoint": "s3-api.us-geo.objectstorage.softlayer.net",
  "ibmAuthEndpoint": "https://iam.ng.bluemix.net/oidc/token",
  "serviceInstanceId": "crn:v1:bluemix:public:cloud-object-storage:global:a/b0dbcb17e63ac385fbd92cc44091d078:30adf4ce-69cb-419a-a4e2-ad7315f84f81::"
}
const cos = new ibm.S3(config);

const file = fs.readFileSync(path.join(__dirname, '/cos.txt'))

const params = {
  ACL: 'private',
  Body: file,//fs.createReadStream(file),
  Bucket: 'gaia',
  Key: `gaia/key`,
  ContentType: file.type,
}

cos.upload(params, (log, data) => {
  if (data) {
    console.log(data);
  }
  console.log(data);
})

I am wondering where the error is, if it is the apiKeyId or Key.

[x-raay]

1. Copy service credentials as shown above
2. Hit the URL specified in the endpoints key
3. Find a regional endpoint that's same as your bucket location

and use it for endpoint in your config.

[engineerpassion]

I tried the above-mentioned steps of using the URL of the location where the bucket resides, but I'm still seeing the issue.

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ibm_botocore/client.py in _make_api_call(self, operation_name, api_params)
    659             error_code = parsed_response.get("Error", {}).get("Code")
    660             error_class = self.exceptions.from_code(error_code)
--> 661             raise error_class(parsed_response, operation_name)
    662         else:
    663             return parsed_response

NoSuchKey: An error occurred (NoSuchKey) when calling the PutObject operation: The specified key does not exist.

Here is my code:

import ibm_boto3
from ibm_botocore.client import Config, ClientError

cos_client = ibm_boto3.resource("s3",
    ibm_api_key_id=COS_API_KEY_ID,
    ibm_service_instance_id=COS_RESOURCE_CRN,
    ibm_auth_endpoint="https://iam.bluemix.net/oidc/token",
    config=Config(signature_version="oauth"),
    endpoint_url=COS_ENDPOINT
)
with open(training_data_file_name, "rb") as file_data:
    cos_client.Object('credit-risk-training-data', training_data_file_name).upload_fileobj(
        Fileobj=file_data
    )

By bucket resides in us-geo and I'm using https://s3.us.cloud-object-storage.appdomain.cloud as the COS_ENDPOINT.