search

IBM / ibm-cos-sdk-js

ibm-cos-sdk-js
Apache License 2.0
38 stars 19 forks source link

Not able to copy object from a bucket to another bucket (Flow Logs for VPC related) #87

Closed victorshinya closed 3 years ago

victorshinya commented 3 years ago

I am working with Flow Logs for VPC and I've developed a Serverless function to send all logs from COS to IBM Log Analysis (LogDNA), with a similar structure as my another project (logdna-cos). The function downloads the log file, unzip and format to perform a HTTP request to LogDNA through Ingestion API, then copy the object from the source bucket to another bucket within the COS instance.

The Flow Logs for VPC saves the log file in a different way, with several subfolders In a bucket. When I execute the function, it always returns a NoSuchKey: The specified key does not exist error when it performs the (COS) copyObject method. I attached the log I am receiving with the name of the file and the method I am using to call the copyObject method.

Node.js function

async function uploadAndDeleteBucket(bucketReceiver, fileName) {
  try {
    const decodedFileName = decodeURIComponent(fileName);
    console.log(`DEBUG: Uploading the log file = ${bucketReceiver}/${decodedFileName}`);
    await cos
      .copyObject({
        Bucket: BUCKET_ARCHIVE,
        CopySource: `${bucketReceiver}/${decodedFileName}`,
        Key: fileName,
      })
      .promise();
    console.log("DEBUG: Deleting the log file");
    await cos
      .deleteObject({ Bucket: bucketReceiver, Key: fileName })
      .promise();
    return { status: 200, message: "Update and delete log file DONE" };
  } catch (e) {
    console.error(e);
    return e;
  }
}

The error log

DEBUG: Uploading the log file = bucket-flow-logs-collector/ibm_vpc_flowlogs_v1/account=<account-id>/region=us-south/vpc-id=<vpc-id>/subnet-id=<vpc-subnet-id>/endpoint-type=vnics/instance-id=<instance-id>/vnic-id=<vnic-id>/record-type=egress/year=2021/month=08/day=10/hour=18/stream-id=<stream-id>/00000000.gz

NoSuchKey: The specified key does not exist.
    at Request.extractError (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/services/s3.js:588:35)
    at Request.callListeners (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/request.js:683:14)
    at Request.transition (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/state_machine.js:14:12)
    at /Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/request.js:685:12)
    at Request.callListeners (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/sequential_executor.js:116:18)
    at Request.emit (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/request.js:683:14)
    at Request.transition (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/state_machine.js:14:12)
    at /Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/request.js:685:12)
    at Request.callListeners (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/sequential_executor.js:116:18)
    at callNextListener (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/sequential_executor.js:96:12)
    at IncomingMessage.onEnd (/Users/victorshinya/Developer/flow-logs-logdna/node_modules/ibm-cos-sdk/lib/event_listeners.js:305:13)
    at IncomingMessage.emit (node:events:377:35)
    at IncomingMessage.emit (node:domain:470:12) {
  code: 'NoSuchKey',
  region: null,
  time: 2021-08-21T15:43:36.665Z,
  requestId: '5e3b67d2-c993-41e2-8b2b-ebec82b6d253',
  extendedRequestId: undefined,
  cfId: undefined,
  statusCode: 404,
  retryable: false,
  retryDelay: 37.04740133118023
}
DEBUG: downloadAndSend = "The specified key does not exist."
victorshinya commented 3 years ago

Ignore the const decodedFileName = decodeURIComponent(fileName); line. I tried to decode the URI before the copyObject call method, but the error is the same as without it.

IBMeric commented 3 years ago

A few things to check:

  1. Does the source bucket actually have the data you expect when doing a listing?
  2. Are you handling encoding special characters correctly? Equal signs and other characters are problematic: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html. URL encoding is required for CopySource.
victorshinya commented 3 years ago

@IBMeric yes, the object exists. Before the execution of CopyObject method, I actually access the object, download it and process so the Serverless function can send the logs to IBM Log Analysis (LogDNA).

And thanks for your answer. I had to enforce the encoding the fileName (including folders and subfolders) and it worked.

Just replaced the decodeURIComponent() to const encodedURI = encodeURI(fileName); and it worked.