search

thenickdude / snap-to-s3

Upload EBS volume snapshots to Amazon S3/Glacier
Other
114 stars 24 forks source link

TypeError: Cannot read property '1' of null #14

Closed JustOneMoreDog closed 3 years ago

JustOneMoreDog commented 3 years ago

I have been trying to figure out what is going on here for a while but am stuck due to my very limited understanding of Javascript. 

ubuntu@ip-192-168-16-140:~$ sudo snap-to-s3 --migrate --dd --snapshots snap-0059a3b2152d58613 --bucket snap-to-s3-archive --validate
sudo: unable to resolve host ip-192-168-16-140: Connection timed out
[snap-0059a3b2152d58613] Migrating snap-0059a3b2152d58613 to S3
[snap-0059a3b2152d58613] Tagging snapshot with "migrating"...
[snap-0059a3b2152d58613] A temporary volume for snap-0059a3b2152d58613 already exists, using vol-02d14a82d004733a5
[snap-0059a3b2152d58613] Volume vol-02d14a82d004733a5 is already attached here
[snap-0059a3b2152d58613] Waiting for vol-02d14a82d004733a5's partitions to become visible to the operating system...
/usr/lib/node_modules/snap-to-s3/lib/aws-tools.js:76
                                                                } if ("vol-" + matches[1] === volume.VolumeId) {
                                                                                      ^

TypeError: Cannot read property '1' of null
    at /usr/lib/node_modules/snap-to-s3/lib/aws-tools.js:76:31
    at ChildProcess.exithandler (child_process.js:294:7)
    at ChildProcess.emit (events.js:315:20)
    at ChildProcess.EventEmitter.emit (domain.js:483:12)
    at maybeClose (internal/child_process.js:1021:16)
    at Socket.<anonymous> (internal/child_process.js:443:11)
    at Socket.emit (events.js:315:20)
    at Socket.EventEmitter.emit (domain.js:483:12)
    at Pipe.<anonymous> (net.js:674:12)

So I went into the code and tried to modify it but with no luck

case "nvme":
                                                child_process.execFile("nvme", ["id-ctrl", device.DEVICEPATH], function (error, stdout, stderr) {
                                                        if (error) {
                                                                reject("Failed to nvme " + stdout + " " + stderr);
                                                        } else {
                                                                console.log("in that one function");
                                                                let matches = stdout.match(/vol([0-9a-zA-Z]+)/);
                                                                console.log(matches);
                                                                console.log("checking device");
                                                                console.log(device);
                                                                //if (matches === null){
                                                                //      console.log("it equaled null");
                                                                //} else {
                                                                        if (!matches) {
                                                                                reject("Failed to parse output of ebsnvme-id: " + stdout);
                                                                        } if (matches === null) {
                                                                                console.log("it is null");
                                                                                reject("Failed to parse output of ebsnvme-id: " + stdout);
                                                                        } if ((matches != null || matches !== null) && "vol-" + matches[1] === volume.VolumeId) {
                                                                                resolve(device);
                                                                        } else {
                                                                                resolve(null);
                                                                        }
                                                                //}
                                                        }
                                                });
                                                break;

I have no clue what I am doing with Javascript so I could be completely wrong here but it would seem like there is a missing null check and or a null case not being handled?

This is what I get when I run the above modified code

ubuntu@ip-192-168-16-140:~$ sudo snap-to-s3 --migrate --dd --snapshots snap-0059a3b2152d58613 --bucket snap-to-s3-archive --validate
sudo: unable to resolve host ip-192-168-16-140: Connection timed out
[snap-0059a3b2152d58613] Migrating snap-0059a3b2152d58613 to S3
[snap-0059a3b2152d58613] Tagging snapshot with "migrating"...
[snap-0059a3b2152d58613] A temporary volume for snap-0059a3b2152d58613 already exists, using vol-02d14a82d004733a5
[snap-0059a3b2152d58613] Volume vol-02d14a82d004733a5 is already attached here
[snap-0059a3b2152d58613] Waiting for vol-02d14a82d004733a5's partitions to become visible to the operating system...
in that one function
[
  'vol02d14a82d004733a5',
  '02d14a82d004733a5',
  index: 70,
  input: 'NVME Identify Controller:\n' +
    'vid     : 0x1d0f\n' +
    'ssvid   : 0x1d0f\n' +
    'sn      : vol02d14a82d004733a5\n' +
    'mn      : Amazon Elastic Block Store              \n' +
    'fr      : 1.0     \n' +
    'rab     : 32\n' +
    'ieee    : dc02a0\n' +
    'cmic    : 0\n' +
    'mdts    : 6\n' +
    'cntlid  : 0\n' +
    'ver     : 0\n' +
    'rtd3r   : 0\n' +
    'rtd3e   : 0\n' +
    'oaes    : 0x100\n' +
    'oacs    : 0\n' +
    'acl     : 4\n' +
    'aerl    : 0\n' +
    'frmw    : 0x3\n' +
    'lpa     : 0\n' +
    'elpe    : 0\n' +
    'npss    : 1\n' +
    'avscc   : 0x1\n' +
    'apsta   : 0\n' +
    'wctemp  : 0\n' +
    'cctemp  : 0\n' +
    'mtfa    : 0\n' +
    'hmpre   : 0\n' +
    'hmmin   : 0\n' +
    'tnvmcap : 0\n' +
    'unvmcap : 0\n' +
    'rpmbs   : 0\n' +
    'sqes    : 0x66\n' +
    'cqes    : 0x44\n' +
    'nn      : 1\n' +
    'oncs    : 0\n' +
    'fuses   : 0\n' +
    'fna     : 0\n' +
    'vwc     : 0\n' +
    'awun    : 0\n' +
    'awupf   : 0\n' +
    'nvscc   : 0\n' +
    'acwu    : 0\n' +
    'sgls    : 0\n' +
    'ps    0 : mp:0.01W operational enlat:1000000 exlat:1000000 rrt:0 rrl:0\n' +
    '          rwt:0 rwl:0 idle_power:- active_power:-\n' +
    'ps    1 : mp:0.00W operational enlat:0 exlat:0 rrt:0 rrl:0\n' +
    '          rwt:0 rwl:0 idle_power:- active_power:-\n',
  groups: undefined
]
checking device
{
  NAME: 'nvme2n1',
  FSTYPE: '',
  MOUNTPOINT: '',
  SIZE: 17179869184,
  TYPE: 'disk',
  'LOG-SEC': 512,
  'PHY-SEC': 512,
  PKNAME: '',
  PARTNAME: '',
  DEVICEPATH: '/dev/nvme2n1'
}
in that one function
null
checking device
{
  NAME: 'nvme1n1',
  FSTYPE: '',
  MOUNTPOINT: '',
  SIZE: 75000000000,
  TYPE: 'disk',
  'LOG-SEC': 512,
  'PHY-SEC': 512,
  PKNAME: '',
  PARTNAME: '',
  DEVICEPATH: '/dev/nvme1n1'
}
it is null
[snap-0059a3b2152d58613] An error occurred, tagging snapshot with "migrate" so it can be retried later
in that one function
[
  'vol0d6e4f47f0eb6a0b3',
  '0d6e4f47f0eb6a0b3',
  index: 70,
  input: 'NVME Identify Controller:\n' +
    'vid     : 0x1d0f\n' +
    'ssvid   : 0x1d0f\n' +
    'sn      : vol0d6e4f47f0eb6a0b3\n' +
    'mn      : Amazon Elastic Block Store              \n' +
    'fr      : 1.0     \n' +
    'rab     : 32\n' +
    'ieee    : dc02a0\n' +
    'cmic    : 0\n' +
    'mdts    : 6\n' +
    'cntlid  : 0\n' +
    'ver     : 0\n' +
    'rtd3r   : 0\n' +
    'rtd3e   : 0\n' +
    'oaes    : 0x100\n' +
    'oacs    : 0\n' +
    'acl     : 4\n' +
    'aerl    : 0\n' +
    'frmw    : 0x3\n' +
    'lpa     : 0\n' +
    'elpe    : 0\n' +
    'npss    : 1\n' +
    'avscc   : 0x1\n' +
    'apsta   : 0\n' +
    'wctemp  : 0\n' +
    'cctemp  : 0\n' +
    'mtfa    : 0\n' +
    'hmpre   : 0\n' +
    'hmmin   : 0\n' +
    'tnvmcap : 0\n' +
    'unvmcap : 0\n' +
    'rpmbs   : 0\n' +
    'sqes    : 0x66\n' +
    'cqes    : 0x44\n' +
    'nn      : 1\n' +
    'oncs    : 0\n' +
    'fuses   : 0\n' +
    'fna     : 0\n' +
    'vwc     : 0\n' +
    'awun    : 0\n' +
    'awupf   : 0\n' +
    'nvscc   : 0\n' +
    'acwu    : 0\n' +
    'sgls    : 0\n' +
    'ps    0 : mp:0.01W operational enlat:1000000 exlat:1000000 rrt:0 rrl:0\n' +
    '          rwt:0 rwl:0 idle_power:- active_power:-\n' +
    'ps    1 : mp:0.00W operational enlat:0 exlat:0 rrt:0 rrl:0\n' +
    '          rwt:0 rwl:0 idle_power:- active_power:-\n',
  groups: undefined
]
checking device
{
  NAME: 'nvme0n1p1',
  FSTYPE: 'ext4',
  MOUNTPOINT: '/',
  SIZE: 8588869120,
  TYPE: 'part',
  'LOG-SEC': 512,
  'PHY-SEC': 512,
  PKNAME: 'nvme0n1',
  PARTNAME: 'p1',
  DEVICEPATH: '/dev/nvme0n1p1'
}
in that one function
[
  'vol0d6e4f47f0eb6a0b3',
  '0d6e4f47f0eb6a0b3',
  index: 70,
  input: 'NVME Identify Controller:\n' +
    'vid     : 0x1d0f\n' +
    'ssvid   : 0x1d0f\n' +
    'sn      : vol0d6e4f47f0eb6a0b3\n' +
    'mn      : Amazon Elastic Block Store              \n' +
    'fr      : 1.0     \n' +
    'rab     : 32\n' +
    'ieee    : dc02a0\n' +
    'cmic    : 0\n' +
    'mdts    : 6\n' +
    'cntlid  : 0\n' +
    'ver     : 0\n' +
    'rtd3r   : 0\n' +
    'rtd3e   : 0\n' +
    'oaes    : 0x100\n' +
    'oacs    : 0\n' +
    'acl     : 4\n' +
    'aerl    : 0\n' +
    'frmw    : 0x3\n' +
    'lpa     : 0\n' +
    'elpe    : 0\n' +
    'npss    : 1\n' +
    'avscc   : 0x1\n' +
    'apsta   : 0\n' +
    'wctemp  : 0\n' +
    'cctemp  : 0\n' +
    'mtfa    : 0\n' +
    'hmpre   : 0\n' +
    'hmmin   : 0\n' +
    'tnvmcap : 0\n' +
    'unvmcap : 0\n' +
    'rpmbs   : 0\n' +
    'sqes    : 0x66\n' +
    'cqes    : 0x44\n' +
    'nn      : 1\n' +
    'oncs    : 0\n' +
    'fuses   : 0\n' +
    'fna     : 0\n' +
    'vwc     : 0\n' +
    'awun    : 0\n' +
    'awupf   : 0\n' +
    'nvscc   : 0\n' +
    'acwu    : 0\n' +
    'sgls    : 0\n' +
    'ps    0 : mp:0.01W operational enlat:1000000 exlat:1000000 rrt:0 rrl:0\n' +
    '          rwt:0 rwl:0 idle_power:- active_power:-\n' +
    'ps    1 : mp:0.00W operational enlat:0 exlat:0 rrt:0 rrl:0\n' +
    '          rwt:0 rwl:0 idle_power:- active_power:-\n',
  groups: undefined
]
checking device
{
  NAME: 'nvme0n1',
  FSTYPE: '',
  MOUNTPOINT: '',
  SIZE: 8589934592,
  TYPE: 'disk',
  'LOG-SEC': 512,
  'PHY-SEC': 512,
  PKNAME: '',
  PARTNAME: '',
  DEVICEPATH: '/dev/nvme0n1'
}
in that one function
[
  'vol02d14a82d004733a5',
  '02d14a82d004733a5',
  index: 70,
  input: 'NVME Identify Controller:\n' +
    'vid     : 0x1d0f\n' +
    'ssvid   : 0x1d0f\n' +
    'sn      : vol02d14a82d004733a5\n' +
    'mn      : Amazon Elastic Block Store              \n' +
    'fr      : 1.0     \n' +
    'rab     : 32\n' +
    'ieee    : dc02a0\n' +
    'cmic    : 0\n' +
    'mdts    : 6\n' +
    'cntlid  : 0\n' +
    'ver     : 0\n' +
    'rtd3r   : 0\n' +
    'rtd3e   : 0\n' +
    'oaes    : 0x100\n' +
    'oacs    : 0\n' +
    'acl     : 4\n' +
    'aerl    : 0\n' +
    'frmw    : 0x3\n' +
    'lpa     : 0\n' +
    'elpe    : 0\n' +
    'npss    : 1\n' +
    'avscc   : 0x1\n' +
    'apsta   : 0\n' +
    'wctemp  : 0\n' +
    'cctemp  : 0\n' +
    'mtfa    : 0\n' +
    'hmpre   : 0\n' +
    'hmmin   : 0\n' +
    'tnvmcap : 0\n' +
    'unvmcap : 0\n' +
    'rpmbs   : 0\n' +
    'sqes    : 0x66\n' +
    'cqes    : 0x44\n' +
    'nn      : 1\n' +
    'oncs    : 0\n' +
    'fuses   : 0\n' +
    'fna     : 0\n' +
    'vwc     : 0\n' +
    'awun    : 0\n' +
    'awupf   : 0\n' +
    'nvscc   : 0\n' +
    'acwu    : 0\n' +
    'sgls    : 0\n' +
    'ps    0 : mp:0.01W operational enlat:1000000 exlat:1000000 rrt:0 rrl:0\n' +
    '          rwt:0 rwl:0 idle_power:- active_power:-\n' +
    'ps    1 : mp:0.00W operational enlat:0 exlat:0 rrt:0 rrl:0\n' +
    '          rwt:0 rwl:0 idle_power:- active_power:-\n',
  groups: undefined
]
checking device
{
  NAME: 'nvme2n1p1',
  FSTYPE: 'ext4',
  MOUNTPOINT: '',
  SIZE: 17178803712,
  TYPE: 'part',
  'LOG-SEC': 512,
  'PHY-SEC': 512,
  PKNAME: 'nvme2n1',
  PARTNAME: 'p1',
  DEVICEPATH: '/dev/nvme2n1p1'
}
[snap-0059a3b2152d58613] SnapshotMigrationError: snap-0059a3b2152d58613: Failed to parse output of ebsnvme-id: NVME Identify Controller:
vid     : 0x1d0f
ssvid   : 0
sn      : AWSBB1A925AA9DB2BBB4
mn      : Amazon EC2 NVMe Instance Storage
fr      : 0
rab     : 0
ieee    : 40b4cd
cmic    : 0
mdts    : 5
cntlid  : b
ver     : 0
rtd3r   : 0
rtd3e   : 0
oaes    : 0
oacs    : 0
acl     : 3
aerl    : 4
frmw    : 0x3
lpa     : 0
elpe    : 63
npss    : 0
avscc   : 0
apsta   : 0
wctemp  : 0
cctemp  : 0
mtfa    : 0
hmpre   : 0
hmmin   : 0
tnvmcap : 0
unvmcap : 0
rpmbs   : 0
sqes    : 0x66
cqes    : 0x44
nn      : 1
oncs    : 0x4
fuses   : 0
fna     : 0
vwc     : 0
awun    : 0
awupf   : 0
nvscc   : 0
acwu    : 0
sgls    : 0
ps    0 : mp:0.00W operational enlat:0 exlat:0 rrt:0 rrl:0
          rwt:0 rwl:0 idle_power:- active_power:-

    at /usr/lib/node_modules/snap-to-s3/lib/snap-to-s3.js:1727:12
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  error: 'Failed to parse output of ebsnvme-id: NVME Identify Controller:\n' +
    'vid     : 0x1d0f\n' +
    'ssvid   : 0\n' +
    'sn      : AWSBB1A925AA9DB2BBB4\n' +
    'mn      : Amazon EC2 NVMe Instance Storage        \n' +
    'fr      : 0\n' +
    'rab     : 0\n' +
    'ieee    : 40b4cd\n' +
    'cmic    : 0\n' +
    'mdts    : 5\n' +
    'cntlid  : b\n' +
    'ver     : 0\n' +
    'rtd3r   : 0\n' +
    'rtd3e   : 0\n' +
    'oaes    : 0\n' +
    'oacs    : 0\n' +
    'acl     : 3\n' +
    'aerl    : 4\n' +
    'frmw    : 0x3\n' +
    'lpa     : 0\n' +
    'elpe    : 63\n' +
    'npss    : 0\n' +
    'avscc   : 0\n' +
    'apsta   : 0\n' +
    'wctemp  : 0\n' +
    'cctemp  : 0\n' +
    'mtfa    : 0\n' +
    'hmpre   : 0\n' +
    'hmmin   : 0\n' +
    'tnvmcap : 0\n' +
    'unvmcap : 0\n' +
    'rpmbs   : 0\n' +
    'sqes    : 0x66\n' +
    'cqes    : 0x44\n' +
    'nn      : 1\n' +
    'oncs    : 0x4\n' +
    'fuses   : 0\n' +
    'fna     : 0\n' +
    'vwc     : 0\n' +
    'awun    : 0\n' +
    'awupf   : 0\n' +
    'nvscc   : 0\n' +
    'acwu    : 0\n' +
    'sgls    : 0\n' +
    'ps    0 : mp:0.00W operational enlat:0 exlat:0 rrt:0 rrl:0\n' +
    '          rwt:0 rwl:0 idle_power:- active_power:-\n',
  snapshotID: 'snap-0059a3b2152d58613'
}

Terminating due to fatal errors.

Any help on this would be greatly appreciated

For reference the instance type I am using is M5ad.large

thenickdude commented 3 years ago

Thanks for the thorough debugging. It looks like there's a bug in my code here:

if (!matches) {
    reject("Failed to parse output of ebsnvme-id: " + stdout);
} if (matches[0] === volume.VolumeId) {
    resolve(device);
} else {
    resolve(null);
}

That "if" on the same line as the } should have an "else" written before it. However fixing that will not resolve the problem, the root cause is that your instance has NVMe instance store drives attached with names that my code doesn't expect to find (it expects all NVMe devices to have a corresponding vol-xxx name). I'll test out that instance type you're using and get the routine fixed for it.

JustOneMoreDog commented 3 years ago

Of course happy to help in any way I can. I need to archive about a thousand snapshots so this tool has been an absolute life saver. Do you have any recommendations for a good instance type that would work the best? It would seem that the tool needs a lot of hard drive speed hence why I was trying to use an NVME instance. I moved back to using a m5a.large and it is working currently

On Tue, Aug 18, 2020, 1:44 PM Nicholas Sherlock notifications@github.com wrote:

Thanks for the thorough debugging. It looks like there's a bug in my code here:

if (!matches) { reject("Failed to parse output of ebsnvme-id: " + stdout); } if (matches[0] === volume.VolumeId) { resolve(device); } else { resolve(null); }

That "if" on the same line as the } should have an "else" written before it. However fixing that will not resolve the problem, the root cause is that your instance has NVMe instance store drives attached with names that my code doesn't expect to find (it expects all NVMe devices to have a corresponding vol-xxx name). I'll test out that instance type you're using and get the routine fixed for it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/thenickdude/snap-to-s3/issues/14#issuecomment-675620401, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKBLTI7KYU66YXSOU6J5CN3SBK4ZVANCNFSM4QDS4JOQ .

thenickdude commented 3 years ago

Actually the speed is entirely capped by how long it takes Amazon to load the blocks from S3 that back the snapshot into EBS, and I don't think the instance type can speed this up.

The way to get more speed is to transfer multiple snapshots simultaneously, the speed scales up nicely with the number of simultaneous processes. You can set --upload-streams to 1 to reduce snap-to-s3's memory usage to allow you to pack more processes onto the same instance.

JustOneMoreDog commented 3 years ago

That makes complete sense actually. Very good point. I had it backwards. Thank you so much!

On Tue, Aug 18, 2020, 5:49 PM Nicholas Sherlock notifications@github.com wrote:

Actually the speed is entirely capped by how long it takes Amazon to load the blocks from S3 that back the snapshot into EBS, and I don't think the instance type can speed this up.

The way to get more speed is to transfer multiple snapshots simultaneously, the speed scales up nicely with the number of simultaneous processes. You can set --upload-streams to 1 to reduce snap-to-s3's memory usage to allow you to pack more processes onto the same instance.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/thenickdude/snap-to-s3/issues/14#issuecomment-675737375, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKBLTIYSBLJW27MTI7UQ5HLSBLZNJANCNFSM4QDS4JOQ .

thenickdude commented 3 years ago

Thanks for the bug report, this has now been fixed in snap-to-s3@0.4.4