search

thenickdude / snap-to-s3

Upload EBS volume snapshots to Amazon S3/Glacier
Other
114 stars 24 forks source link

Getting an error on while snap-to-s3 trying to mount the snapshot #9

Closed santoshkatageri closed 3 years ago

santoshkatageri commented 4 years ago

I assume it's waiting for the previous volume to be removed completely.

Not sure if it's caused by the snapshot I am trying to copy. I am getting the following error.

`> { Error: snap-xxxx: InvalidParameterValue: Invalid value '/dev/sdi' for unixDevice. Attachment point /dev/sdi is already in use

at SnapshotMigrationError (/usr/lib/node_modules/snap-to-s3/lib/snap-to-s3.js:1957:3)
at _raceToMarkSnapshot.then.then (/usr/lib/node_modules/snap-to-s3/lib/snap-to-s3.js:1691:12)
at process._tickDomainCallback (internal/process/next_tick.js:135:7)

error: { InvalidParameterValue: Invalid value '/dev/sdi' for unixDevice. Attachment point /dev/sdi is already in use at Request.extractError (/usr/lib/node_modules/snap-to-s3/node_modules/aws-sdk/lib/services/ec2.js:50:35)

   at Request.callListeners (/usr/lib/node_modules/snap-to-s3/node_modules/aws-sdk/lib/sequential_executor.js:106:20)

   at Request.emit (/usr/lib/node_modules/snap-to-s3/node_modules/aws-sdk/lib/sequential_executor.js:78:10)

   at Request.emit (/usr/lib/node_modules/snap-to-s3/node_modules/aws-sdk/lib/request.js:683:14)

   at Request.transition (/usr/lib/node_modules/snap-to-s3/node_modules/aws-sdk/lib/request.js:22:10)

   at AcceptorStateMachine.runTo (/usr/lib/node_modules/snap-to-s3/node_modules/aws-sdk/lib/state_machine.js:14:12)

   at /usr/lib/node_modules/snap-to-s3/node_modules/aws-sdk/lib/state_machine.js:26:10

   at Request.<anonymous> (/usr/lib/node_modules/snap-to-s3/node_modules/aws-sdk/lib/request.js:38:9)

   at Request.<anonymous> (/usr/lib/node_modules/snap-to-s3/node_modules/aws-sdk/lib/request.js:685:12)

   at Request.callListeners (/usr/lib/node_modules/snap-to-s3/node_modules/aws-sdk/lib/sequential_executor.js:116:18)

 message: 'Invalid value \'/dev/sdi\' for unixDevice. Attachment point /dev/sdi is already in use',

 code: 'InvalidParameterValue',

 time: 2019-09-23T06:29:45.275Z,

 requestId: ‘a737-xxxx-94-aa',

 statusCode: 400,

 retryable: false,

 retryDelay: 58.26884347307608 },

snapshotID: 'snap-xxxx' }

Terminating due to fatal errors.`

It would be very helpful if you can tell if I need to make any changes on the AWS side as well.

thenickdude commented 4 years ago

Which instance type are you using?

santoshkatageri commented 4 years ago

Thanks for the response @thenickdude

Instance type is c3.4xlarge AMI - baked from Amazon Linux 2

santoshkatageri commented 4 years ago

I can see that this issue popping up whenever I run multiple snap-to-s3 jobs and the new job was unable to figure out the next mount point exactly like /dev/sdh, /dev/sdi, /dev/sdj, etc.

thenickdude commented 4 years ago

I couldn't reproduce this, can you let me know how many simultaneous snap-to-s3 jobs you are running and if there is any delay between launches of snap-to-s3?

It's possible that the jobs are either exhausting the total mountpoint pool (it only attempts to use letters f-z), or that two jobs attempt to claim the same mountpoint at the same moment and only one of them wins. If I know which is the case I can add retry logic in here to fix it.

santoshkatageri commented 4 years ago

I am currently running 4 snap-to-s3 jobs simultaneously. I am getting this error when I run next snap-to-s3 job within a short time after the previous one.

It will not through any error if I run it again.

thenickdude commented 4 years ago

I've now pushed a new version 0.4.3 which should fix this issue (by adding a retry loop to the attachment procedure), please try it out.

santoshkatageri commented 4 years ago

Thanks for the new change you pushed. This solved the issue.