Error during npm install

[sslupsky]

During the npm install step (Amazon Linux) I get an error message:

[root@ip-10-0-0-22 ~]# npm install -g snap-to-s3 npm ERR! Error while executing: npm ERR! /usr/bin/git ls-remote -h -t ssh://git@github.com/thenickdude/node-progress.git npm ERR! npm ERR! fatal: Cannot come back to cwd: Permission denied npm ERR! npm ERR! exited with error code: 128

npm ERR! A complete log of this run can be found in: npm ERR! /root/.npm/_logs/2017-10-08T18_24_59_631Z-debug.log

Here is a copy of the log file:

0 info it worked if it ends with ok 1 verbose cli [ '/root/.nvm/versions/node/v8.6.0/bin/node', 1 verbose cli '/root/.nvm/versions/node/v8.6.0/bin/npm', 1 verbose cli 'install', 1 verbose cli '-g', 1 verbose cli 'snap-to-s3' ] 2 info using npm@5.3.0 3 info using node@v8.6.0 4 verbose npm-session b828424b2b1dccaa 5 silly install loadCurrentTree 6 silly install readGlobalPackageData 7 http fetch GET 304 https://registry.npmjs.org/snap-to-s3 436ms (from cache) 8 silly pacote tag manifest for snap-to-s3@latest fetched in 468ms 9 silly install loadIdealTree 10 silly install cloneCurrentTreeToIdealTree 11 silly install loadShrinkwrap 12 silly install loadAllDepsIntoIdealTree 13 silly resolveWithNewModule snap-to-s3@0.3.0 checking installable status 14 http fetch GET 304 https://registry.npmjs.org/aws-sdk 81ms (from cache) 15 http fetch GET 304 https://registry.npmjs.org/deep-equal 83ms (from cache) 16 http fetch GET 304 https://registry.npmjs.org/clone 85ms (from cache) 17 http fetch GET 304 https://registry.npmjs.org/command-line-args 86ms (from cache) 18 http fetch GET 304 https://registry.npmjs.org/command-line-usage 85ms (from cache) 19 silly pacote range manifest for aws-sdk@^2.55.0 fetched in 89ms 20 silly resolveWithNewModule aws-sdk@2.130.0 checking installable status 21 http fetch GET 304 https://registry.npmjs.org/filesize 88ms (from cache) 22 silly pacote range manifest for deep-equal@^1.0.1 fetched in 90ms 23 silly resolveWithNewModule deep-equal@1.0.1 checking installable status 24 silly pacote range manifest for clone@^2.1.1 fetched in 93ms 25 silly resolveWithNewModule clone@2.1.1 checking installable status 26 silly pacote range manifest for command-line-args@^4.0.4 fetched in 94ms 27 silly resolveWithNewModule command-line-args@4.0.7 checking installable status 28 silly pacote range manifest for command-line-usage@^4.0.0 fetched in 94ms 29 silly resolveWithNewModule command-line-usage@4.0.1 checking installable status 30 silly pacote range manifest for filesize@^3.5.9 fetched in 108ms 31 silly resolveWithNewModule filesize@3.5.10 checking installable status 32 silly fetchPackageMetaData error for progress@github:thenickdude/node-progress#f901750478a76057b9271bda333dd1dcdd5406dd Error while executing: 32 silly fetchPackageMetaData /usr/bin/git ls-remote -h -t ssh://git@github.com/thenickdude/node-progress.git 32 silly fetchPackageMetaData 32 silly fetchPackageMetaData fatal: Cannot come back to cwd: Permission denied 32 silly fetchPackageMetaData 32 silly fetchPackageMetaData exited with error code: 128 33 http fetch GET 304 https://registry.npmjs.org/binary-split 160ms (from cache) 34 http fetch GET 304 https://registry.npmjs.org/csv-parse 159ms (from cache) 35 http fetch GET 304 https://registry.npmjs.org/mkdirp 73ms (from cache) 36 silly pacote range manifest for binary-split@^1.0.3 fetched in 165ms 37 silly resolveWithNewModule binary-split@1.0.3 checking installable status 38 silly pacote range manifest for csv-parse@^1.2.0 fetched in 164ms 39 silly resolveWithNewModule csv-parse@1.2.4 checking installable status 40 http fetch GET 304 https://registry.npmjs.org/moment 74ms (from cache) 41 silly pacote range manifest for mkdirp@^0.5.1 fetched in 81ms 42 silly resolveWithNewModule mkdirp@0.5.1 checking installable status 43 http fetch GET 304 https://registry.npmjs.org/js-logger 163ms (from cache) 44 silly pacote range manifest for moment@^2.18.1 fetched in 83ms 45 silly resolveWithNewModule moment@2.18.1 checking installable status 46 http fetch GET 304 https://registry.npmjs.org/gunzip-maybe 171ms (from cache) 47 silly pacote range manifest for js-logger@^1.3.0 fetched in 171ms 48 silly resolveWithNewModule js-logger@1.4.1 checking installable status 49 http fetch GET 304 https://registry.npmjs.org/multipipe 89ms (from cache) 50 http fetch GET 304 https://registry.npmjs.org/object.values 89ms (from cache) 51 silly pacote range manifest for gunzip-maybe@^1.4.0 fetched in 178ms 52 silly resolveWithNewModule gunzip-maybe@1.4.1 checking installable status 53 silly fetchPackageMetaData error for tar-stream@github:thenickdude/tar-stream#3160e7d60fe142f04e126b8b261248d023200e1b Error while executing: 53 silly fetchPackageMetaData /usr/bin/git ls-remote -h -t ssh://git@github.com/thenickdude/tar-stream.git 53 silly fetchPackageMetaData 53 silly fetchPackageMetaData fatal: Cannot come back to cwd: Permission denied 53 silly fetchPackageMetaData 53 silly fetchPackageMetaData exited with error code: 128 54 http fetch GET 304 https://registry.npmjs.org/sprintf-js 51ms (from cache) 55 silly pacote range manifest for multipipe@^1.0.2 fetched in 93ms 56 silly resolveWithNewModule multipipe@1.0.2 checking installable status 57 silly pacote range manifest for object.values@^1.0.4 fetched in 94ms 58 silly resolveWithNewModule object.values@1.0.4 checking installable status 59 silly pacote range manifest for sprintf-js@^1.1.0 fetched in 54ms 60 silly resolveWithNewModule sprintf-js@1.1.1 checking installable status 61 http fetch GET 304 https://registry.npmjs.org/which 28ms (from cache) 62 silly pacote range manifest for which@^1.2.14 fetched in 30ms 63 silly resolveWithNewModule which@1.3.0 checking installable status 64 http fetch GET 304 https://registry.npmjs.org/rmdir 148ms (from cache) 65 silly pacote range manifest for rmdir@^1.2.0 fetched in 150ms 66 silly resolveWithNewModule rmdir@1.2.0 checking installable status 67 verbose stack Error: exited with error code: 128 67 verbose stack at ChildProcess.onexit (/root/.nvm/versions/node/v8.6.0/lib/node_modules/npm/node_modules/mississippi/node_modules/end-of-stream/index.js:39:36) 67 verbose stack at emitTwo (events.js:125:13) 67 verbose stack at ChildProcess.emit (events.js:213:7) 67 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12) 68 verbose cwd /root 69 verbose Linux 4.9.51-10.52.amzn1.x86_64 70 verbose argv "/root/.nvm/versions/node/v8.6.0/bin/node" "/root/.nvm/versions/node/v8.6.0/bin/npm" "install" "-g" "snap-to-s3" 71 verbose node v8.6.0 72 verbose npm v5.3.0 73 error Error while executing: 73 error /usr/bin/git ls-remote -h -t ssh://git@github.com/thenickdude/node-progress.git 73 error 73 error fatal: Cannot come back to cwd: Permission denied 73 error 73 error exited with error code: 128 74 verbose exit [ 1, true ]

[thenickdude]

Do you have git installed? (I thought it was installed by default, but maybe not)

[sslupsky]

Hi, Yes, I performed the step "sudo yum install git".

Though I think I recall that I did the installation using sudo (and not the interactive sudo session). I tried to "reinstall" it using an interactive sudo session and it said it was already installed.

[thenickdude]

Hm, if you run this command on its own, do you get any error output?

/usr/bin/git ls-remote -h -t ssh://git@github.com/thenickdude/node-progress.git

[sslupsky]

I should have tried that. I will fire up the EC2 instance again and try that and let you know.

[sslupsky]

Fired up the instance again and installed lz4 and git while in sudo interactive mode. Received the same error mentioned earlier. When I tried the /usr/bin/git there was no error message but nothing appeared to happen ... the command hung. I had to ctrl-c to get back to the command prompt:

[root@ip-10-0-0-63 ~]# /usr/bin/git ls-remote -h -t ssh://git@github.com/thenickdude/node-progress.git ^C

[sslupsky]

Give me a sec ... there may be some sort of network issue with the instance. I can ssh into my instance fine but for some reason the instance cannot ping google.com ... I'll try to figure out what is going and get back to you.

[thenickdude]

Ah yep, that was my next question, heh. I think the instance either needs to be assigned a public IP address, or there needs to be a NAT instance in your VPC to be able to access the internet.

On 9 Oct. 2017 8:24 am, "sslupsky" notifications@github.com wrote:

Give me a sec ... there may be some sort of network issue with the instance. I can ssh into my instance fine but for some reason the instance cannot ping google.com ... I'll try to figure out what is going and get back to you.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/thenickdude/snap-to-s3/issues/3#issuecomment-335031401, or mute the thread https://github.com/notifications/unsubscribe-auth/AB1Rg6Qe--pk9zjcdWlMUS1jJohaKfbXks5sqSFygaJpZM4PxzYr .

[thenickdude]

Alternatively you can install snap-to-s3 locally on your machine and upload that entire folder to your instance (it's a ton of tiny files so it's better to tar it up first).

EDIT: Although in this case you would also have to set up a VPC endpoint for S3, which I've not tried before myself.

On 9 Oct. 2017 8:31 am, "Nicholas Sherlock" n.sherlock@gmail.com wrote:

Ah yep, that was my next question, heh. I think the instance either needs to be assigned a public IP address, or there needs to be a NAT instance in your VPC to be able to access the internet.

On 9 Oct. 2017 8:24 am, "sslupsky" notifications@github.com wrote:

Give me a sec ... there may be some sort of network issue with the instance. I can ssh into my instance fine but for some reason the instance cannot ping google.com ... I'll try to figure out what is going and get back to you.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/thenickdude/snap-to-s3/issues/3#issuecomment-335031401, or mute the thread https://github.com/notifications/unsubscribe-auth/AB1Rg6Qe--pk9zjcdWlMUS1jJohaKfbXks5sqSFygaJpZM4PxzYr .

[sslupsky]

Yeah, there were some weird ACL's on the VPC. I created a new VPC and launched an instance again. I have outgoing network working now. I am now seeing some sort of permissions error. I tried it on the EC2 instance and on my local machine and get the same error:

git ls-remote -h -t ssh://git@github.com/thenickdude/node-progress.git Permission denied (publickey). fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

If I use the GitHub HTTPS URL, I see the following:

[root@ip-10-0-0-111 ~]# /usr/bin/git ls-remote -h -t https://github.com/thenickdude/node-progress.git f901750478a76057b9271bda333dd1dcdd5406dd refs/heads/master 2d87576b9dd831de221e2024e640ee727dff583f refs/heads/time-format 684f627ab1cee46ae0ae09095dd8550f3788e6b6 refs/tags/0.0.2 e6194546b6f48fd4ecd37a63238c04a3281a5e45 refs/tags/0.0.3 d83fa84f73b4dadd86d03dfb6c1bf832eb713dbd refs/tags/0.0.4 aa38082c4194bcadddf2df6b260fba2f5623d2f9 refs/tags/0.0.5 2b48b1d6cb42fe9352d2a1a87f8aad4a2e8fc726 refs/tags/0.0.5^{} 8628a2c919928181804b5ffd0557095df6103a54 refs/tags/0.1.0 5a12cafdee006ce4dd8624783d483b8371d32ed8 refs/tags/0.1.0^{} 783ca37c06f0f49a49b6efbbb0957806d849f5be refs/tags/1.0.0 0c0066de85818485257b1c20476f485928c97037 refs/tags/1.0.0^{} 1ac9fec5866152e4f51806a009d1761269326295 refs/tags/1.0.1 8cf6883d06e419caec653e10f0e5ef253c5033f0 refs/tags/1.0.1^{} 5641744a66ce4e024d495ae7fbfc8dda56eca9ab refs/tags/1.1.0 66b7f73c93d5bb89ed96709937656a0b05e4c2a8 refs/tags/1.1.0^{} aeac72488eac33b0572928a65d248ef6e764e78a refs/tags/1.1.1 26510351b88188c6e5273fb86cd558ac1cdaafb8 refs/tags/1.1.2 496e7cf984dc039df74bf72d8e7aa8fad3d94cce refs/tags/1.1.4 44e98342224e0f77bfa8c8105f428df36bdda3e1 refs/tags/1.1.4^{} 66495c58fa48cf0f35a894a791c9baad3ad7d1b2 refs/tags/1.1.5 fe178d7f9cff9903b93eae660eefa7e3ed2641ff refs/tags/1.1.8 6b9524c0d07df9555d20ae95c65918020c50e3e2 refs/tags/1.1.8^{} d84326ed9ab7720592b6bbc9c108849cd2a79908 refs/tags/2.0.0

[thenickdude]

Okay yeah I thought it was weird that npm picked the SSH connection method instead of HTTPS too (SSH seems like it requires a key stored at GitHub). I'll poke around with it here and see if I can get it to switch.

On 9 Oct. 2017 9:20 am, "sslupsky" notifications@github.com wrote:

Yeah, there were some weird ACL's on the VPC. I created a new VPC and launched an instance again. I have outgoing network working now. I am now seeing some sort of permissions error. I tried it on the EC2 instance and on my local machine and get the same error:

git ls-remote -h -t ssh://git@github.com/thenickdude/node-progress.git Permission denied (publickey). fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

If I use the GitHub HTTPS URL, I see the following:

[root@ip-10-0-0-111 ~]# /usr/bin/git ls-remote -h -t https://github.com/ thenickdude/node-progress.git f901750478a76057b9271bda333dd1dcdd5406dd refs/heads/master 2d87576b9dd831de221e2024e640ee727dff583f refs/heads/time-format 684f627ab1cee46ae0ae09095dd8550f3788e6b6 refs/tags/0.0.2 e6194546b6f48fd4ecd37a63238c04a3281a5e45 refs/tags/0.0.3 d83fa84f73b4dadd86d03dfb6c1bf832eb713dbd refs/tags/0.0.4 aa38082c4194bcadddf2df6b260fba2f5623d2f9 refs/tags/0.0.5 2b48b1d6cb42fe9352d2a1a87f8aad4a2e8fc726 refs/tags/0.0.5^{} 8628a2c919928181804b5ffd0557095df6103a54 refs/tags/0.1.0 5a12cafdee006ce4dd8624783d483b8371d32ed8 refs/tags/0.1.0^{} 783ca37c06f0f49a49b6efbbb0957806d849f5be refs/tags/1.0.0 0c0066de85818485257b1c20476f485928c97037 refs/tags/1.0.0^{} 1ac9fec5866152e4f51806a009d1761269326295 refs/tags/1.0.1 8cf6883d06e419caec653e10f0e5ef253c5033f0 refs/tags/1.0.1^{} 5641744a66ce4e024d495ae7fbfc8dda56eca9ab refs/tags/1.1.0 66b7f73c93d5bb89ed96709937656a0b05e4c2a8 refs/tags/1.1.0^{} aeac72488eac33b0572928a65d248ef6e764e78a refs/tags/1.1.1 26510351b88188c6e5273fb86cd558ac1cdaafb8 refs/tags/1.1.2 496e7cf984dc039df74bf72d8e7aa8fad3d94cce refs/tags/1.1.4 44e98342224e0f77bfa8c8105f428df36bdda3e1 refs/tags/1.1.4^{} 66495c58fa48cf0f35a894a791c9baad3ad7d1b2 refs/tags/1.1.5 fe178d7f9cff9903b93eae660eefa7e3ed2641ff refs/tags/1.1.8 6b9524c0d07df9555d20ae95c65918020c50e3e2 refs/tags/1.1.8^{} d84326ed9ab7720592b6bbc9c108849cd2a79908 refs/tags/2.0.0

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/thenickdude/snap-to-s3/issues/3#issuecomment-335035213, or mute the thread https://github.com/notifications/unsubscribe-auth/AB1Rg_Ew4w9LyjOUJP-mEckW0HcE7zc7ks5sqS6ngaJpZM4PxzYr .

[thenickdude]

I've now pushed a new version to NPM, 0.3.1, which I think should solve this issue by explicitly using HTTPS URLs for the git dependencies, please give it a go.

[sslupsky]

Thank you.

The update has cleared up the ssh public key issue. However, when I tried with the new dependency URL's I am seeing the same error at the command line:

[root@ip-10-0-0-111 ~]# npm install -g snap-to-s3 npm ERR! Error while executing: npm ERR! /usr/bin/git ls-remote -h -t https://github.com/thenickdude/node-progress.git npm ERR! npm ERR! fatal: Cannot come back to cwd: Permission denied npm ERR! npm ERR! exited with error code: 128

npm ERR! A complete log of this run can be found in: npm ERR! /root/.npm/_logs/2017-10-09T14_38_00_114Z-debug.log

The error's reported in the log are similar as reported earlier as well:

38 silly fetchPackageMetaData error for progress@git+https://github.com/thenickdude/node-progress.git#f901750478a76057b9271bda333dd1dcdd5406dd Error while executing: 38 silly fetchPackageMetaData /usr/bin/git ls-remote -h -t https://github.com/thenickdude/node-progress.git 38 silly fetchPackageMetaData 38 silly fetchPackageMetaData fatal: Cannot come back to cwd: Permission denied 38 silly fetchPackageMetaData 38 silly fetchPackageMetaData exited with error code: 128 39 silly fetchPackageMetaData error for tar-stream@git+https://github.com/thenickdude/tar-stream.git#3160e7d60fe142f04e126b8b261248d023200e1b Error while executing: 39 silly fetchPackageMetaData /usr/bin/git ls-remote -h -t https://github.com/thenickdude/tar-stream.git 39 silly fetchPackageMetaData 39 silly fetchPackageMetaData fatal: Cannot come back to cwd: Permission denied 39 silly fetchPackageMetaData 39 silly fetchPackageMetaData exited with error code: 128

When I run the git command from the command line I get the following output:

[root@ip-10-0-0-111 ~]# /usr/bin/git ls-remote -h -t https://github.com/thenickdude/node-progress.git f901750478a76057b9271bda333dd1dcdd5406dd refs/heads/master 2d87576b9dd831de221e2024e640ee727dff583f refs/heads/time-format 684f627ab1cee46ae0ae09095dd8550f3788e6b6 refs/tags/0.0.2 e6194546b6f48fd4ecd37a63238c04a3281a5e45 refs/tags/0.0.3 d83fa84f73b4dadd86d03dfb6c1bf832eb713dbd refs/tags/0.0.4 aa38082c4194bcadddf2df6b260fba2f5623d2f9 refs/tags/0.0.5 2b48b1d6cb42fe9352d2a1a87f8aad4a2e8fc726 refs/tags/0.0.5^{} 8628a2c919928181804b5ffd0557095df6103a54 refs/tags/0.1.0 5a12cafdee006ce4dd8624783d483b8371d32ed8 refs/tags/0.1.0^{} 783ca37c06f0f49a49b6efbbb0957806d849f5be refs/tags/1.0.0 0c0066de85818485257b1c20476f485928c97037 refs/tags/1.0.0^{} 1ac9fec5866152e4f51806a009d1761269326295 refs/tags/1.0.1 8cf6883d06e419caec653e10f0e5ef253c5033f0 refs/tags/1.0.1^{} 5641744a66ce4e024d495ae7fbfc8dda56eca9ab refs/tags/1.1.0 66b7f73c93d5bb89ed96709937656a0b05e4c2a8 refs/tags/1.1.0^{} aeac72488eac33b0572928a65d248ef6e764e78a refs/tags/1.1.1 26510351b88188c6e5273fb86cd558ac1cdaafb8 refs/tags/1.1.2 496e7cf984dc039df74bf72d8e7aa8fad3d94cce refs/tags/1.1.4 44e98342224e0f77bfa8c8105f428df36bdda3e1 refs/tags/1.1.4^{} 66495c58fa48cf0f35a894a791c9baad3ad7d1b2 refs/tags/1.1.5 fe178d7f9cff9903b93eae660eefa7e3ed2641ff refs/tags/1.1.8 6b9524c0d07df9555d20ae95c65918020c50e3e2 refs/tags/1.1.8^{} d84326ed9ab7720592b6bbc9c108849cd2a79908 refs/tags/2.0.0

All the other dependencies appear to be fine. The problem appears to be associated with only the two dependencies that are specified to your GitHub repos for some reason.

I think the issue has something to do with the user permissions. If I install nvm and node from my user account (instead of using the interactive sudo mode), the installation of snap-to-s3 goes smoothly and I can run snap-to-s3 from the command line. However, I cannot execute snap-to-s3 as sudo as you discussed during the installation steps.

So, what I did was edited the .bashrc for the root user and changed the NVM_DIR to point to the user account:

export NVM_DIR="/home/ec2-user/.nvm" [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" # This loads nvm [ -s "$NVM_DIR/bash_completion" ] && . "$NVM_DIR/bash_completion" # This loads nvm bash_completion

To get sudo to execute snap-to-s3, you need to also run it with the "simulate as initial login" (-i) option"

sudo -i snap-to-s3 --help

This gets me to the point of being able to run snap-to-s3 as sudo. I have not actually tested to see if it works yet ... I will do that now and report back.

[sslupsky]

snap-to-s3 is now executing the migrations. The output has confirmed that two migrations have been successful so far and I confirmed that the tar files are in S3. When it is complete I will do the validation and confirm the migration.

Thank you for your patience. This is an amazing tool. Thank you for making it public.