search

linnarsson-lab / loom-viewer

Tool for sharing, browsing and visualizing single-cell data stored in the Loom file format
BSD 2-Clause "Simplified" License
35 stars 6 forks source link

Storing credentials #17

Closed slinnarsson closed 8 years ago

slinnarsson commented 8 years ago

Need a good way to store credentials so that they are easily accessible for developer, yet do not leak into GitHub. One day we will make this repo public, so there should be no credentials here.

One possibility is secrets.

slinnarsson commented 8 years ago

By avoiding the use of containers, we can get around this issue. When we use gcloud and the Cloud SDK (configured once and for all), the credentials are stored on the VM itself.

JobLeonard commented 8 years ago

I'm getting the following error when trying to run ./deploy post-merge:

scp: /srv/loom//python/uploader.py: Permission denied
scp: /srv/loom//python/requirements.txt: Permission denied
scp: /srv/loom//python/loom_server_LOCAL_15475.py: Permission denied
scp: /srv/loom//python/loom_pipeline.py: Permission denied
scp: /srv/loom//python/loom_server_BACKUP_15475.py: Permission denied
scp: /srv/loom//python/loom_server_REMOTE_15475.py: Permission denied
scp: /srv/loom//python/static/index.html: Permission denied
scp: /srv/loom//python/static/img/marker-shadow.png: Permission denied
scp: /srv/loom//python/static/img/marker-icon.png: Permission denied
scp: /srv/loom//python/static/img/marker-icon-2x.png: Permission denied
scp: /srv/loom//python/static/img/layers-2x.png: Permission denied
scp: /srv/loom//python/static/img/layers.png: Permission denied
scp: /srv/loom//python/static/js/bundle.7579c2e9cc052b16cc18.js: Permission denied
scp: /srv/loom//python/static/css/bundle.min.css: Permission denied
scp: /srv/loom//python/loom_cache.py: Permission denied
scp: /srv/loom//python/loom_server_BASE_15475.py: Permission denied
scp: /srv/loom//python/loom_server.py: Permission denied

This means I'm somewhat stuck for now, since I need a loom file to run the server off-line. Currently the cloud version still appears to have an older version of the server (because the /loom endpoint gives me the old-style dataset list), and without the /clone/ endpoint I can't dowload a loom file to test things out.

I'll try to get a loom file from someone next to me, see if I can test with that.

slinnarsson commented 8 years ago

This seems to happen because of file permissions on the server side. Since I was the one deploying first all the files, folders and subfolders are owned by me and not writeable by anyone else. Not sure how to fix it (because it'll happen at every deployment), but here's what it looks like:

Sten@loom:~$ ls -lht /srv
total 8.0K
drwxrwxrwx 2 root root 4.0K Aug  3 14:14 datasets
drwxrwxrwx 3 root root 4.0K Aug  3 13:17 loom
Sten@loom:~$ ls -lht /srv/loom/
total 4.0K
drwxr-xr-x 4 Sten Sten 4.0K Aug 12 13:31 python
Sten@loom:~$ ls -lht /srv/loom/python/
total 260K
-rw-r--r-- 1 root root 6.9K Aug 12 13:31 loom_server.pyc
-rw-r--r-- 1 root root  15K Aug 12 13:31 loom_pipeline.pyc
-rw-r--r-- 1 root root 3.8K Aug 12 13:31 loom_cache.pyc
-rw-r--r-- 1 Sten Sten   50 Aug 12 13:30 requirements.txt
-rwxr-xr-x 1 Sten Sten 4.5K Aug 12 13:30 loom_server.py
-rwxr-xr-x 1 Sten Sten  16K Aug 12 13:30 loom_pipeline.py
-rw-r--r-- 1 Sten Sten  11K Aug 12 13:30 loom_cmd.py
-rw-r--r-- 1 Sten Sten 4.2K Aug 12 13:30 loom_cache.py
-rw-r--r-- 1 root root 6.0K Aug  3 15:14 loom_cloud.pyc
-rw-r--r-- 1 root root 5.6K Aug  3 15:14 cmdstan.pyc
-rw-r--r-- 1 Sten Sten 6.4K Aug  3 15:14 ts_model.py
-rw-r--r-- 1 Sten Sten 6.3K Aug  3 15:13 loom_cloud.py
-rw-r--r-- 1 Sten Sten  21K Aug  3 15:13 loom_backspin.py
-rw-r--r-- 1 Sten Sten 2.9K Aug  3 15:13 download_all.py
-rw-r--r-- 1 Sten Sten 5.8K Aug  3 15:13 cmdstan.py
-rw-r--r-- 1 Sten Sten 6.0K Aug  3 15:03 ts_model.pyc
-rw-r--r-- 1 Sten Sten  26K Aug  3 15:02 ome.pyc
-rw-r--r-- 1 Sten Sten  17K Aug  3 15:02 loom_backspin.pyc
-rw-r--r-- 1 Sten Sten  37K Aug  3 15:02 loom.pyc
-rw-r--r-- 1 Sten Sten 2.4K Aug  3 15:02 cloud_stan.pyc
-rw-r--r-- 1 Sten Sten 4.0K Aug  3 15:02 bregression.pyc
-rw-r--r-- 1 Sten Sten 3.5K Aug  3 15:02 bh_tsne.pyc
drwxr-xr-x 5 Sten Sten 4.0K Aug  3 13:17 static
drwxr-xr-x 2 Sten Sten 4.0K Aug  3 13:17 cache

For now, I've built and deployed the latest master version. It's at http://104.197.181.233.

JobLeonard commented 8 years ago

Fixed it.

A quick google search pointed to this StackOverflow Q&A, which suggests inserting root@ before the instance name. It also mentions how to choose a default zone to deploy to, which I've set to the one we currently use (us-central1-a). Deploy script now reads:

echo "DEPLOY: Stopping the server and pipeline"
gcloud compute ssh --zone "us-central1-a" loom --command "sudo /home/Sten/anaconda2/bin/supervisorctl stop all"
echo "DEPLOY: Copying local build to the server"
rm ./python/*.pyc  # Needed to prevent access denied errors
gcloud compute copy-files --zone "us-central1-a" python/ root@loom:/srv/loom/
echo "DEPLOY: Starting the server and pipeline"
gcloud compute ssh --zone "us-central1-a" loom --command "sudo /home/Sten/anaconda2/bin/supervisorctl start all"
slinnarsson commented 8 years ago

Good. I removed the —zone argument, since it is probably redundant. The VM is already deployed there, and I guess it won’t move even if you designate a different zone.

Also cleaned up a little to make the script less noisy.

Progress: I can now see scatterplots on the remote server!

/Sten

Sten Linnarsson, PhD Professor of Molecular Systems Biology Karolinska Institutet Unit of Molecular Neurobiology Department of Medical Biochemistry and Biophysics Scheeles väg 1, 171 77 Stockholm, Sweden +46 8 52 48 75 77 (office) +46 70 399 32 06 (mobile)

On 12 Aug 2016, at 16:09, Job van der Zwan notifications@github.com<mailto:notifications@github.com> wrote:

Fixed it.

A quick google search pointed to this StackOverflow Q&Ahttp://stackoverflow.com/questions/27807018/gcloud-compute-copy-files-permission-denied-when-copying-files, which suggests inserting root@ before the instance name. It also mentions how to choose a default zone to deploy to, which I've set to the one we currently use (us-central1-a). Deploy script now reads:

echo "DEPLOY: Stopping the server and pipeline" gcloud compute ssh --zone "us-central1-a" loom --command "sudo /home/Sten/anaconda2/bin/supervisorctl stop all" echo "DEPLOY: Copying local build to the server" rm ./python/*.pyc # Needed to prevent access denied errors gcloud compute copy-files --zone "us-central1-a" python/ root@loom:/srv/loom/ echo "DEPLOY: Starting the server and pipeline" gcloud compute ssh --zone "us-central1-a" loom --command "sudo /home/Sten/anaconda2/bin/supervisorctl start all"

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHubhttps://github.com/linnarsson-lab/Loom/issues/17#issuecomment-239456098, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKKagzQ91zhBWDCvsGzjj-ISQdu1x4hHks5qfH6HgaJpZM4IVYkd.

JobLeonard commented 8 years ago

Yes, it's working for me too! (except heatmap and sparkline, but those are currently broken for other reasons).

Have you tried clicking the dropdowns yet? You can type-to-find selections :)

The zone argument isn't redundant, without it I get this prompt every time a gcloud command runs:

DEPLOY: Copying local build to the server
For the following instances:
 - [loom]
choose a zone:
 [1] asia-east1-a
 [2] asia-east1-b
 [3] asia-east1-c
 [4] europe-west1-c
 [5] europe-west1-b
 [6] europe-west1-d
 [7] us-central1-f
 [8] us-central1-a
 [9] us-central1-b
 [10] us-central1-c
 [11] us-east1-c
 [12] us-east1-b
 [13] us-east1-d
 [14] us-west1-b
 [15] us-west1-a
Please enter your numeric choice:

Call me lazy, but I don't feel like waiting to type an 8 three times in a row; I just want to ./deploy and be done with it!

slinnarsson commented 8 years ago

Ok weird, that doesn't happen to me. So put it back then.

Sten

Skickat från min iPhone

12 aug. 2016 kl. 17:02 skrev Job van der Zwan notifications@github.com<mailto:notifications@github.com>:

Yes, it's working for me too! (except heatmap and sparkline, but those are currently broken for other reasons).

Have you tried clicking the dropdowns yet? You can type-to-find selections :)

The zone argument isn't redundant, without it I get this prompt every time a gcloud command runs:

DEPLOY: Copying local build to the server For the following instances:

Call me lazy, but I don't feel like waiting to type an 8 three times in a row; I just want to ./deploy and be done with it!

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHubhttps://github.com/linnarsson-lab/Loom/issues/17#issuecomment-239470340, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKKagyq3EDJHPvqS6ZvbQSsT33cj21sfks5qfIrMgaJpZM4IVYkd.