search

kfrajer / kfrajer.github.src

Hugo src code for kfrajer.github.io site
0 stars 0 forks source link

Running GCSFuse in gcloud DOCKER container #27

Open kfrajer opened 4 years ago

kfrajer commented 4 years ago

REFERENCES: [docker gcloud] https://hub.docker.com/r/google/cloud-sdk/ REFERENCES: [gcloud versions] https://groups.google.com/forum/#!forum/google-cloud-sdk-announce REFERENCES: [GCSFuse: umount + credentials] https://github.com/GoogleCloudPlatform/gcsfuse/blob/a8d9f02/docs/mounting.md#unmounting

export GCP_INIT_SCRIPT=gcp-init-cmd.sh
export MYGCPPROJECT=data4good
export HOST_VOL_PATH=/home/xorr/tmp-gcsfuse-docker/gcsfuse-folder
export CONTAINER_VOL_PATH=/usr/gcp/gcsfuse
export BUCKET_NAME=bucket-xima-dev

mkdir -p /home/xorr/tmp-gcsfuse-docker/
cd /home/xorr/tmp-gcsfuse-docker/
echo '#!/bin/bash' > $GCP_INIT_SCRIPT
echo 'echo "Project provided $1"' >> $GCP_INIT_SCRIPT
echo 'gcloud config set project "$1"' >> $GCP_INIT_SCRIPT
echo gcloud auth application-default login --no-launch-browser  >> $GCP_INIT_SCRIPT
echo "## HANDY command to explore files sorted by creation time - newest first" >> $GCP_INIT_SCRIPT
echo  'alias left="ls -lt"' >> $GCP_INIT_SCRIPT
chmod +x $GCP_INIT_SCRIPT
mkdir -p $HOST_VOL_PATH

================= DAC flow

docker run -ti -v /home/xorr/gcsfuse-docker/gcp-init-cmd.sh:/usr/bin/gcp/gcp-init-cmd.sh --name gcloud-config google/cloud-sdk  /bin/bash
chmod +x /usr/bin/gcp/gcp-init-cmd.sh && /usr/bin/gcp/gcp-init-cmd.sh
## Now open browser and do OAuth2.0 flow. Next confirm you have the credential json file:
more /root/.config/gcloud/application_default_credentials.json
exit
docker stop gcloud-config

================= GCLOUD CLI TEST

docker run -ti --rm google/cloud-sdk:304.0.0 gcloud version 
docker run -ti --rm google/cloud-sdk:304.0.0 gcloud-config gcloud config list
docker run -ti --rm  --volumes-from gcloud-config google/cloud-sdk gcloud compute instances list --project data4good
## gcloud config project set $MYGCPPROJECT
## gcloud compute instances list --project $MYGCPPROJECT

================= MOUNT bucket via gcsfuse 

## --privileged : Temporal flag to access /dev/fuse
## Issues: Current mounting overrides second mounted point in docker run aka files are only available in container and not in host FS
## Issue: privilege flag probably too high. How to access /dev/fuse in a more restrictive way?  
## - https://docs.docker.com/engine/reference/commandline/run/#full-container-capabilities---privileged
## - https://docs.docker.com/engine/reference/commandline/run/#add-host-device-to-container---device
## - POSSIBLY addresses SO: https://serverfault.com/questions/968611/gcsfuse-on-alpine-docker
docker run -ti --privileged -e MYGCPPROJECT-e HOST_VOL_PATH -e CONTAINER_VOL_PATH -e BUCKET_NAME --volumes-from gcloud-config --name gcli -v "$HOST_VOL_PATH":"$CONTAINER_VOL_PATH" google/cloud-sdk /bin/bash
echo "---------------"
mkdir -p $CONTAINER_VOL_PATH 
echo REPORT: $MYGCPPROJECT$HOST_VOL_PATH $CONTAINER_VOL_PATH $BUCKET_NAME
export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
echo "deb http://packages.cloud.google.com/apt $GCSFUSE_REPO main" | tee /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
apt-get update && apt-get install -y gcsfuse
gcsfuse -v
echo "---------------"
echo "Mounting $BUCKET_NAME $CONTAINER_VOL_PATH"
gcsfuse $BUCKET_NAME $CONTAINER_VOL_PATH 
cd $CONTAINER_VOL_PATH && left

================= Unmount

fusermount -u /usr/gcp/gcsfuse

================= TODO