search

sillsdev / silnlp

A set of pipelines for performing experiments on various NLP tasks with a focus on resource-poor/minority languages.
Other
30 stars 3 forks source link

RClone into S3 bucket #67

Closed johnml1135 closed 3 years ago

johnml1135 commented 3 years ago

RClone can work - specifically, it can mount an S3 bucket as a network drive for free.

  1. Download and unzip rclone: https://rclone.org/downloads/
  2. Put this folder onto your desktop and name the folder "rclone"
  3. Open cmd and run rclone config - fill it out. Instructions: https://rclone.org/s3/ 
    [s3aqua]
type = s3
provider = AWS
env_auth = false
region = us-east-1
acl = bucket-owner-full-control
access_key_id = XXXXXXXXXXXXXXX
secret_access_key = YYYYYYYYYYYYYYYY

    The output should look like this (with proper keys) and will be in C:/Users/myname/.config/rclone/rclone.conf

  4. Install WINFSP (Windows File System Proxy) http://www.secfs.net/winfsp/rel/
  5. Create a bat file to mount the drive
    • Create a file in the "rclone" folder that has the rclone.exe executable and name it "mount_s3.bat"
    • Edit mount_s3.bat an add: call rclone mount s3aqua:aqua-ml-data S:
    • Double-click the mount_s3bat file to mount the drive. This window will need to stay open to keep the drive mounted. This could possibly be put into a Windows service and made automatic.

Now you can access the data on the S network drive by just clicking one file! (edited

johnml1135 commented 3 years ago

Note - this will only work to look at files in the S3 bucket on a windows machine - not for mounting on docker images, etc.