help needed to set up S3

[cccs-ip]

I just set up an S3 account and uploaded data to it. The UI via AWS is very limited, and doesn't allow me to do basic things like sorting files by name and counting the number of objects in a bucket. I had created user credentials and configured my AWS CLI, but apparently I need to write permissions per user to access the bucket? (I thought my user had permissions by default?)

I would appreciate an interactive tutorial session this week to familiarize me with S3 and to help me gain better control of file system management using S3.

[pwhipp]

Ping me whenever you are ready. However, the interfacing is pretty basic generally - s3 is very fast and very simple (apart from setting up permissions).

For command line stuff: sudo apt-get install s3cmd

Documentation - http://s3tools.org/usage

You can string the commands with grep and sort to achieve any result you need (hardcore though).

There are Python GUIs too (s3cmd is written in Python but does not have a good Python API as such - tinyS3 is the easiest I've used).

[cccs-ip]

I'll be ready in another half hour or so. My initial issue appears to have to do with permissions. The limits of the UI are exactly why I am looking for CLI.

Aaron Dennis Director, Business Development and Research Cross-Cultural Consulting Services www.crossculturalconsult.com

On Sun, Oct 5, 2014 at 8:03 PM, Paul Whipp notifications@github.com wrote:

Assigned #169 https://github.com/cccs-web/core/issues/169 to @cccs-ip https://github.com/cccs-ip.

— Reply to this email directly or view it on GitHub https://github.com/cccs-web/core/issues/169#event-174318974.

[cccs-ip]

Hi Paul!

I continue to use the AWS credentials I shared with you. Have we yet set up AWS in such a way to allow you to use and manage permissions on S3?

Also, could you please write up a small segment on the website somewhere for how you set the user credentials for our different applications on S3?

Finally, it is my understanding that we could link our QGIS server and map application to the S3 backend. Not understanding how this all works, it's hard for me to see how the deploy script that Admire wrote could access S3 to load in data. For the moment, we can do his initial upload and database configuration referencing theshapefiles folder (under source_materials) in our map-app repository. I would, however, like to eventually remove this repository form GitLab and archive that project (since we never got very far on it anyway). It remains unclear to me whether S3 would be viable in this regard.

I look forward to your advice.

[pwhipp]

My own access

For my use, I've added the user pwhipp to your AWS account. I have full admin privileges and log in permission.

Documenting the S3 access method

The setting of credentials and specification of policies is documented on AWS but it is complex. I've included further notes here.

Linking QGIS server and map application

S3 is just a file store so it should be viable. I don't know enough about QGIS to say how complex it will be to do this.

There are two main routes to managing the shape data:

1. Shapefiles are a mature system for small collections presented onto maps using applications such as QGIS.
2. PostGIS is used to supply shape data for larger collections where querying allows access to meaningful subsets of the data (for example displaying a small region when there is data available for the entire world).

If the amount of shapefile data is such that we're thinking of moving it to S3 then it is probably more appropriate to look at moving it into PostGIS for efficient retrieval. Retrieving large files from S3 will always be time consuming.

[cccs-ip]

We'll need to clarify why we are using different forms of document management. With regard to the use of S3 for shapefile data, please see this discussion:

https://github.com/cccs-web/soc-maps/issues/11