5 - Upload directly to Google Cloud Storage

[engagingcomms]

I had a quick search to see if there was a tutorial for this but no luck. Is there a way to modify the server side script so that Fineuploader saves to Google Storage?

https://developers.google.com/appengine/docs/php/googlestorage/

This would be very handy.

[uriahcarpenter]

Link to native JSON docs

[rnicholus]

@engagingcomms You can send the files to whatever location you choose, as the server is entirely under your control. Fine Uploader is exclusively a client-side library. You can use Fine Uploader's traditional endpoint handler to send files to your specific server, where you can then relay the files to Google Storage using the GS API.

[engagingcomms]

Has anyone done this before / have some sample code? As I have no idea :)

[engagingcomms]

In particular with Google App engine you can't save locally at all, has to be direct to storage.

[rnicholus]

It looks like GCS supports CORS, so it seems like we could upload directly to a bucket from the browser, à la Fine Uploader S3/Azure. Is this what you are wondering about?

[engagingcomms]

Yes that would be ideal. Upload to a Google Bucket with a unique file name, then return that filename to the Google APP. Not to sure where to start though.

[rnicholus]

It's something we can look into directly supporting in a future release. Theoretically, it's possible without our intervention, but a specific Google Cloud Storage module for Fine Uploader would make it easier, so you don't have to worry about the details of the GCS API. We'll discuss this internally.

[engagingcomms]

Nice one cheers.

[rnicholus]

We might be able to easily support this via the s3 compatibility layer provided by cloud storage.

[engagingcomms]

That would be awesome. The Google APP engine doesn't allow you to store directly to server so this would be a very valuable script.

[rnicholus]

The work completed in #1332 may make this possible, assuming the Google Cloud Storage S3 integration layer is enabled. Any Google Cloud Storage users want to confirm with the pre-release of 5.1.0? If you find issues, we'll fix them as part of 5.1.0. Otherwise, we'll look into this more in a future release.

[kngtr]

Hi - How can I get a copy of the pre-release of 5.1.0? Thanks.

[rnicholus]

We plan to release 5.1.0 this week, so you will be able to give this a spin in the released version.

On Mon, Dec 29, 2014 at 5:57 PM, kngtr notifications@github.com wrote:

Hi - How can I get a copy of the pre-release of 5.1.0? Thanks.

Reply to this email directly or view it on GitHub: https://github.com/FineUploader/fine-uploader/issues/1158#issuecomment-68316969

[rnicholus]

@kngtr Have you downloaded 5.1? If so, are you interested in using it to upload directly to Google Cloud Storage?

[engagingcomms]

I would be, do you have a working php sample I could try?

R

On 7 Jan 2015, at 8:09 am, Ray Nicholus notifications@github.com wrote:

@kngtr Have you downloaded 5.1? If so, are you interested in using it to upload directly to Google Cloud Storage?

— Reply to this email directly or view it on GitHub.

[rnicholus]

We have a number of Fine Uploader S3 PHP samples in the server-examples repo. You'd use Fine Uploader S3 and setup Google Cloud Storage to use S3 compatibility mode. The S3 feature page will provide some additional details, if you need them. Uploading to Cloud Storage with S3 compatibility mode enabled would qualify as an "S3-like" endpoint, which, in theory, should be supported now after changes made in 5.1.

When constructing Fine Uploader S3 client-side, you'd need to specify the Cloud Storage endpoint as request.endpoint and the bucket in objectProperties.bucket.

[kngtr]

Thanks rnicholus. I will download and give it a try.

[ghost]

I'm a google cloud user and I'd love to test this out. I'm a little confused on how to integrate it though using the s3 compatibility mode. Has anyone tried this or explain it a little more in depth?

[rnicholus]

@bryangoldberg What, specifically, are you stuck on? The Google Cloud Storage docs cover a "simple migration" from S3 to Cloud Storage, which allows a CS endpoint to be treated as if it were an S3 bucket.

[rnicholus]

Has anyone had a chance to test this out with Google Cloud Storage's S3 compatibility mode yet?

[kngtr]

Not yet. Hopefully I will let you know next week.

[kngtr]

Ray - I gave a quick try this morning and ran into 'Invalid policy document or request headers!' error. I already replaced AWS-ACCESS-KEY with GOOG-ACCESS-KEY so I will debug more later.

[rnicholus]

You'll need to ensure all access keys server side are your cloud storage keys. Also, the credentials.accessKey client-side Fine Uploader S3 option must be set to your GCS access key.

[ghost]

It's far too confusing to set up a google cloud storage upload through the s3. If anyone is trying to, an easy alternative I found is the following (if you are using app engine):

require_once 'google/appengine/api/cloud_storage/CloudStorageTools.php';
use google\appengine\api\cloud_storage\CloudStorageTools;

$options = [ 'gs_bucket_name' => 'bucket' ];
$upload_url = CloudStorageTools::createUploadUrl('/upload-url', $options);

Then you'll want to store $upload_url into a variable like so:

<script>var upload_link = "<?php echo $upload_url; ?>"</script>

And then continue your fineuploader like normal. Notice the endpoint is the new upload_link variable.

var upload_template = $("#upload_template").fineUploader({
    request: { endpoint: upload_link },
    autoUpload: false, validation: { sizeLimit: 5000000, itemLimit: 1,  allowedExtensions: ['pdf'] },
}).on('complete', function (event, id, name, responseJSON) {

});

Then in your /upload-url have the following to move your new temporary uploaded file:

move_uploaded_file($_FILES['qqfile']['tmp_name'], 'gs://bucket/new_file.pdf');

And boom, you've uploaded a file to your cloud storage to do whatever you want using fine uploader.

[rnicholus]

It looks like you are essentially uploading the file twice (once to your server, then again from your server to the final storage destination), something that Fine Uploader S3 aims to prevent. So, you're not using Fine Uploader S3 at all. Is this correct?

[ghost]

App Engine doesn't allow for writes to the server, only the Cloud Storage, so I "think" it's only uploading it once and then basically renaming the file and making it available in google cloud storage. It's in a hidden temp storage at the time of upload and needs to be moved with move_uploaded_file().

But that is correct, I am currently just using the default Fine Uploader script. I wish I had the time to wrap my head around the Fine Uploader S3, but since I've never used anything CORS or anything in S3 before it's a tad confusing for me.

[rnicholus]

Hmm, ok. I'll admit I'm not very familiar with app engine. Not very = not at all. Thanks for posting this, perhaps it will be useful for others using AE.

I hope to spend some time playing around with GCS after the current in-progress release of Fine Uploader is out the door (5.2).

[ghost]

No problem, that was the goal. Thanks for such a simple and easy uploader tool. It REALLY helped me out. Looking forward to the next release. If you need any help messing with GCS in the future, I'd be happy to try some things out.

[rnicholus]

Shelving this until sometime after 5.3, unless someone reports success using Fine Uploader S3 w/ CDN support in the meantime.

[LusciousPear]

Has anyone tried this lately?

[nateweiss]

I was able to make this work (uploading to Google Cloud Storage using their interoperability feature), after making a few minor changes in s3.fine-uploader.js. Let me know if you would like details or a PR, etc.

[rnicholus]

Can you describe the changes, or simply open up a pull request so we can discuss further?

[nateweiss]

Hi, submitted PR #1674. Feel free to adapt or ignore as needed.

If you like I could write up a short list of the steps one has to take on the Google Cloud Storage side to enable the bucket to actually accept the uploads. Basically you have to (1) enable the S3 interoperability feature (easy, it's in the admin console), and also enable CORS (not as easy to find, you have to use their "gsutil" command-line tools or enable it via their API).

Once those steps are done, the fine-uploader support seems to work fine with the google service. Perhaps there would be some advantage to having a separate GCS-specific module at the same level as the S3 one, but honestly I'm not sure that it would matter for most folks.

[rnicholus]

If you could include that list in the PR or at least reference this issue, that would be helpful. I'm currently traveling, but will look closer once I get past a number of other items in my fine uploader queue first.

[aaronjolson]

Has any progress been made on this issue? I would like to have some documentation made available that instructed how to get this set up. I'm very interested in being able to upload files client-side directly to a GCS bucket via fineuploadler. @nateweiss You don't happen to have a blog describing how you did this do you?

[nateweiss]

_Hi @aaronjolson, I've typed up some instructions (as markdown), which I am pasting below. Hopefully this helps you get GCS uploads working. Because my PR isn't merged in at this point, you'll need to use my branch to get the couple of very minor changes to the fine-uploader code that are needed . To @rnicholus, I wasn't sure where to add these instructions in the docs, maybe the text below should be added to http://docs.fineuploader.com/branch/master/features/s3.html with an "alert" callout in http://docs.fineuploader.com/branch/master/quickstart/02-setting_options-s3.html that points to the former? Also I wasn't sure if there was a build step needed for doc changes, the Makefile doesn't seem to include that unless I missed it._

Using Fine Uploader with Google Cloud Storage

You can use the S3 support in Fine Uploader to upload documents to Google Cloud Storage (GCS) buckets.

There are three basic steps:

1. Enable the S3 interoperability feature.
2. Enable CORS for your bucket.
3. Use the normal S3 support in Fine Uploader to upload files to your GCS bucket.

1. Enable the S3 interoperability feature.

This will allow Fine Uploader to communicate with GCS in the same way it normally talks to S3.

In the Google Cloud admin, navigate to Cloud Storage > Settings > Interoperability. There you can generate the S3-style developer access key (and corresponding secret key) to provide to Fine Uploader. Note that your access key will start with GOOG. For more details about this step, see simple migration in the GCS docs.

2. Enable CORS for your bucket.

Without this step, your user's browser will refuse to post information to GCS due to the same-origin security policy.

At the time of this writing, there appeared to be no way to do this step in the Google Cloud admin, so you need to enable it using either the gsutil command-line tool or with their XML/JSON API. The steps to take are explained pretty clearly under Configuring CORS on a Bucket in the GCS docs. Note that the examples in the GCS docs show granting GET, HEAD, and DELETE, while you want to grant HEAD and POST to enable uploading via Fine Uploader (GET and DELETE aren't needed).

3. Use the normal S3 support in Fine Uploader to upload files to your GCS bucket.

Now you should be able to take advantage of Fine Uploader's S3 support to upload files to GCS. Follow the instructions in the Fine Uploader documentation as if you were using an S3 bucket, except:

• For endpoint in the request option, use storage.cloud.google.com where you see s3.amazonaws.com.

• For accessKey, use your GCS-provided access key (starting with GOOG) that you created above.

• You still need to implement a signature endpoint on your side to sign upload requests, as discussed here. As you get that set up, note that you'll use the GCS-provided access/public key (starting with GOOG) and the corresponding private/secret key in the places where the docs show keys from AWS.

[rnicholus]

No build steps are needed for doc changes. I have a webhook setup that listens for pushes to each branch, and docs are updated for the associated branch at http://docs.fineuploader.com. Develop = "in progress", master = root, other branch docs can be seen at http://docs.fineuploader.com/branch.

Pull requests that only cover documentation can be made against master, otherwise all PRs should be against develop. What changes to Fine Uploader code, if any, are required?

[nateweiss]

Hi @rnicholus, there are a couple of minor changes to the Fine Uploader code, which can be found here https://github.com/FineUploader/fine-uploader/pull/1674/files. If you think you're likely to merge those changes into master, I can add the instructions I provided in the comments above to the docs somewhere as part of the PR.

[artfree]

Hi @rnicholus. What's the status of this issue? I'm totally new to Fine Uplaoder, but also need to be able to upload - directly - to Google Cloud Storage (GCS). The instructions given by nateweiss, above, are a bit too complex for me. Will you have GCS as a supported endpoint in Fine Uploader (just as you now have direct support for S3) ? I'm in need of this feature currently. How long before it's implemented? Thanks much.

[rnicholus]

Will you have GCS as a supported endpoint in Fine Uploader

I don't have any plans to implement this myself in the foreseeable future. Your best bet is to use GCS in S# compatibility mode and follow #1674. Once that is completed, it should be relatively easy to upload to GCS from Fine Uploader.

[pankitgami]

I have created the demo repository using Laravel and Fineuploader to upload to GCS.

https://github.com/pankitgami/fineuploader-gcs-example Heroku app: https://quiet-anchorage-26485.herokuapp.com/