Accessing bucket files from Compute Instance (VM)

[rulrok]

I have an application which is deployed to GAE and that is storing files to my bucket using this plugin. All of this works just fine.

The problem is: I have one virtual machine with the same laravel application on it, but just for processing beanstalkd queues. I have described it more on my other issue #30 . When trying to access the file using the filesystem abstraction, it is getting the wrong file path.

For example, on my GAE instance, the following code

$disk = Storage::disk('pano_images');
        Log::info("Disk path: ".$disk->getDriver()->getAdapter()->getPathPrefix());

returns

Disk path: gs://panopla-staging-4f8x/storage/app/pano_images/

which is correct.

When running the same command on my virtual machine, I get the local path

Disk path: /lamp/pano-cms/storage/app/pano_images/.

It is probably because of the following mthod on filesystem.php config file.

    'disks' => [

        'pano_images' => [
            'driver' => 'gae',
            'root' => storage_path() . '/app/pano_images'
        ],

    ],

I've tried to manually replace the call for storage_path() method with the direct bucket url.

    'disks' => [

        'pano_images' => [
            'driver' => 'gae',
           'root' => 'gs://<bucket_id>/storage' . '/app/pano_images'
        ],

    ],

But now I get the following error:

[ErrorException]                                                                                  
  is_dir(): Unable to find the wrapper "gs" - did you forget to enable it when you configured PHP?  

What can I do for accessing this file on my cloud bucket from the VM instance? I was expecting that by using the 'gae' driver, the plugin would take care of all the necessary abstraction.

Note: I'm using the VM to process the beanstalkd queue because I need to run a third-party linux script on PHP and that is just not allowed on GAE.

[shpasser]

Since your VM is running on Google Compute Engine it does not have the the gs:// filesystem wrapper(which is part of GAE PHP SDK). I suppose that you could try using GCS Fuse in order to mount your GCS bucket on your GCE VM and access it through Laravel local disk abstraction(of course you'll have to set the correct path).

https://cloud.google.com/storage/docs/gcs-fuse

[rulrok]

That's neat. I'm starting now with GAE and their documentation is so huge that I didn't find this info before. It worked for me and it was far too simpler compared if having to use google's php api client library. (More details here and here.

But it would be really awesome if it could happen. But in that case, for what I was researching, it would be necessary to issue an API token from GAE control panel and add it plus the bucket id and app id in the configs. Just an idea, but I think it would be more complicated.

I'll just leave the issue open for now if you want to discuss the idea.

Thank you again.

[shpasser]

The idea is doable. But when I think about what you are trying to achieve and the tools that you would like to use in order to achieve it, I can see a very significant gap. If you plan to access the GCS like a filesystem you'll need to build your own adapter, which will use Google's PHP API Client on one side and present its contents as a filesystem on the other side. When using GAE you have gs:// wrapper doing the job, on GCE it is GCS Fuse. If you are going to implement the adapter I would suggest to start by going through the code of gs:// wrapper:

https://chromium.googlesource.com/external/googleappengine/python/+/200fcb767bdc358a3acb5cf7cad1376fe69f12c5/php/sdk/google/appengine/ext/cloud_storage_streams

[shpasser]

I would like to close this one unless you would like to discuss the matter further.

[dakodner]

Hello, I Have the same situation. How I need to make with GCS Fuse? Thanks.

[shpasser]

In general you will need to mount your GCS using GCS Fuse on your GCE VM. For more info look here: https://cloud.google.com/storage/docs/gcs-fuse