Google Cloud Platform - Google Storage Bucket - image storage (same as AWS)

[gaitskell]

Would it be possible to create an option to automatically move files to Google Storage Buckets as an alternative to AWS Buckets (in a way identical to that in which indi-allsky currently moves them to AWS Buckets)?

We are doing serial image analyses by students in Python Notebooks on Google Colab. Colab access is well integrated for the Google Storage Buckets. (We can also make subsequent examples of the Python-based image/observation analyses in Colab available publicly to help others.)

My thanks, Rick

[aaronwmorris]

The latest versions of indi-allsky supports Google Cloud Platform [GCP], Oracle OCI, and AWS buckets with the same level of functionality.

If you have existing files that you want uploaded to GCP, I can provide the SQL to reset the file bucket keys so they can be re-uploaded to GCP.

[gaitskell]

@aaronwmorris For the Config web interface. https://piallsky.local/indi-allsky/config

Forgive me but I have been unable to get the GCP upload to work. Do you have a help page that explains how to work in GCP with the "Credentials File" field?

• Which type of credential file is expected? (There are quite a number of different options in GCP.)
• Should I upload the file to the unix-based file system and then put its path into this field.

Thanks, Rick

[aaronwmorris]

I do not have a page for it, but there are a couple of ways to access the Service Accounts.

• Navigate to IAM & Admin -> Service accounts
• Click CREATE SERVICE ACCOUNT at the top
  • Set the service account name
  • Click CREATE AND CONTINUE
  • Grant Storage Object Admin role
  • Click Done at bottom
• On the new service account, click the vertical ellipsis at the right, and Manage keys
  • Click ADD KEY -> Create new key
  • JSON Key type
  • Click CREATE

The JSON file should automatically download. This is the file you place on the filesystem and then in the path in the config field.

[gaitskell]

@aaronwmorris Thank you so much for the additional instructions - your suggestions above worked. indi-allsky is now uploading to a GCP (Google Cloud Platform) bucket.

I should add that it was necessary in GCP to change Bucket Access Control from IAM to Fine-grained so that the ACL settings from indi-allsky specified in Config "publicRead" can be specified during upload.

Initially, I wanted to manage the permissions working just within GCP but under IAM I couldn't get the public read settings for the entire bucket to be allowed. "IAM policy update failed. A domain restriction organization policy is in place. Only principals in allowed domains can be added as principals in the policy. Correct the principal emails and try again."

[aaronwmorris]

I had to go back into the GCP console to figure this out since I do this infrequently...

If you go to Permissions for the bucket and set allUsers principal to Storage Object Viewer then all objects in the bucket are public and you do not need to specify the ACL.