search

fastlane / fastlane

🚀 The easiest way to automate building and releasing your iOS and Android apps
https://fastlane.tools
MIT License
38.88k stars 5.64k forks source link

fastlane does not like GCS buckets with manually created folders #21692

Open jamiestrachan-shopify opened 7 months ago

jamiestrachan-shopify commented 7 months ago

New Issue Checklist

Issue Description

I ran into an issue where fastlane was failing to download an image of the specified google bucket, and it turned out to be because the bucket API behaves differently when you manually add a folder to it.

In particular, fastlane sets up the bucket without folders - it just writes files into the destination path. With this setup, the call to bucket.files returns only files. EG., 

"N8X438SEU2/certs/distribution/XD9G7QCACF.cer",
"N8X438SEU2/certs/distribution/XD9G7QCACF.p12",
etc...

However, if you go and manually add folders to the bucket, (ie., you manually configure signing), the google API will include the folders in the call to bucket.list:

"N8X438SEU2/",
"N8X438SEU2/certs/",
"N8X438SEU2/certs/distribution/",
"N8X438SEU2/certs/distribution/XD9G7QCACF.cer"
etc...

The interesting thing is that the bucket web admin looks identical - both setups show a folder hierarchy.

When bucket.list includes a folder, you will see an error such as:

[...]/.gem/ruby/3.2.2/gems/google-apis-core-0.11.2/lib/google/apis/core/download.rb:42:in `initialize': No such file or directory @ rb_sysopen - /var/folders/79/k0q0rm155b96nd6w3vg6cm1h0000gn/T/d20231201-3793-7uw1y6/N8X438SEU2/ (Errno::ENOENT)

This is because of this line in download.rb:

FileUtils.mkdir_p(File.expand_path("..", download_path))

The first file in the bucket is (eg) "N8X438SEU2/", and so call to expand_path is effectively ..

My proposed fix is to ignore the current_file if it ends with a /:

--- a/match/lib/match/storage/google_cloud_storage.rb
+++ b/match/lib/match/storage/google_cloud_storage.rb
@@ -152,6 +152,8 @@ module Match
         self.working_directory = Dir.mktmpdir

         bucket.files.each do |current_file|
+          continue if current_file.end_with?("/")
+
           file_path = current_file.name # e.g. "N8X438SEU2/certs/distribution/XD9G7QCACF.cer"
           download_path = File.join(self.working_directory, file_path)```
Command executed

A match command

Complete output when running fastlane, including the stack trace and command used
gem/ruby/3.2.2/gems/google-apis-core-0.11.2/lib/google/apis/core/download.rb:42:in `initialize': No such file or directory @ rb_sysopen - /var/folders/g2/lpkqjcq9397f416kq3jbgtwh0000gn/T/d20231204-3788-841t0n/N8X438SEU2/ (Errno::ENOENT)
    from /Users/user/.gem/ruby/3.2.2/gems/google-apis-core-0.11.2/lib/google/apis/core/download.rb:42:in `open'
    from /Users/user/.gem/ruby/3.2.2/gems/google-apis-core-0.11.2/lib/google/apis/core/download.rb:42:in `prepare!'
    from /Users/user/.gem/ruby/3.2.2/gems/google-cloud-storage-1.37.0/lib/google/cloud/storage/service.rb:848:in `execute_with_response'
    from /Users/user/.gem/ruby/3.2.2/gems/google-cloud-storage-1.37.0/lib/google/cloud/storage/service.rb:834:in `execute_or_queue_command_with_response'
    from /Users/user/.gem/ruby/3.2.2/gems/google-cloud-storage-1.37.0/lib/google/cloud/storage/service.rb:821:in `get_object_with_response'
    from /Users/user/.gem/ruby/3.2.2/gems/google-cloud-storage-1.37.0/lib/google/cloud/storage/service.rb:491:in `block in download_file'
    from /Users/user/.gem/ruby/3.2.2/gems/google-cloud-storage-1.37.0/lib/google/cloud/storage/service.rb:770:in `execute'
    from /Users/user/.gem/ruby/3.2.2/gems/google-cloud-storage-1.37.0/lib/google/cloud/storage/service.rb:490:in `download_file'
    from /Users/user/.gem/ruby/3.2.2/gems/google-cloud-storage-1.37.0/lib/google/cloud/storage/file.rb:1041:in `download'
    from /Users/user/.gem/ruby/3.2.2/gems/fastlane-2.217.0/match/lib/match/storage/google_cloud_storage.rb:160:in `block in download'
    from /opt/rubies/3.2.2/lib/ruby/3.2.0/delegate.rb:349:in `each'
    from /opt/rubies/3.2.2/lib/ruby/3.2.0/delegate.rb:349:in `block in delegating_block'
    from /Users/user/.gem/ruby/3.2.2/gems/fastlane-2.217.0/match/lib/match/storage/google_cloud_storage.rb:154:in `download'
    from /Users/user/.gem/ruby/3.2.2/gems/fastlane-2.217.0/match/lib/match/runner.rb:37:in `run'
    from /Users/user/.gem/ruby/3.2.2/bundler/gems/redacted///certs_and_profiles.rb:74:in `fastlane_match'

Environment

Google Cloud Storage bucket.

✅ fastlane environment ✅ ### Stack | Key | Value | | --------------------------- | ------------------------------------------- | | OS | 14.1 | | Ruby | 3.2.2 | | Bundler? | false | | Git | git version 2.42.0 | | Installation Source | ~/.gem/ruby/3.2.2/bin/fastlane | | Host | macOS 14.1 (23B74) | | Ruby Lib Dir | /opt/rubies/3.2.2/lib | | OpenSSL Version | OpenSSL 1.1.1t 7 Feb 2023 | | Is contained | false | | Is homebrew | false | | Is installed via Fabric.app | false | | Xcode Path | /Applications/Xcode.app/Contents/Developer/ | | Xcode Version | 15.0.1 | | Swift Version | 5.9 | ### System Locale | Variable | Value | | | -------- | ----------- | - | | LANG | en_CA.UTF-8 | ✅ | | LC_ALL | en_US.UTF-8 | ✅ | | LANGUAGE | en_US.UTF-8 | ✅ | ### fastlane files: **No Fastfile found** **No Appfile found** ### fastlane gems | Gem | Version | Update-Status | | -------- | ------- | ------------- | | fastlane | 2.217.0 | ✅ Up-To-Date | ### Loaded fastlane plugins: **No plugins Loaded**
Loaded gems | Gem | Version | | ------------------------------- | ------------ | | error_highlight | 0.5.1 | | did_you_mean | 1.6.3 | | syntax_suggest | 1.0.2 | | artifactory | 3.0.15 | | babosa | 1.0.4 | | CFPropertyList | 3.0.6 | | colored | 1.2 | | highline | 2.0.3 | | commander | 4.6.0 | | dotenv | 2.8.1 | | emoji_regex | 3.2.3 | | faraday-em_http | 1.0.0 | | faraday-em_synchrony | 1.0.0 | | faraday-excon | 1.1.0 | | faraday-httpclient | 1.0.1 | | faraday-multipart | 1.0.4 | | faraday-net_http | 1.0.1 | | faraday-net_http_persistent | 1.2.0 | | faraday-patron | 1.0.0 | | faraday-rack | 1.0.0 | | faraday-retry | 1.0.3 | | ruby2_keywords | 0.0.5 | | faraday | 1.10.3 | | http-cookie | 1.0.5 | | faraday-cookie_jar | 0.0.7 | | gh_inspector | 1.1.3 | | mini_magick | 4.12.0 | | naturally | 2.2.1 | | optparse | 0.1.1 | | rubyzip | 2.3.2 | | security | 0.1.3 | | simctl | 1.6.10 | | terminal-notifier | 2.0.0 | | terminal-table | 3.0.2 | | tty-screen | 0.8.1 | | tty-cursor | 0.7.1 | | tty-spinner | 0.9.3 | | word_wrap | 1.0.0 | | rouge | 2.0.7 | | xcpretty | 0.3.0 | | xcpretty-travis-formatter | 1.0.1 | | set | 1.0.3 | | public_suffix | 5.0.4 | | addressable | 2.8.5 | | aws-eventstream | 1.3.0 | | aws-sigv4 | 1.8.0 | | aws-partitions | 1.862.0 | | jmespath | 1.6.2 | | aws-sdk-core | 3.190.0 | | aws-sdk-kms | 1.74.0 | | aws-sdk-s3 | 1.141.0 | | bundler | 2.4.22 | | rexml | 3.2.6 | | excon | 0.105.0 | | multipart-post | 2.3.0 | | faraday_middleware | 1.2.0 | | domain_name | 0.6.20231109 | | fastimage | 2.2.7 | | webrick | 1.8.1 | | httpclient | 2.8.3 | | multi_json | 1.15.0 | | jwt | 2.7.1 | | signet | 0.18.0 | | os | 1.1.4 | | googleauth | 1.8.1 | | mini_mime | 1.1.5 | | retriable | 3.1.2 | | trailblazer-option | 0.1.2 | | declarative | 0.0.20 | | uber | 0.1.0 | | representable | 3.2.0 | | google-apis-core | 0.11.2 | | google-apis-androidpublisher_v3 | 0.53.0 | | google-apis-playcustomapp_v1 | 0.13.0 | | rake | 13.1.0 | | digest-crc | 0.6.5 | | google-apis-storage_v1 | 0.29.0 | | google-apis-iamcredentials_v1 | 0.17.0 | | google-cloud-errors | 1.3.1 | | google-cloud-env | 1.6.0 | | google-cloud-core | 1.6.0 | | google-cloud-storage | 1.45.0 | | json | 2.6.3 | | plist | 3.7.0 | | unicode-display_width | 2.5.0 | | nanaimo | 0.3.0 | | colored2 | 3.1.2 | | claide | 1.1.0 | | atomos | 0.1.3 | | xcodeproj | 1.23.0 | | forwardable | 1.3.3 | | logger | 1.5.3 | | pathname | 0.2.1 | | shellwords | 0.1.0 | | cgi | 0.3.6 | | date | 3.3.3 | | timeout | 0.4.0 | | securerandom | 0.2.2 | | uri | 0.12.1 | | openssl | 3.1.0 | | digest | 3.1.1 | | ipaddr | 1.2.5 | | resolv | 0.2.2 | | time | 0.2.2 | | open-uri | 0.3.0 | | stringio | 3.1.0 | | mutex_m | 0.2.0 | | net-http | 0.3.2 | | net-protocol | 0.2.1 | | rack | 2.2.8 | | rack-test | 2.1.0 | | fileutils | 1.7.0 | | tempfile | 0.1.3 | | delegate | 0.3.0 | | tmpdir | 0.1.3 | | ostruct | 0.5.5 | | english | 0.7.2 | | erb | 4.0.2 | | abbrev | 0.1.1 | | base64 | 0.2.0 | | singleton | 0.1.1 | | open3 | 0.1.2 | | nkf | 0.1.2 | | nokogiri | 1.15.4 | | racc | 1.7.3 | | prettyprint | 0.1.1 | | pp | 0.4.0 | | find | 0.1.1 | | yaml | 0.2.1 | | psych | 5.1.1.1 | | ox | 2.14.17 |
*generated on:* **2023-12-04**
khalidrinaldy commented 3 months ago

have you solve this issue?

jamiestrachan-shopify commented 3 months ago

have you solve this issue?

Hi, I have put a proposed fix into my problem description, but I have not done up a PR or anything.

Instead, I just went and recreated my bucket with the gcs command-line tools instead of the web admin, so that bucket.files would not return any folders.

khalidrinaldy commented 3 months ago

have you solve this issue?

Hi, I have put a proposed fix into my problem description, but I have not done up a PR or anything.

Instead, I just went and recreated my bucket with the gcs command-line tools instead of the web admin, so that bucket.files would not return any folders.

thank you, this solved the issue