Closed kdybicz closed 8 months ago
Keeps happening to us too randomly with 30% rate. No pattern. We just rerun the pipeline and this time it goes.
I agree with @bsz0206 approach. It would be nice to have a validation and retry step, to make sure the credentials file was downloaded without any issues OR to fail with a clear error message
Today, thanks to enabled debug mode, we got some additional details. This time it doesn't seem to be an issue with downloading the credentials, rather a Google error while uploading the binary?
Configs:
* firebase_token:
* service_credentials_file: [REDACTED]
* app_path: /Users/vagrant/deploy/SecretApp.ipa
* app: 1:xxxxxxxxxxxx:ios:yyyyyyyyyyyyyyyy
* release_notes:
* release_notes_length:
* release_notes_file:
* testers:
* groups: team
* flags:
* is_debug: true
* upgrade_firebase_tools: true
App path contains a file, great!! 👍
Service Credentials File is a remote url, downloading it ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 2335 100 2335 0 0 6759 0 --:--:-- --:--:-- --:--:-- 6807
Downloaded Service Credentials File to path: /Users/vagrant/git/credentials.json
-- Checking for existing firebase-tools on PATH...
-- Your machine has firebase-tools@11.30.0, attempting upgrade...
-- Checking your machine type...
-- Downloading binary from https://firebase.tools/bin/macos/latest
######################################################################## 100.0%
#=#=#
##=#=#
##=O#- #
# 2.0%
####### 9.8%
############## 19.6%
###################### 31.9%
############################## 42.9%
########################################## 58.9%
################################################### 71.8%
############################################################### 87.8%
######################################################################## 100.0%
-- Setting permissions on binary...
-- Checking your PATH variable...
-- firebase-tools@12.4.7 is now installed
-- All Done!
Deploying build to Firebase
firebase appdistribution:distribute "/Users/vagrant/deploy/SecretApp.ipa" --app "1:xxxxxxxxxxxx:ios:yyyyyyyyyyyyyyyy" --groups "team" --debug
[2023-08-10T10:14:18.305Z] > command requires scopes: ["email","openid","https://www.googleapis.com/auth/cloudplatformprojects.readonly","https://www.googleapis.com/auth/firebase","https://www.googleapis.com/auth/cloud-platform"]
[2023-08-10T10:14:18.402Z] <<< [apiv2][status] GET https://firebase-public.firebaseio.com/cli.json 200
[2023-08-10T10:14:18.404Z] <<< [apiv2][body] GET https://firebase-public.firebaseio.com/cli.json {"cloudBuildErrorAfter":1594252800000,"cloudBuildWarnAfter":1590019200000,"defaultNode10After":1594252800000,"minVersion":"3.0.5","node8DeploysDisabledAfter":1613390400000,"node8RuntimeDisabledAfter":1615809600000,"node8WarnAfter":1600128000000}
[2023-08-10T10:14:18.428Z] FetchError: Invalid response body while trying to fetch https://www.googleapis.com/oauth2/v4/token: read ECONNRESET
at Gunzip.<anonymous> (/Users/vagrant/.cache/firebase/tools/lib/node_modules/node-fetch/lib/index.js:400:12)
at Gunzip.emit (node:events:539:35)
at Gunzip.emit (node:domain:475:12)
at emitErrorNT (node:internal/streams/destroy:157:8)
at emitErrorCloseNT (node:internal/streams/destroy:122:3)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
Error: Failed to authenticate, have you run firebase login?
Has anyone found a solution for this? Our builds are failing more and more often lately.
Comparing two builds for the same code and settings within 20 minutes apart from each other:
The discrepancies can be found after this line: command requires scopes: |
Failed | Succeed |
---|---|---|
Same issue here...
Hello There,
At the moment the pipeline is working, we encounter issues with other steps relying on Google API. The issue seems related to Google instead of the step itself. You guys can find more information https://github.com/bitrise-steplib/steps-google-play-deploy/issues/153
Hello, still facing this issue
I have prepared a workaround, which works in almost every case. It retries up to three times, and sleeps for 3 seconds between each attempt. Below is a full script.
#source: https://github.com/guness/bitrise-step-firebase-app-distribution/blob/master/step.sh
set -e
THIS_SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
#=======================================
# Functions
#=======================================
RESTORE='\033[0m'
RED='\033[00;31m'
YELLOW='\033[00;33m'
BLUE='\033[00;34m'
GREEN='\033[00;32m'
function color_echo {
color=$1
msg=$2
echo -e "${color}${msg}${RESTORE}"
}
function echo_fail {
msg=$1
echo
color_echo "${RED}" "${msg}"
exit 1
}
function echo_warn {
msg=$1
color_echo "${YELLOW}" "${msg}"
}
function echo_info {
msg=$1
echo
color_echo "${BLUE}" "${msg}"
}
function echo_details {
msg=$1
echo " ${msg}"
}
function echo_done {
msg=$1
color_echo "${GREEN}" " ${msg}"
}
function validate_required_input {
key=$1
value=$2
if [ -z "${value}" ] ; then
echo_fail "Missing required input: ${key}"
fi
}
function escape {
token=$1
quoted=$(echo "${token}" | sed -e 's/\"/\\"/g' )
echo "${quoted}"
}
function validate_required_input_with_options {
key=$1
value=$2
options=$3
validate_required_input "${key}" "${value}"
found="0"
for option in "${options[@]}" ; do
if [ "${option}" == "${value}" ] ; then
found="1"
fi
done
if [ "${found}" == "0" ] ; then
echo_fail "Invalid input: (${key}) value: (${value}), valid options: ($( IFS=$", "; echo "${options[*]}" ))"
fi
}
#=======================================
# Additional functions
#=======================================
function truncate_release_notes {
notes=$1
max_length=$2
original_length=${#notes}
if (( $original_length > $max_length )); then
end_message="..."
cut_limit=$(($max_length-${#end_message}))
echo "${notes:0:$cut_limit}${end_message}"
else
echo "${notes}"
fi
}
#=======================================
# Main
#=======================================
#
# Validate parameters
echo_info "Configs:"
echo_details "* firebase_token: $firebase_token"
echo_details "* service_credentials_file: $service_credentials_file"
echo_details "* app_path: $app_path"
echo_details "* app: $app"
echo_details "* release_notes: $release_notes"
echo_details "* release_notes_length: $release_notes_length"
echo_details "* release_notes_file: $release_notes_file"
echo_details "* testers: $testers"
echo_details "* groups: $groups"
echo_details "* flags: $flags"
echo_details "* is_debug: $is_debug"
echo_details "* upgrade_firebase_tools: $upgrade_firebase_tools"
echo
if [ -z "${app_path}" ] ; then
echo_fail "App path for APK, AAB or IPA is not defined"
fi
case "${app_path}" in
\|\|*)
echo_warn "App path starts with || . Manually fixing path: ${app_path}"
app_path="${app_path:2}"
;;
*\|\|)
echo_warn "App path ends with || . Manually fixing path: ${app_path}"
app_path="${app_path%??}"
;;
\|*\|)
echo_warn "App path starts and ends with | . Manually fixing path: ${app_path}"
app_path="${app_path:1}"
app_path="${app_path%?}"
;;
*\|*)
echo_fail "App path contains | . You need to make sure only one build path is set: ${app_path}"
;;
*)
echo_info "App path contains a file, great!! 👍"
;;
esac
if [ ! -f "${app_path}" ] ; then
echo_fail "App path defined but the file does not exist at path: ${app_path}"
fi
if [ -n "${FIREBASE_TOKEN}" ] && [ -z "${FIREBASE_TOKEN}" ] ; then
echo_warn "FIREBASE_TOKEN is defined but empty. This may cause a problem with the binary."
fi
if [ -z "${firebase_token}" ] ; then
if [ -z "${service_credentials_file}" ]; then
echo_fail "No authentication input was defined, please fill one of Firebase Token or Service Credentials Field."
elif [ ! -f "${service_credentials_file}" ]; then
if [[ $service_credentials_file == http* ]]; then
echo_info "Service Credentials File is a remote url, downloading it ..."
curl $service_credentials_file --output credentials.json
service_credentials_file=$(pwd)/credentials.json
echo_info "Downloaded Service Credentials File to path: ${service_credentials_file}"
else
echo_fail "Service Credentials File defined but does not exist at path: ${service_credentials_file}"
fi
fi
fi
if [ -n "${FIREBASE_TOKEN}" ] && [ -n "${service_credentials_file}" ]; then
echo_warn "Both authentication methods are defined: Firebase Token (via FIREBASE_TOKEN environment variable) and Service Credentials Field, one is enough."
fi
if [ -n "${firebase_token}" ] && [ -n "${service_credentials_file}" ]; then
echo_warn "Both authentication inputs are defined: Firebase Token and Service Credentials Field, one is enough."
fi
if [ -z "${app}" ] ; then
echo_fail "Firebase App ID is not defined"
fi
if [ -n "${release_notes_length}" ] && [ "${release_notes_length}" -gt 0 ] ; then
echo_info "Release notes length is defined: ${release_notes_length}. Truncating release notes ..."
release_notes=$(truncate_release_notes "${release_notes}" "${release_notes_length}")
fi
if [ ! -z "${release_notes_file}" ] && [ ! -f "${release_notes_file}" ] ; then
echo_warn "Path for Release Notes specified, but file does not exist at path: ${release_notes_file}"
fi
# Install Firebase
if [ "${upgrade_firebase_tools}" = true ] ; then
curl -sL firebase.tools | upgrade=true bash
else
curl -sL firebase.tools | bash
fi
# Export Service Credentials File
if [ -n "${service_credentials_file}" ] ; then
export GOOGLE_APPLICATION_CREDENTIALS="${service_credentials_file}"
fi
# Deploy
echo_info "Deploying build to Firebase"
submit_cmd="firebase appdistribution:distribute \"${app_path}\""
submit_cmd="$submit_cmd --app \"${app}\""
## Optional params
if [ -n "${firebase_token}" ] ; then
submit_cmd="$submit_cmd --token \"${firebase_token}\""
fi
if [ -n "${release_notes}" ] ; then
submit_cmd="$submit_cmd --release-notes \"$(escape "$release_notes")\""
fi
if [ -n "${release_notes_file}" ] && [ -f "${release_notes_file}" ] ; then
submit_cmd="$submit_cmd --release-notes-file \"${release_notes_file}\""
fi
if [ -n "${testers}" ] ; then
submit_cmd="$submit_cmd --testers \"${testers}\""
fi
if [ -n "${groups}" ] ; then
submit_cmd="$submit_cmd --groups \"${groups}\""
fi
if [ -n "${flags}" ] ; then
submit_cmd="$submit_cmd \"${flags}\""
fi
if [ "${is_debug}" = true ] ; then
submit_cmd="$submit_cmd --debug"
fi
echo_details "$submit_cmd"
echo
number_of_allowed_retries=3 # Set the number of allowed retries before the loop
retry_count=0 # Initialize retry_count before the loop starts
while true; do
if eval "${submit_cmd}"; then
echo_details "Submission successful."
break
else
((retry_count++))
echo_details "Submission failed, retry #${retry_count}..."
if [ "${retry_count}" -eq "${number_of_allowed_retries}" ]; then # Ensure quotes around variables for safety
echo_details "Third attempt failed. Exiting."
exit 1
fi
sleep 3
fi
done
if [ $? -eq 0 ] ; then
echo_done "Success"
else
echo_fail "Fail"
fi
@Sebulec can you provide instructions how to add this workaround to bitwise workflow? I'm not much into it yet
I found a related post, https://github.com/firebase/firebase-tools/issues/1863#issuecomment-578178000.
It seems as if the API used to authenticate users looks for a specific environment variable that points to the path of your Service Account Credentials File, I have set this up on my pipeline and am having consistent successful runs now of this.
export GOOGLE_APPLICATION_CREDENTIALS=
This can be added to the script for this Step, maybe I will make a PR for this...
export GOOGLE_APPLICATION_CREDENTIALS=$service_credentials_file
https://www.npmjs.com/package/firebase-tools#using-with-ci-systems
@Mounix99 Would you be able to try the above and see if it works for you as well?
Interestingly enough, the step has
# Export Service Credentials File if [ -n "${service_credentials_file}" ] ; then export GOOGLE_APPLICATION_CREDENTIALS="${service_credentials_file}" fi
Inside of the script, but it doesn't seem to be picking it up, maybe it's happening too fast? I did this manually at an earlier point of my pipeline.
Hi @guness, Could you publish a new version with this fix?
@ggdiez It was merged into the master branch, but then reverted for some reason...
Error still occurs, is there any valid solution for this?
@Mounix99 I will take out a release. Until then, you can use a version from develop branch. However, I am not sure if this will be a solution. The issue is not on step, root cause is on the firebase side.
@guness Do you mind sharing how the cause would be on Firebase's side? I'm curious... because if my solution "worked" I believe the real issue is a race condition, like I explained in the PR.
@michalowskim @Mounix99 For the time being, you can use a git reference in your bitrise.yml to use my fork of this repo where I have a branch named fix/race-condition with my solution. You can see all I did was move the export of the environment variable earlier into the file, which allows the machine to have more time to register that new variable before we run the firebase test run command.
In your bitrise.yml file, add the following as a step, or replace the current firebase distribution step reference with:
I just added a new commit, I forgot to export the variable when using the HTTP URL method for the service account, so now it will work with both.
@guness But this change is not present in develop branch. Looks like after merging PR with a fix, someone reverted it(?)
@michalowskim I have reverted it because it was merged to the main branch initially. Ideally, it should be merged to develop the branch first. Now I am trying to merge it develop with over original PR. the worst case, I will create a separate PR.
@guness Do you mind sharing how the cause would be on Firebase's side? I'm curious... because if my solution "worked" I believe the real issue is a race condition, like I explained in the PR.
@AnthonyDadeWT I think I confused this with another issue with similar symptoms. I am about the take out the release but it seems you have targetted to main branch instead of develop on your PR. can you please create another PR targeting develop instead? I wasn't able to do it without losing your contribution.
@guness
Yes, Here is the PR. thanks for letting us know!
https://github.com/guness/bitrise-step-firebase-app-distribution/pull/58
I can see that develop branch with fix was merged already to master, but can you make an actual release so bitrise could see the update? @guness
@michalowskim I did not know a release
is required. I had the tag set already, but now a release
is also created.
@guness Do you have any idea why bitrise still sees the 0.10.0 version as the latest one?
@guness Do you have any idea why bitrise still sees the 0.10.0 version as the latest one?
the PR is not merged yet:
We have a similar issue to #29 which was closed but doesn't seem to be a solution. It's strange as the problem is showing up only from time to time. Any idea what might be wrong?