[Experiment] The saved credit cards cannot be synced with other devices on Firefox 116 release channel

[SoftVision-CosminMuntean]

[Notes]

• On the Firefox 116.0 release installed from Test Flight the "Autofill Credit Cards" is not enabled by default. But, after enrolling in the Treatment branch of the experiment and logging into a FxA account, the saved credit cards are not synced.
• The issue is also reproducible if you save a credit card on iOS, the saved credit card is NOT synced on the Desktop.
• Please note that the Passwords are correctly synced, only the credit cards are not synced.
• On Firefox 116.0 custom builds made with Xcode with the developer channel have the "Autofill Credit Cards" feature enabled by default, and the saved cards can be successfully synced, even though you are enrolled or not in the experiment.

[Affected versions]:

• Firefox Release 116.0 (33025)

[Affected Platforms]:

• iOS 16.5 - iPhone 13 mini

[Prerequisites]:

• Have the latest Firefox Firefox Release 116.0 (33025) installed.
• Have a valid FxA account with at least one saved credit card.

[Steps to reproduce]:

1. Open Firefox Release 116.0 (33025).
2. In the Settings menu tap 5 times on the "Firefox 116.0 (33025)" option to enable debug mode.
3. Tap on the "Experiments" option.
4. Tap on the "Edit" option from the "Experiments" screen.
5. Toggle the "Use Preview Collection" option.
6. Restart the browser 3 times.
7. Open the browser and repeat Steps 1-3.
8. Tap on the "iOS CC Autofill" experiment.
9. Tap on the "treatment-a".
10. Go back to the Settings menu and "Sign in" to the FxA account.
11. After the sync is complete, tap the "Autofill Credit Cards" option.
12. Verify the synced saved credit cards.

[Expected results]:

• All the saved cards are successfully synced.

[Actual results]:

• No credit cards are synced.

┆Issue is synchronized with this Jira Task

[abodea]

Hello, @nbhasin2 👀

[nbhasin2]

I believe I found the issue and why it doesn't work

The above method shows that enablement is false when user first installs the app an experiment is false / not running however when user gets the experiment it still remains false so basically sync never happens for credit card even though in the settings it is enabled

[nbhasin2]

cc @dnarcese I am looking into a fix for this atm

[SoftVision-CosminMuntean]

@nbhasin2 I have re-tested the issue on the new Firefox 116.0 (33117) build and Firefox 117.0 (33126) build installed from Test Flight on two iOS devices.

Unfortunately, we can still reproduce the issue with these release builds. After enrolling in the experiment in the treatment branch, the autofill credit card feature is enabled, and cards can be added, but the saved cards are not synced. We have used two different FxA accounts, a new one and an older one, we have also manually synced and verified that the FxA accounts have the credit cards option enabled, but the saved cards are still not synced.

I have also tried with the latest Firefox 9000 (33131), but on these builds the feature is enabled by default, and after enrolling in the experiment, the saved cards are successfully synced.

Please let me know if you need any other information.

[nbhasin2]

cc @lougeniaC64 still broken it seems

[SoftVision-CosminMuntean]

While performing more investigation on this issue, I have noticed the following:

• If I disable the "Credit Cards" option from the "Firefox Accounts" screen on iOS device then go back to the "Settings" screen and then again to the "Firefox Accounts", the "Credit Cards" option is automatically re-enabled. In order to disable the "Credit Cards" option on the iOS device, I need to disable the option from Firefox Desktop, sync the options, then also sync the options on the iOS device.
• This happens also if I want to enable the "Credit "cards" option.

I am not sure if this is related to this issue, here is a screen recording.: https://drive.google.com/file/d/1uI6twhIpIsJxcvc6qaggXKNAugsqAg19/view?usp=sharing

[SoftVision-CosminMuntean]

I guess this is related to the issue, because I just tested the above behavior in Firefox 9000 latest, and here, the "Credit Card" option remains disabled.

[lougeniaC64]

cc @lougeniaC64 still broken it seems

Just want to note that I followed the steps in the screen recording and I'm not able to replicate this locally.

[lougeniaC64]

While performing more investigation on this issue, I have noticed the following:

* If I disable the "Credit Cards" option from the "Firefox Accounts" screen on iOS device then go back to the "Settings" screen and then again to the "Firefox Accounts", the "Credit Cards" option is automatically re-enabled. In order to disable the "Credit Cards" option on the iOS device, I need to disable the option from Firefox Desktop, sync the options, then also sync the options on the iOS device.

* This happens also if I want to enable the "Credit "cards" option.

@SoftVision-CosminMuntean After the credit cards engine is re-enabled, what is the desktop credit card enablement?

[SoftVision-CosminMuntean]

@lougeniaC64 This is only reproducible with the Firefox release builds installed from Test Flight, where the autofill credit cards feature is disabled by default and we are using the experiment to enable the feature. From what I have observed the local builds are made on the developer channel where the feature is already enabled by default.

@SoftVision-CosminMuntean After the credit cards engine is re-enabled, what is the desktop credit card enablement?

On the Desktop side, nothing is changed because on mobile the option always returns to the state that it is set on the Desktop. What I am trying to say is that on mobile you cannot enable or disable the "Credit Card" option, you need to change it from the Desktop if you want to change the option. And this also happens only on Firefox release builds from Test Flight where the Autofill Credit Card feature is disabled by default and is enabled through the experiment.

[lougeniaC64]

Are you using the hidden menu in Settings to enable the experiment?

Also, I asked about the desktop enablement because I was trying to gauge whether the disabling of the credit cards engine on the local device is sent to the server and changes the desktop enablement. From what you are saying, it seems the local device credit card engine enablement is overwritten locally and never affects the desktop enablement. Is this true?

[SoftVision-CosminMuntean]

Are you using the hidden menu in Settings to enable the experiment?

Yes, I am using the hidden menu.

Also, I asked about the desktop enablement because I was trying to gauge whether the disabling of the credit cards engine on the local device is sent to the server and changes the desktop enablement. From what you are saying, it seems the local device credit card engine enablement is overwritten locally and never affects the desktop enablement. Is this true?

I am sorry if I am using the wrong terms, but I am a bit confused about how the settings are synced. However, in my case, it looks like the enablement on mobile is overwritten by the Desktop after each sync. Because, for example in the screen recording provided above, the option keeps resetting even though I manually change it locally on the mobile device. On the Desktop, the option remains enabled.

[lougeniaC64]

@SoftVision-CosminMuntean No worries. The way that engine enablement syncing works at a high level is the that any local enablement changes on a device that occur since the last sync will be sent to the sync server on the next sync and propagated to all connected devices.

This leads me to believe that when you disabled the credit card engine on your iOS device it was overwritten before it could be synced and sent to the server. I don't think desktop would overwrite the local changes in this instance because there were no manual changes to the credit card enablement on desktop that would have been sent to the server.

If you can grab the sync logs, we can check for certain what was sent and received from the server.

[SoftVision-CosminMuntean]

I have verified this issue and is no longer reproducible with the latest Firefox 116.0 (33174) build. Tested on an iPhone 13 min - iOS 16.5.

• After enrolling in the experiment I managed to sync the saved credit cards with other devices.