package version create --async-validation not immediately returning packageId

MarcDBehr commented 1 month ago

Note Before you submit your issue, make sure that:

You're using the latest version of Salesforce CLI.

You've searched both open and closed issues for related posts.

You've used the doctor command to diagnose common issues.

You understand that GitHub Issues don't adhere to any agreement or SLA.

If you require immediate assistance, use official channels such as Salesforce Customer Support.

Summary

According to the documentation (https://developer.salesforce.com/docs/atlas.en-us.pkg2_dev.meta/pkg2_dev/sfdx_dev_dev2gp_create_pkg_ver.htm) and this post (https://trailhead.salesforce.com/trailblazer-community/feed/0D54V00007eqMri), the --async-validation flag should result in a

Steps To Reproduce

We have a package that has 8 package dependencies. These include managed packages and other local 2G packages. Creating new package versions takes between 20-30 minutes currently, so I wanted to try using the new --async-validation flag to see if it would help us in speeding up our CI/CD processes.

Ran the command

$ sf package version create --package xxxx_alarms -x  -c --async-validation

Expected result

Per the documentation

Version create.... Create version status: PerformingValidations
The validations for this package version are in progress, but you can now begin testing this package version.
To determine whether all package validations complete successfully, run "sf package version create report --package-create-request-id  08cxx", and review the Status.
Async validated package versions can be promoted only if all validations complete successfully.
Successfully created the package version [08cxx. Subscriber Package Version Id: 04txx
Package Installation URL: https://login.salesforce.com/packaging/installPackage.apexp?p0=04txx
As an alternative, you can use the "sf package:install" command.

So a Subscriber Package Version Id should have been returned immediately that I could install as other validations continued.

Actual result

Got back a message telling me to query for status

Warning: Record types defined in the scratch org definition file will stop being capitalized by default in a future release.
Set the `org-capitalize-record-types` config var to `true` to enforce capitalization.
Version create.... Create version status: Initializing
Package version creation request status is 'Initializing'. Run "sf package:version:create:report -i 08cPJ00000003NhYAI" to query for status.

Making the query, I got

$ sf package:version:create:report -i 08cPJ00000003NhYAI
=== Package Version Create Request

 Name                          Value                                   
 ───────────────────────────── ─────────────────────────────────────── 
 ID                            08cPJ00000003NhYAI                      
 Status                        Verifying Features And Settings         
 Package Id                    0Hof4000000Gmj1CAC                      
 Package Version Id                                                    
 Subscriber Package Version Id                                         
 Tag                                                                   
 Branch                                                                
 Created Date                  2024-07-24 07:37                        
 Installation URL                                                      
 Created By                    005f4000000n3moAAA                      
 Converted From Version Id     Request is in progress. ID unavailable.

So, no Subscriber Package Version Id yet...

10-15 minutes later...

=== Package Version Create Request

 Name                          Value                                   
 ───────────────────────────── ─────────────────────────────────────── 
 ID                            08cPJ00000003NhYAI                      
 Status                        Verifying Dependencies                  
 Package Id                    0Hof4000000Gmj1CAC                      
 Package Version Id                                                    
 Subscriber Package Version Id                                         
 Tag                                                                   
 Branch                                                                
 Created Date                  2024-07-24 07:37                        
 Installation URL                                                      
 Created By                    005f4000000n3moAAA                      
 Converted From Version Id     Request is in progress. ID unavailable.

still no Subscriber Package Version Id

Eventually (I started doing other work, so stopped checking for updates)

 Name                          Value                                                                             
 ───────────────────────────── ───────────────────────────────────────────────────────────────────────────────── 
 ID                            08cPJ00000003NhYAI                                                                
 Status                        Success                                                                           
 Package Id                    0Hof4000000Gmj1CAC                                                                
 Package Version Id            05iPJ0000006gFtYAI                                                                
 Subscriber Package Version Id 04tPJ000000UvFNYA0                                                                
 Tag                                                                                                             
 Branch                                                                                                          
 Created Date                  2024-07-24 07:37                                                                  
 Installation URL              https://login.salesforce.com/packaging/installPackage.apexp?p0=04tPJ000000UvFNYA0 
 Created By                    005f4000000n3moAAA

Got the ID. Generally this takes 20-30 minutes, so I don't thing the --async-validation flag made any difference

System Information

{
  "architecture": "darwin-arm64",
  "cliVersion": "@salesforce/cli/2.50.6",
  "nodeVersion": "node-v20.15.0",
  "osVersion": "Darwin 23.5.0",
  "rootPath": "/Users/mdbehr/.local/share/sf/client/2.50.6-6e9d7f7",
  "shell": "bash",
  "pluginVersions": [
    "@dxatscale/sfpowerscripts 25.5.3 (user)",
    "@oclif/plugin-autocomplete 3.1.6 (core)",
    "@oclif/plugin-commands 3.3.4 (user)",
    "@oclif/plugin-help 5.2.17 (user)",
    "@oclif/plugin-not-found 3.2.10 (core)",
    "@oclif/plugin-plugins 5.3.4 (core)",
    "@oclif/plugin-search 1.2.2 (core)",
    "@oclif/plugin-update 4.4.7 (core)",
    "@oclif/plugin-version 2.2.6 (core)",
    "@oclif/plugin-warn-if-update-available 3.1.8 (core)",
    "@oclif/plugin-which 3.2.6 (core)",
    "@salesforce/analytics 1.4.22 (user)",
    "@salesforce/cli 2.50.6 (core)",
    "apex 3.2.4 (core)",
    "auth 3.6.33 (core)",
    "data 3.5.5 (core)",
    "deploy-retrieve 3.9.13 (core)",
    "info 3.3.17 (core)",
    "limits 3.3.17 (core)",
    "marketplace 1.2.17 (core)",
    "org 4.3.4 (core)",
    "packaging 2.7.0 (core)",
    "schema 3.3.19 (core)",
    "settings 2.3.8 (core)",
    "sobject 1.4.19 (core)",
    "source 3.5.3 (core)",
    "telemetry 3.6.1 (core)",
    "templates 56.3.2 (core)",
    "trust 3.7.13 (core)",
    "user 3.5.17 (core)"
  ]
}

Additional information

github-actions[bot] commented 1 month ago

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

github-actions[bot] commented 1 month ago

Hello @MarcDBehr :wave: None of the versions of sf you shared match the latest release.

Shared: 2.50.6 Latest: 2.51.6

Update to the latest version of Salesforce CLI (docs) and confirm that you're still seeing your issue. You can also try the rc and nightly releases! (docs)

After updating, share the full output of sf version --verbose --json

MarcDBehr commented 1 month ago

I was using 2.50.6 (which was the latest version yesterday when I ran the command). I upgraded to 2.51.6 today and tried again

{
  "architecture": "darwin-arm64",
  "cliVersion": "@salesforce/cli/2.51.6",
  "nodeVersion": "node-v20.15.0",
  "osVersion": "Darwin 23.5.0",
  "rootPath": "/Users/mdbehr/.local/share/sf/client/2.51.6-e49d0be",
  "shell": "bash",
  "pluginVersions": [
    "@dxatscale/sfpowerscripts 25.5.3 (user)",
    "@oclif/plugin-autocomplete 3.1.7 (core)",
    "@oclif/plugin-commands 3.3.4 (user)",
    "@oclif/plugin-help 5.2.17 (user)",
    "@oclif/plugin-not-found 3.2.11 (core)",
    "@oclif/plugin-plugins 5.3.7 (core)",
    "@oclif/plugin-search 1.2.3 (core)",
    "@oclif/plugin-update 4.4.9 (core)",
    "@oclif/plugin-version 2.2.8 (core)",
    "@oclif/plugin-warn-if-update-available 3.1.9 (core)",
    "@oclif/plugin-which 3.2.8 (core)",
    "@salesforce/analytics 1.4.22 (user)",
    "@salesforce/cli 2.51.6 (core)",
    "apex 3.2.5 (core)",
    "auth 3.6.36 (core)",
    "data 3.5.5 (core)",
    "deploy-retrieve 3.9.17 (core)",
    "info 3.3.18 (core)",
    "limits 3.3.18 (core)",
    "marketplace 1.2.18 (core)",
    "org 4.3.5 (core)",
    "packaging 2.7.1 (core)",
    "schema 3.3.19 (core)",
    "settings 2.3.8 (core)",
    "sobject 1.4.19 (core)",
    "source 3.5.5 (core)",
    "telemetry 3.6.2 (core)",
    "templates 56.3.3 (core)",
    "trust 3.7.13 (core)",
    "user 3.5.18 (core)"
  ]
}

I also tried putting the package name in quotes (as suggested in the community thread (https://trailhead.salesforce.com/trailblazer-community/feed/0D54V00007eqMri)

sf package version create --package "xxxx_alarms" -x -c --async-validation

and I got back

Warning: Record types defined in the scratch org definition file will stop being capitalized by default in a future release.
Set the `org-capitalize-record-types` config var to `true` to enforce capitalization.
Version create.... Create version status: Queued
Package version creation request status is 'Queued'. Run "sf package:version:create:report -i 08cPJ00000003QvYAI" to query for status.

So, no change.

sethmuss commented 1 month ago

Hello @MarcDBehr , thanks for reaching out. I can see why it's confusing. The --async-validation should result in a package version (04t) "sooner" than it would otherwise but not immediately, async-validation will be most beneficial if there are lots of tests that will be run. If you use the --wait flag it might(?) make your CI/CD more deterministic so you don't have to come back and poll.

Have you considered using --skip-validation for CI? (I realize those version can't be promoted, but maybe that could fit into your pipeline...)

Thanks, Seth

MarcDBehr commented 1 month ago

Hi @sethmuss - we do utilize the --wait flag during the ci/cd (currently we set it to 60 minutes) and that works fine. The --skip-validation would be possible, but since we plan on promoting the version that is created if everything works, we don't use it (basically we would either need to wait now, or wait later, and since we are not usually making that many changes, the wait now seems to work for us).

Obviously I was excited to see a "faster" method of moving along was available, but if all it does is slightly shorten the time, there is no benefit for me. Maybe a documentation update would be in order, since the way that it is written, It seemed a little too good to be true :-)

github-actions[bot] commented 1 month ago

We have determined that the issue you reported exists in code owned by another team that uses only the official support channels. To ensure that your issue is addressed, open an official Salesforce customer support ticket with a link to this issue. We encourage anyone experiencing this issue to do the same to increase the priority. We will keep this issue open for the community to collaborate on.

sethmuss commented 1 month ago

@MarcDBehr : I get that. --skip-validation can come in very handy, but some use-cases aren't a fit. We know 2GP/Unlocked build times are a pain point and we're hoping to address as soon as we can.

I'll pass along the doc update idea

avesolovksyy commented 1 week ago

@sethmuss Maybe you happen to know, how can we get back test results execution (i.e. failures with stack trace) if package version has been created with --async-validation option? I've tried using sf package version create report, but seems like it is just returns success right away when package version is created and no mentioning of tests being in progress etc.

Also --async-validation is not mentioned in sf package version create --help output command (I hope it is not in beta or pilot, but GA already) - would be really great to have it there along with explanations/examples how to get to know whether tests succeeded or failed eventually - because if failed such package version cannot be promoted, so we need to fix it to be able to upgrade production.

forcedotcom / cli