Temporary folder "shape" not cleared out before `force:org:create` command executes

[ImJohnMDaniel]

Summary

If you have multiple DX projects with varying configurations within each project's project-scratch-def.json files, the system temporary folder that the Salesforce CLI uses to define the shape of the scratch org can include definition files from previous force:org:create executions. This can lead to errors during the scratch org creation that are not present on other machines or from previous scratch org creation events.

Steps To Reproduce:

Repository to reproduce: cli-issue-753

1. execute the following command

   sfdx force:org:create --setdefaultusername --wait 30 --setalias scratch-A --durationdays 1 --definitionfile config/a-project-scratch-def.json

2. examine the contents of the temporary folder used to define the "shape" of the scratch org.

   It's contents are:

       /shape
           /objects
               Account.object
           package.xml 

   The Account.object file contains:

   <?xml version="1.0" encoding="UTF-8"?>
   <Object xmlns="http://soap.sforce.com/2006/04/metadata">
       <recordTypes>
           <fullName>Default</fullName>
           <label>Default</label>
           <active>true</active>
       </recordTypes>
   </Object>

   The package.xml file contains:

   <?xml version="1.0" encoding="UTF-8"?>
   <Package xmlns="http://soap.sforce.com/2006/04/metadata">
       <types>
   
           <members>Account</members>
           <name>CustomObject</name>
       </types>
       <types>
   
       <members>Account.Default</members>
           <name>RecordType</name>
       </types>
   
       <version>50.0</version>
   </Package>

3. execute the following command

   sfdx force:org:create --setdefaultusername --wait 30 --setalias scratch-B --durationdays 1 --definitionfile config/b-project-scratch-def.json

4. examine the contents of the temporary folder used to define the "shape" of the scratch org.

   It's contents are:

       /shape
           /objects
               Account.object
               Case.object
               Contact.object
           /settings
               FieldService.settings
           package.xml 

   The Account.object file remains the same as above.

   The package.xml file now contains:

   <?xml version="1.0" encoding="UTF-8"?>
   <Package xmlns="http://soap.sforce.com/2006/04/metadata">
       <types>
   
           <members>FieldService</members>
           <name>Settings</name>
       </types>
       <types>
   
           <members>Account</members>
           <members>Case</members>
           <members>Contact</members>
           <name>CustomObject</name>
       </types>
       <types>
   
       <members>Account.Default</members>
       <members>Case.Default</members>
       <members>Contact.Default</members>
           <name>RecordType</name>
       </types>
       <types>
   
       <members>Case.DefaultProcess</members>
           <name>BusinessProcess</name>
       </types>
   
       <version>50.0</version>
   </Package>

5. execute the first force:org:create command again. This time with a different alias.

   sfdx force:org:create --setdefaultusername --wait 30 --setalias scratch-FAILS --durationdays 1 --definitionfile config/a-project-scratch-def.json

   The following failure message seen at this point:

   === Component Failures [4]
   TYPE   FILE                                  NAME                 PROBLEM
   ─────  ────────────────────────────────────  ───────────────────  ──────────────────────────────────────────────
   Error  shape/objects/Case.object             Case.Default         Not in package.xml
   Error  shape/objects/Case.object             Case.DefaultProcess  Not in package.xml
   Error  shape/objects/Contact.object          Contact.Default      Not in package.xml
   Error  shape/settings/FieldService.settings  FieldService         Not available for deploy for this organization

This error results from the fact that the "shape definition temp files" from the previous scratch org creation were not cleaned out at the beginning of the third execution of the force:org:create command.

Additional information

SFDX CLI Version(to find the version of the CLI engine run sfdx --version):

sfdx-cli/7.81.0-7b953c80d1 darwin-x64 node-v12.18.3

SFDX plugin Version(to find the version of the CLI plugin run sfdx plugins --core)

@dx-cli-toolbox/sfdx-toolbox-package-utils 0.3.0 (link) /Users/john/workspace/_cli-related/sfdx-toolbox-package-utils
@oclif/plugin-autocomplete 0.2.0
@oclif/plugin-commands 1.3.0 (core)
@oclif/plugin-help 3.2.0 (core)
@oclif/plugin-not-found 1.2.4 (core)
@oclif/plugin-plugins 1.9.1 (core)
@oclif/plugin-update 1.3.10 (core)
@oclif/plugin-warn-if-update-available 1.7.0 (core)
@oclif/plugin-which 1.0.3 (core)
@salesforce/sfdx-diff 0.0.6
@salesforce/sfdx-trust 3.4.3 (core)
alias 1.1.3 (core)
analytics 1.12.1 (core)
auth 1.3.0 (core)
config 1.2.1 (core)
generator 1.1.3 (core)
salesforcedx 50.6.0 (core)
├─ schema 1.0.1 (core)
├─ templates 50.1.0 (core)
├─ salesforce-alm 50.6.0 (core)
├─ @salesforce/sfdx-plugin-lwc-test 0.1.7 (core)
├─ custom-metadata 1.0.10 (core)
└─ apex 0.1.4 (core)
sfdmu 3.7.21
sfdx-cli 7.81.0 (core)
shane-sfdx-plugins 4.43.0
├─ @mshanemc/plugin-streaming 1.1.7
└─ @mshanemc/sfdx-sosl 1.1.0
soqlx-opener 0.1.12

OS and version: MacOS v11.0.1

attention: @amphro, @clairebianchi

[github-actions[bot]]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[github-actions[bot]]

We have determined that the issue you reported exists in code owned by another team that uses only the official support channels. To ensure that your issue is addressed, open an official Salesforce customer support ticket with a link to this issue. We encourage anyone experiencing this issue to do the same to increase the priority. We will keep this issue open for the community to collaborate on.

[git2gus[bot]]

This issue has been linked to a new work item: W-8881495

[github-actions[bot]]

We have determined that the issue you reported exists in code owned by another team that uses only the official support channels. To ensure that your issue is addressed, open an official Salesforce customer support ticket with a link to this issue. We encourage anyone experiencing this issue to do the same to increase the priority. We will keep this issue open for the community to collaborate on.

[azlam-abdulsalam]

@ImJohnMDaniel while this is irritating, It would be great if the cli can print out the path to the shape file and provide an option to retain it, as this is one of the mechanism we use to update shape of the pooled orgs

[yippie]

This just affected my ability to create a 2GP version as well. If 2GP packages are being built against an org with unknown features/settings based on what projects I worked on previously that is very scary. This could mean my package is unexpectedly dependent on a feature I had not specified and had no intent to use.

This needs to be fixed yesterday if I am going to trust this system at all to develop a product.

[nvuillam]

Where can I find the temp shape folder ?

[shetzel]

The shape folder is created in the <os_temp_dir>/shape directory, unless the SFDX_MDAPI_TEMP_DIR env var is set, in which case it will use that directory.

[nvuillam]

Thanks !

I automatically fixed this issue with sfdx-hardis plugin so now I don't have to manually delete stuff before creating a new scratch :)

sfdx hardis:scratch:create

[ImJohnMDaniel]

@mshanemc, @shetzel, @WillieRuemmele -- when you have a moment, can you check on this issue please? It has been 11 months since the last update. Thanks for the help.

[WillieRuemmele]

Hi @ImJohnMDaniel 👋

can you try it out with the org:beta:create command?

[ImJohnMDaniel]

@WillieRuemmele, is the standard force:org:create command still using $TMPDIR/shape?? I just tried to create a scratch org and I don't see the shape folder even being created anymore.

[WillieRuemmele]

@ImJohnMDaniel I don't believe it is, you could set SFDX_MDAPI_TEMP_DIR which should be respected by either command

[ImJohnMDaniel]

@WillieRuemmele, it would appear that the force:org:create command is now setting up a randomized folder name like shape_1649799707998. This effectively resolves this particular issue since each execution of the command would use a new location. Thanks for the help.