Moving source between the local project and the org via the Salesforce CLI

[clairebianchi]

The Salesforce CLI team launched the source:deploy and source:retrieve commands in Beta in winter 19. After collecting your feedback we have realized that the overlap between the mdapi and source deploy and retrieve commands is causing confusion about what commands should be used with which workflow (org development or package development). Due to this feedback, we are proposing the following changes:

The Proposal

Overall:

There will only be 2 sets of commands that move source to and from orgs:

• source:push, source:pull - these commands will track changes between the local project and the org

• mdapi:deploy, mdapi:retrieve - these commands will allow for fine-grained or bulk manipulation of source

Both sets of commands will default to maintaining the source file folder layout (i.e. source format), with an option to override the default for deploy and retrieve if you want to work in the older project file format (i.e. mdapi format).

Push and pull will work in source tracked orgs, and deploy and retrieve in all orgs

Other changes:

• We are going to add the additional selectivity options to mdapi:deploy/retrieve that are currently available on source:deploy/retrieve and then deprecate source:deploy/retrieve.

• mdapi:convert will be moved under the force:project topic and likely renamed to project:import. We will also add the additional selectivity options available for source:deploy/retrieve.

• source:convert will be moved under the force:project topic and likely renamed to project:export. We will also add the additional selectivity options available for source:deploy/retrieve.

• The mdapi:delete command with selectivity options will be implemented.

[FishOfPrey]

Should mdapi:deploy and mdapi:retrieve default to the source format rather than the mdapi format?

AFAIK the underlying Metadata API is still working on the original mdapi format. I'd use that as a case for the default behavior to be mdapi format. Then the source format, which doesn't match what the mdapi is doing on the wire, could be the optional behavior.

What will happen to any selectivity options that applied to source:deploy/retrieve that don't make sense to the direct mdapi?

[RonzGore]

I second Daniel's comment here!

[JimBTek]

I think it's ideal to be able to just work with sfdx folder structure and then control pushing it to/from scratch orgs, or to/from sandboxes/production. It's also ideal to not have to worry about accidentally pushing to prod, so the different commands for each are ideal. Also ideal to be able to just retrieve from Prod straight into source folder (sfdx) and trying pushing to a scratch org to quickly get into the sfdx flow. A big issue for adoption for consulting is easily making the switch, so anything around that is delaying a lot of devs it seems. We only use it for ISV projects and not consulting.

As with any big procedural change like this, it would be fantastic to have a flow chart of how to use sfdx cli to deploy/retrieve and push/pull to make it easy to transition. The video in the Falcon Template is very helpful to understand how the internal team envisions the folder structure, and workflow.

Also the old Force-CLI has more flexibility for things like selective deploy/retrieve it seems. it would be nice to easily only deploy some things. On the Consulting side we can't always do a 100% deploy of source from git, and may not want to, so tools like Gearset come in handy to handpick items to deploy.

[catalandres]

+1 to @JimBTek

What will happen if we need to use both flavors (package.xml and DevHub tracked source) of deploy/retrieve in a project? Will it be easy to differentiate between them?

Also, please ensure that no DevHub will be required for package.xml-based deploy/retrieve. We have a project in which the client will not turn on DevHub because their approach to security is that absolutely no one should have any special access rights to production, so we are completely missing out on being able to track changes in metadata.

[FabienTaillon]

I like the naming being clear. Commands with "mdapi" manipulates metadata api format, while commands with "source" manipulates source format.

So instead of deprecating source:deploy/retrieve, why not deprecating mdapi:deploy/retrieve instead, as source format will be the default ? Command would do the same, I'm just talking about the command name here.

Also, why moving convert under project ? From time to time I convert from mdapi to source and never understood why this command requires a project. I also find that "project:import" is not really helping to know what it's really doing, compared to mdapi:convert which is a pretty obvious name.

[catalandres]

@FabienTaillon is indirectly scratching an itch that I have had for a while regarding the naming of mdapi.

The convention in Salesforce CLI is to have nouns in all the components of the command except for the last one. Thus, source:push means "push source" and org:open means "open the org". But with mdapi it gets weird... mdapi:deploy means... "deploy the metadata API"? If anything mdapi could be renamed to metadata, and then we call it "source" when it's follows the "sfdx" structure and "metadata" when it follows the "classic" structure.

[britishboyindc]

I'd really like the ability to use the .forceignore file with just push/pull and have a separate one that works with convert/deploy - specifically around working with Gen 1 packages. The use case here is that I often want files like permission sets or non packaged code to be part of my push/pull, but do not want that to be part of the convert/deploy process since they aren't part of the package. Right now, I don't know of a way to do that and as far as I know it isn't possible?

[gdoenlen]

I personally think force:source:push and force:source:pull should be able to work against any org, source tracked or sandbox / prod.

Additionally, I am completely against renaming the mdapi commands, as it would be a breaking change that could possibly break CI.

[JimBTek]

@britishboyindc Agreed.

the Falcon Template has some scripting for things like that and places in the folder structure for code that is for testing only. the video is helpful. not sure it does exactly what you want, and would be nice to not need a template with scripts to achieve it

[tsalb]

+1 to @gdoenlen

If I am understanding the proposal correctly, I agree with these changes since it will enforce having just one set of commands to pull and push for prod, sandbox and scratch orgs.

Personally, I agree with the renaming proposal to under force:project as it gives a clear intention what I am about to do to my project.

I envision at the end the push/pull of sandbox org development workflows to use force:project:export to bridge the gap to ANT or any CI.

[catalandres]

@gdoenlen There's a feature in OCLIF that allows you to create aliases of commands. You could probably provide backwards compatibility without breaking a sweat.

[gdoenlen]

@catalandres breaking changes are still breaking changes and they shouldn't be taken lightly.

"Never break user space."

[abd3]

I like the proposals from @clairebianchi.

1. Adding the selectivity options will be a great addition to those commands (these allow you to select based on package.xml, based on */ style glob, and based on named metadata)
2. @catalandres makes a nitpicky point about the name 'mdapi' and @gdoenlen points out that changing mdapi to default to source format would be a breaking change. So how about this:
   1. Create a new topic 'metadata' that contains the same commands as in 'mdapi' but defaults to expecting source format, with a flag --mdapi to allow use in the old format.
   2. Deprecate the entire mdapi topic, but keep them functioning as aliases for the 'metadata' commands that use the --mdapi flag
3. For any deprecated commands, consider presenting a warning message but allowing those aliases to keep working indefinitely. People have built tooling around these commands and you should avoid making breaking changes unless you really have to.
4. I personally get confused between source:convert and mdapi:convert and always have to re-read the help text before using them, so I would welcome the project:import/export names
5. adding those selectivity options even for project:import/export is important since it will allow teams to do things like pull down a package.xml relating to an unmanaged package and then convert just that subset of files out of metadata API format and into source format (or vice versa).
6. One thing to consider would be a 'move/destructive' conversion as a flag so that if a team does a partial import/export it will actually remove the original metadata after creating the converted metadata. That will provide an easy way for teams to, say, extract a single field out of the Account object into source format, without having to manually remove a chunk of XML out of the mdapi format.

[alpha-bytes]

I am 100% in favor of bifurcating commands intended to work with source-tracked environments (scratch orgs) and production / sandbox envs. Having additional selectivity on the mdapi commands will be a big win (e.g. -c flag for validate-only deployment).

Additionally in favor of standardizing to one source format - source:convert and mdapi:convert are additional overhead that just set the dev up for heartburn.

Finally, BIG WIN bringing the mdapi:delete command in for parity with the Metadata API. Having to remove components via the UI, or implementing the Metadata API manually, is a big time drag.

[alan-morey]

Current behavior of mdapi:retreieve and mdapi:source should not change, they should continue to operate on the "classic" format, by default and not the new "source" format. This currently provides good feature parity between Ant Migration Tool and sfdx, allowing for gradual transition of old tools and scripts. mdapi commands are working as expected now, if it ain't broke...

I agree with @gdoenlen, the source:push and source:push commands, which already use the new "source" format, should be updated to work with sandboxes and production orgs in addition to scratch orgs, instead of breaking the mdapi commands.

I like @abd3's suggestion for adding a new metadata topic that works with "source" format and having mdapi alias into. Avoids breaking changes and solves naming issue raised by @FabienTaillon and @catalandres.

[VivekMChawla]

@clairebianchi - These are interesting proposals. I'm glad you've started this dialogue with the community. Lots of great comments and suggestions so far, too. It's heartening to see the passion that goes with this toolset. :-)

Here's what I like and what I wish might (and might not) happen.

Two Command Sets To Rule Them All: GREAT IDEA! I really like the idea of having only two "metadata movement" command sets. The combo of source:push/pull and mdapi:deploy/retrieve just feels right, and makes it clear which flavor of on-disk Salesforce metadata you're working with.

Source Push/Pull Work Against Any Org: PLEASE DO! I'm strongly in favor of making source:push/pull automagically work against any org.

• If you're working against a scratch org, you get source synchronization.
• If working against any other org type, source:push/pull overwrites whatever is on the "target" side of the move.

"Source" commands mean I'm dealing with the new decomposed metadata format on disk. Don't make me care too much about how it gets into the target org.

Keep mdapi commands as-is: MUST DO! I agree with @alan-morey that it's very important to keep the mdapi:deploy/retrieve as-is. Specifically, I think they should not deal with "source format" at all.

Metadata API deploy/retrieve are well-known, battle-tested ways of moving Salesforce metadata. Having it work only in a way that's familiar to the ANT Migration Tool makes it that much easier to convice old-school holdouts to dip their toe in the SFDX waters by moving to the Salesforce CLI.

Moving convert commands into the project tree: PLEASE DON'T! I like that the current mdapi:convert and source:convert commands are explicit about what they are acting on. You're taking one type of on-disk metadata and converting it to the other. Perfect! That's all I want to do 90% of the time if I'm building out a rich DevOps toolset for my company.

I agree that it could be useful to have some sort of project:convert command set to deliver higher-level functionality that might appeal to developers. Just don't take away themdapi:convert and source:convert primitives to do so.

Add greater selectivity options to all the "metadata movement" commands: LOVE THIS! Greater control of what gets pulled, pushed, synced, or converted makes a lot of sense. One of the common complaints I see (and have felt) is that changes made to an org sometimes result in changes to SFDX Source that people wish didn't happen. Proper use of .forceignore can help here, but I feel like more control here would be helpful.

[iandrosov]

@clairebianchi Thank you for starting GItHub dialog with the community on these changes, some nice ideas are on this thread. Making separate commands for metadata format vs scratch org new SFDX is a good idea. If SFDX still will require to do convert of a source to use these that is seem like clutter on directory structure in projects.

Here is my idea: mdapi:deploy when used with Scratch ORG source format can automatically convert source format to allow seamless push of code from the same source directory tree and same repository tracking branch to either scratch org or DEV/Sandbox/Prod orgs as a target

mdapi:retrieve - seem to only pull from DEV. Sandboxes in old metadata format. We will need a convert from metadata to SFDX format. I think making SFDX CLI workflow more unified will help developers.

That conversion of source format step may be done by VSCode plugins.

[clairebianchi]

Thank you all for the feedback. I am working on revising the project based on your guidance and will post our final plan here in a few days.

[marktuk]

What's the issue with source:deploy/retrieve ? They've finally allowed me to fully embrace the new source format for my existing projects where I can only work with sandboxes. If they are "merged" with the existing mdapi commands I assume the functionality will stay the same? I agree with others, it would also be a bit strange and confusing using mdapi commands against the source format.

Personally, I'm fairly happy with source:deploy/retrieve as they are, it's clear they intended for deploying/retrieving the source format to/from sandboxes. For scratch orgs we have source:push/pull which doesn't require specifying components/manifest/etc.

The mdapi:deploy/retrieve commands still have their place i.e. working purely with the metadata format. This is still needed because many people will still have projects in the metadata format in version control which can't easily be migrated due to complexity, active development etc.

[Confirm4Crit]

• We are going to add the additional selectivity options to mdapi:deploy/retrieve that are currently available on source:deploy/retrieve and then deprecate source:deploy/retrieve.

Just shouting out that I love this. I want to do precise deploys to prod against git diffs, and so far just running through the git diff files and passing it to source:deploy is working well for sandboxes.

[lopezfra]

Any update on this?

[woutersmet]

Since the confusion seems to be that 'source' can be both interpreted as 'source data structure' and 'source-tracked method of moving data between the local and org', why not make it explicit that the middle part is a NOUN and about the data structure, and the last is a verb and about the method of moving data.

(also see @catalandres 's itch :))

You could for example replace the middle one with 'sourcedata' and 'mdapidata', giving you:

force:sourcedata:push force:sourcedata:pull

force:mdapidata:deploy force:mdapidata:retrieve

...and then what happens in the recent 'magic mix' between the 2 becomes more intuitive: force:sourcedata:deploy force:sourcedata:retrieve

or maybe 'files' is even more clear and explicit than 'data':

force:sourcefiles:push force:sourcefiles:pull

force:mdapifiles:deploy force:mdapifiles:retrieve

...and then what happens in the recent 'magic mix' between the 2 becomes more intuitive: force:sourcefiles:deploy force:sourcefiles:retrieve

Or, MY FAVORITE, because instead of 'data' or 'files', actually 'source' is actually the best noun here for what's being pushed/deployed: how about 'source' vs 'mdapisource'? So is 'source' the short and sweet default, but there's another 'legacy' style source which is mdapisource. Both can be old-fashioned deployed and retrieved, (force:source:deploy and force:mdapisource:deploy), but only 'regular' source can be pulled/pushed the modern way.

[woutersmet]

As an additional thought to my previous comment, you would want to start referring to what you now call 'source format' in a new way, like 'tracking-friendly format' something. Then you can explain things as follows:

'A new sfdx project will by default have its source code and metadata arranged in a modern, tracking-friendly format. You can push/pull between this local source and a scratch org with force:source:push / force:source:pull.

To more drastically deploy/retrieve source in this format, in the old non-tracked way, for example against a developer org, you can also use force:source:deploy and force:source:retrieve.

For the older formatting of source code and metadata, called metadata api format or 'mdapi format', you cannot pull or push but you - can - deploy and retrieve, with mdapisource:deploy and mdapisource:retrieve.'

---

How neat is that? Minimal syntax change, current force:source commands can stay as is, and super intuitive what you are actually doing in the 'old' style with force:mdapisource:deploy / force:mdapisource:retrieve