"briefcase clean" command to delete all build artifacts

What is the problem or limitation you are having?

I opened a PR for discussion earlier that jump started a discussion but as i've gone through the workflow a bit more I realized that my quick off the cuff implementation would not have provided the feature i actually want. Here's a concrete explanation of why a briefcase clean or similar option be useful to me and why the current situation is a bit rough:

Situation

I have setup a macos Xcode project with bootstrap create and then made manual minor modifications to the project.pbxproj to get a couple of compile flags in there that i want. i would like to be able to clean up build artifacts but not have to run briefcase create all over again.

My current workflow for cleaning and rebuilding looks something like this:

rm -fr build/illmaticapp/macos/xcode (not strictly necesssary bc step 2 will do this anyways but illustrative)
briefcase create macos Xcode
git checkout build/illmaticapp/macos/xcode/Illmatic App.xcodeproj/project.pbxproj (to get my changes back - i have my customizations stored in version control)
briefcase build macos Xcode

Describe the solution you'd like

Given that the .xcodeproj dir is a product of briefcase create I would like an approach to cleaning build artifacts that doesn't also blow away the create artifacts.
Above and beyond that basic functionality ideally briefcase clean would also have some kind of intelligence and or optionality around blowing away 3rd party app_packages for the various python package / chip architecture combinations because as far as i know those aren't always really "build products" of the app being developed[^1], they're just fixtures downloaded from pypi or whatever and avoiding redownloading them would speed things up considerably.

The build/illmaticapp/macos/xcode dir has a whole lot of stuff going on though so as a semi-new user it's very unclear to me what should be deleted when i'm trying to delete build artifacts so i end up having to recreate from scratch every time.

Describe alternatives you've considered

We discussed on my PR the possibilities of:

rm -fr build/macos/ (not ideal for reasons outlined above)
implement as briefcase clean macos Xcode (@freakboy3742's suggestion which is in line with classic hits of the 1970s like make clean and makes a lot of sense to me for that reason)
implement as briefcase build macos Xcode --clean (my original suggestion, which made sense to me as a new user - the first place i was looking for any kind of build cleanup option was by running briefcase build --help)

Additional context

No response

[^1]: At least, as far as I know. Briefcase's packaging has worked so well for me that I've never had to look into the details of whate is or isn't happening as far as importing 3rd party packages into the bundle other than this situation, where i'm looking to be absolutely sure no artifacts of previous builds could possibly be interfering / causing problems with the next build.

I understood your original request from #1830 as essentially "a wrapper around rm -rf build/macOS/xcode". That's not hard to do manually, but given an implementation would be reasonably straightfoward, I can see how it might be a nice convenience.

However, based on this description, it sounds like you're talking about a more fine grained "clean" from the perspective of the Xcode (or whatever) project itself. This is a lot harder for me to get behind.

In terms of the broader goals of Briefcase, any manual modification to the build folder is a bug - it should be considered entirely ephemeral. If you've got manual modifications you need to make to the templated project, the current recommended suggestion is to fork the template for your backend of choice, and make the modifications there.

In terms of your specific workflow, it isn't clear to me why an "Xcode clean" is needed at all. You mention "cleaning up build artefacts"... but... why? What artefacts need to be cleaned up? If you are making manual modifications to the Xcode project file... make those changes and rebuilt. Why is a "clean" required at all? Xcode will do an updated build when you modify the Xcode project.

In terms of the broader goals of Briefcase, any manual modification to the build folder is a bug

I guess my question would be why are things created by briefcase create in the build folder at all? shouldn't the build folder be fore things created by briefcase build? at a minimum that's an unexpected directory structure to me as a newish user of briefcase. principle of least surprise and all that.

In terms of your specific workflow, it isn't clear to me why an "Xcode clean" is needed at all.

generally speaking you're right to think that in a perfect world it shouldn't be necessary to have a clean command (either here in briefcase, in Xcode, or even in something like make). probably 98% of the time I or anyone else decides to do a full clean and rebuild it has no impact other than peace of mind because i can be absolutely sure that there's a 0% chance xcode is link to something left over from the last build... but that other 2% of the time you might have Xcode trying to link against the wrong build file or similar.

(also peace of mind is valuable, but i digress)

bigger picture just as far as xcode's "clean build folder" option IMHO xcode's heuristic where it decides what build steps need to be run after a recent set of changes is (again IMHO) close to fundamentally broken. it's way too easy to create an xcode build process here you assume xcode is going to do Step X every time you press "build" but because of the way you've configured the build xcode's heuristic just decides that doesn't need to run Step X because it ran it during the last build. seen it happen many, many times just in the last few weeks (not here, but in a react native app with a lot of cocoapods dependencies). in particular xcode seems really good at getting itself into a situation where cleaning the build dir is fully necessary if anything went wrong during the last build, at least in the react native project i've been working on simultaneously as my briefcase app.

If you've got manual modifications you need to make to the templated project, the current recommended suggestion is to fork the template for your backend of choice

just $0.02 but forking a whole separate template repo to change 1 line is kind of a PITA compared to just making my desired 1-2 lines of changes to a .pbxproj file that is under version control in the same repo that does the actual building. it felt weird at first maybe to commit files in build/ but as a practical matter it's worked out great and gave me a far more reproducible and (at the time, when i had no idea wtf briefcase build was doing) less confusing build process. i also get alerted by git to any change briefcase makes to my xcodeproj file, which gave me a lot more confidence that i understood what briefcase and xcodebuild were actually doing.

FWIW i did this - commit the .xcodeproj files generated by macos Xcode - almost from the first day i used briefcase and if yr curious about the thought process of a new briefcase user here's roughly how it went:

Run briefcase build macos app, fail to get an app that works the way i need because of `LSUIElement or whatever missing build setting (specifics are unimportant)
Run briefcase build macos Xcode because just based on the same ("Xcode") i guessed that maybe that will generate an xcodeproj i can then just open with actual the Xcode IDE to investigate.
Peruse build/app/macos/xcode dir and lo and behold there's an Xcode project.
Open the generated .xcodeproj in the XCode IDE and use Xcode GUI to add the build settings i need as per whatever Medium post i found on the internet explaining what params were neceessary. at the time i had almost never used Xcode and didn't realize that a .pbxproj file is just an oversized plist dictionary so using the actual Xcode IDE seemed the safest bet.
try to build macox Xcode again and voila the first time i got a briefcase build to work.

as a nice side effect i was able to observe what xcode itself changes in .pbxproj etc when you adjust various configurations and build settings which has proved very helpful in a lot of ways.

fwiw one of the things that made me raise my eyebrow about why perhaps files templated by briefcase create shouldn't be thought of as "built" in the same way as the output of briefcase build is the fact that the build/ folder has a subfolder way down inside also called build/

build/illmaticapp/macos/xcode/build

there's nothing inherently wrong with it but it does kind of imply that there's some silent semantic overloading going on with the word "build"

I guess my question would be why are things created by briefcase create in the build folder at all? shouldn't the build folder be fore things created by briefcase build? at a minimum that's an unexpected directory structure to me as a newish user of briefcase. principle of least surprise and all that.

The naming follows the historical Python convention of putting intermediate artefacts in build, and final distributable artefacts in dist.

The complication is that Briefcase requires a step before "invoking a compilation step", because the project sources don't contain all the code needed to perform the "compilation". We call the "fill in the blanks" step "create", and the compilation step "build".

If the name create bothers you, don't use it. briefcase build will infer a create if the project hasn't been rolled out yet.

In terms of your specific workflow, it isn't clear to me why an "Xcode clean" is needed at all.

generally speaking you're right to think that in a perfect world it shouldn't be necessary to have a clean command (either here in briefcase, in Xcode, or even in something like make). probably 98% of the time I or anyone else decides to do a full clean and rebuild it has no impact other than peace of mind because i can be absolutely sure that there's a 0% chance xcode is link to something left over from the last build... but that other 2% of the time you might have Xcode trying to link against the wrong build file or similar. ... bigger picture just as far as xcode's "clean build folder" option IMHO xcode's heuristic where it decides what build steps need to be run after a recent set of changes is (again IMHO) close to fundamentally broken.

I can't speak to your personal experience, but I can't say this has ever been the case for me. The only issues I've ever had with a "clean" build have more to do with OS-level caching, or artefacts that are left on disk in the installed location for an iOS simulator. In this case a "clean" build doesn't fix anything; you need to delete the app from the simulator (and sometimes restart the simulator).

I'll also note that, in terms of Briefcase's design approach - if your build folder is ephemeral, and you're hitting one of these "2% cases" that you describe, deleting the build folder and recreating the project solves the issue completely.

just $0.02 but forking a whole separate template repo to change 1 line is kind of a PITA compared to just making my desired 1-2 lines of changes to a .pbxproj file that is under version control in the same repo that does the actual building.

The key word in my reply is currently. I make no claim that this is a great developer experience; however, it's an approach that is consistent with the broader goal of Briefcase that the build folder is entirely ephemeral.

As I've indicated on other tickets (#1861), we're open to adding additional configuration points that reduce the need for manual modifications to the Xcode (or any other) project template.

it felt weird at first maybe to commit files in build/ but as a practical matter it's worked out great and gave me a far more reproducible and (at the time, when i had no idea wtf briefcase build was doing) less confusing build process.

If you're having success with this technique, then good luck with that. What I'm flagging is that, as a project, we make zero guarantees that this workflow will continue to work, and we're not going to make feature modifications designed to support or reinforce this use case.

Run briefcase build macos app, fail to get an app that works the way i need because of `LSUIElement or whatever missing build setting (specifics are unimportant)

To be clear: is this "briefcase didn't work" or "I've got a very specific set of requirements in my app that aren't part of the advertised feature set of Briefcase"? To me, it sounds like the latter.

fwiw one of the things that made me raise my eyebrow about why perhaps files templated by briefcase create shouldn't be thought of as "built" in the same way as the output of briefcase build is the fact that the build/ folder has a subfolder way down inside also called build/

You may have noticed that Briefcase supports multiple operating systems, using the platform-provided tools. Xcode uses a subfolder called "build". So does Gradle when building for Android, but the location inside the generated code is different. On Windows, Visual Studio uses a folder named "Release".

We're not in control of how the platforms structure their compilation artefacts - we have to accomodate what they provide.

And, again, from Briefcase's perspective - you shouldn't ever need to look in the top level build folder. It's a collection of ephemeral intermediate artefacts. The fact that there's another folder named "build" deep in the generated code isn't a detail that has any external impact on the user.

This discussion is rapidly ratholing into microlitigating some fairly fundamental aspects of the design of Briefcase, unrelated to the original topic of this ticket. To summarise my current position:

I support the introduction of briefcase clean as a top level command, as an analog of rm -rf build/...
I could potentially get behind a --clean argument on the build command, which would be a hook into the "IDE-level clean". It would be a no-op for backends like macOS app; but on macOS Xcode, it would invoke xcodebuild clean (or whatever) to do a project-level clean prior to the "normal" build. It's a niche use case, but it's a feature of pretty much every IDE, so I guess it doesn't hurt to expose it.

The original PR was effectively (1), implemented using the interface for (2).

if your build folder is ephemeral, and you're hitting one of these "2% cases" that you describe

just to be clear i've never had any issue along these lines with briefcase. briefcase has worked impressively well for me. where i've run into this issue is with the React Native part of my app which i'm bundling together with the briefcase part. the underlying problem is that Xcode is good at understanding when it needs to recompile / link / whatever code changes to source files managed by Xcode but the approach it uses to figure out whether build scripts need to be re-run is (IMHO) fundamentally extremely broken when it comes to stuff not managed by Xcode. react native is kind of custom made to run into this issue.

unfortunately bc i had these frustrating issues with the React Native part of my app I found myself wanted to just clear out all my app's build artifacts bc at the time i had no idea what was the actual problem (i know a bit better now) and part of my app is built by briefcase.

What I'm flagging is that, as a project, we make zero guarantees that this workflow will continue to work,

good to know, but also kind of expected. given that my .xcodeproj is in git i'll at least see any incoming changes to the xcode project made by briefcase build which is good enough for me.

To be clear: is this "briefcase didn't work" or "I've got a very specific set of requirements in my app that aren't part of the advertised feature set of Briefcase"? To me, it sounds like the latter.

you are correct; it was the latter.

As I've indicated on other tickets (https://github.com/beeware/briefcase/issues/1861), we're open to adding additional configuration points that reduce the need for manual modifications to the Xcode (or any other) project template.

the more i understand about how briefcase works the more i think you could almost totally solve this just by adding a couple of these additional configuration points.

beeware / briefcase