Command Request: tup clean

[bradjc]

I would like to see a tup clean command. I'm not sure if this was a conscious decision to not have this command or if nobody has asked for it.

Obviously this command would remove all generated output files.

I have two reasons:

1. To get compiler warnings back. If I see some warnings but don't deal with them right away, there isn't an easy way to get the code to recompile so that the warnings reappear.
2. There's just something nice about having a clean working directory.

[rendaw]

This has actually been requested a number of times in the mailing list to my knowledge: https://groups.google.com/d/topic/tup-users/4f6gF_lI0pc/discussion https://groups.google.com/d/topic/tup-users/Kxw1nnXK12o/discussion https://groups.google.com/d/topic/tup-users/1G5bn2dDoG4/discussion

In summary, I believe not having a clean command was intentionally omitted.

As for 2., I think the intended solution for that was variants - are those not working on Mac?

I compulsively fix warnings, so I haven't really had that problem. Just brainstorming, but could you add a "> %B.buildlog.txt" to your build commands? That way you could just grep your logs to find warnings for the entire project, and it wouldn't require any rebuilding. I think that's probably what I would do.

Another solution I've seen a couple times is using git/hg's cleaning commands to remove untracked files.

[bradjc]

Ah I see I should clearly be following the google group (although I'd much rather just see everything on github).

Variants solve many problems, but I wish there was a less clumsy way of getting them to re-build (for whatever reason I may want to rebuild them):

rm -r build
tup variant configs/*
tup upd
    tup warning: variant directory 'build' was deleted outside of tup. This directory will be re-created, unless the corresponding source directory was also removed.

In contrast:

tup clean
tup upd

is just sexy.

[ppannuto]

I was reading through the links to old discussions you posted, and the most relevant ones are in the second link. The issue is precisely when things outside the build environment change. There are many tools that rely on external environment (e.g. /etc/tool.conf) to dictate their behavior. When testing changes, there is no way to tell tup that such a configuration file has changed. I suppose you could hack around this by adding a benign flag to the invokation of the relevant command, but this becomes very manual and hard to track.

Ultimately, Mike's consistant argument seems to be that "if a build system requires a 'clean', it is broken". I understand his point, but my response is that until tup is capable of tracking the entire build environment, it is broken. As an intermediate stopgap then, clean does provide some uses even if there is some ideological opposition to it.

[rendaw]

Let me preface this with a couple of qualifications:

1. I've never tried it myself
2. I don't know if it works on any platform except linux There's an option "updater.full_deps" that does tracking outside of the tup root, I believe, which is I think the official ideal solution to this problem.

[FreddieChopin]

+1 for "clean" - easy way to see the warnings again!

[lu-zero]

+1 for clean as well

[goertzenator]

"if a build system requires a 'clean', it is broken."

I understand the ideology wrt building targets, but there are many uses for clean that have nothing to do with building. Seeing warnings again is one use, and for me I like to clean things up when I have a bunch of new files to commit to source control. And just now, I am browsing the tup source code and there are all these .o files cluttering up my directory listing.

People do so much more with code trees than just build them, and tup should be able to get all the build artifacts out of the way when humans need to do those other things.

+1 for clean

[gorakhargosh]

You can "clean" stuff without polluting tup. I strongly believe a build system should not require "clean". If you wish to force remove all generated files, you can use revision control commands to do that for you.

1. For git, I use the ".gitignore" rule within my Tupfile to automatically manage my .gitignore files for me
2. That combined with git clean -Xfd cleans all generated files.

HTH.

[lorenzogatti]

@gorakhargosh makes the right suggestion: reverting a directory tree to source files is a job for the source control system because it knows exactly what the result should be, allowing a perfect result in any situation. Asking a build system to undo its work is not only complex, but incomplete (no guarantee that other modifications didn't take place) and unreliable (no specification of the desired end state, only a summary of forward changes)

[goertzenator]

Thanks, I'll have to look at that git clean command. It may cause issues with the IDEs I use; sometimes they leave useful artifacts that are neither build or source related.

Today I discovered cleaning is a one-liner shell command:

sqlite3 .tup/db 'select name from node where type=4' | xargs echo rm -f

Tup has perfect knowledge of the build artifacts and could simply and reliably wipe them out. Are there technical complications here I'm not seeing? If it is really that simple (and maybe its not), why not add it?

[FreddieChopin]

As Rendaw wrote above: "In summary, I believe not having a clean command was intentionally omitted." Mike explained numerous times (from what I remember there's even an explanation in his paper) why a build system doesn't need such command, and thus tup doesn't have one (; For me it's not such a big problem, because I usually compile my stuff into a separate build subdirectory, so the job of cleaning that is pretty simple - just delete the folder. In other situations it is possible to generate the list of outputs into a text file, and clean that using shell script, executed from Makefile.

Do note, that technically make also doesn't have "clean" command. It doesn't have any commands - you write your scripts and so it happens that usually you have "all" and "clean", but it's you who describe exactly what and how they do.

[adzenith]

@goertzenator: that one-liner is what I came here to find. It's super nice to be able to have tup delete everything that tup has created without deleting all of the other .gitignored files generated by other things.

I agree that a build system is broken if it needs a clean command, if it needs that clean command in order to be able to build correctly. But this is more like a cleanup command—I want to delete the output artifacts not so that I can build again, but simply so that they're not there anymore.

[jonatanolofsson]

(old thread yada yada...)

The above one-liner assumes all files are in the root directory - the full solution is somewhat more involved.

I made the following addition to my Makefile

clean-all:
    sqlite3 .tup/db " \
        WITH RECURSIVE path(id, level, name, parent) AS ( \
            SELECT id, 0, name, dir FROM node WHERE type=4 \
            UNION ALL \
            SELECT path.id, path.level + 1, node.name, node.dir FROM node \
            JOIN path on node.id = path.parent), \
        path_from_root AS ( \
            SELECT id, name FROM path ORDER BY id, level DESC) \
        SELECT group_concat(name, '/') FROM path_from_root GROUP BY id;" | xargs rm -f

[AndydeCleyre]

I use tup to generate some files that are themselves tracked by git. I'm hitting two cases where the absence of a clean subcommand stumps me.

1. On a fresh pull of the repo and tup initialization, tup then refuses to overwrite any existing files. So I run tup, I see which file it complains about, I manually delete that file, I run tup, see which (next) file it complains about, manually delete that file, run tup, and so on...

2. After getting it all working, whenever I run tup generate, I have to do the same thing (1) over again.

I can't swear that clean is the best answer here, as I think it'd be just as good to overwrite the files rather than return an error. But I don't think that's currently possible either. I welcome any advice on how to re-approach this.

Aside: Is there a way to get tup to run its own generate subcommand whenever the Tupfile changes?

[AndydeCleyre]

Would it be worthwhile to open two more separate issues for the conflict scenarios of pre-existing committed output files, and running tup generate after using tup?

Or is this issue the best place to consider options?

[AndydeCleyre]

A command to list all output files would help to script our own clean operations.

[jonatanolofsson]

I actually encountered this as recent as today, just before your comment =) It's a limitation of my solution above that it cannot be run before the database is built, but the initial tup run panics and exits as soon as the first pre-existing output is found, so like you I've just deleted the files and regenerated them. I would love to see a better solution!