VS2012: Allow For the Ignoring of unpredictable and Unnecessary Outputs

ahicks92 commented 10 years ago

I just tried to move a simple project to Tup because, hey, why not and it looks great. unfortunately, cl.exe, Microsoft's command line C++ compiler, insists on writing a completely meaningless log file somewhere down in the appdata folder. This file is undocumented, cannot be disabled as far as I can tell, and includes the current user's username in the path (and probably varies depending on version of windows). More ironically yet, it contains 5 or 6 characters, one of which is unprintable. It has no bearing on the current build whatsoever, but stops tup from actually building. The file's name is sqmcpp.log, and no documentation as to its function exists anywhere. Can we add something that will allow me to say "This command's file output is suspect, ignore it because I know better?" As it stands, I cannot use Tup for my project and have had to revert to Cmake. Thank you Microsoft. If this exists and is undocumented, then this issue is for that instead.

gittup commented 10 years ago

On Fri, Apr 25, 2014 at 4:09 PM, Austin Hicks notifications@github.comwrote:

I just tried to move a simple project to Tup because, hey, why not and it looks great. unfortunately, cl.exe, Microsoft's command line C++ compiler, insists on writing a completely meaningless log file somewhere down in the appdata folder. This file is undocumented, cannot be disabled as far as I can tell, and includes the current user's username in the path (and probably varies depending on version of windows). More ironically yet, it contains 5 or 6 characters, one of which is unprintable. It has no bearing on the current build whatsoever, but stops tup from actually building. The file's name is sqmcpp.log, and no documentation as to its function exists anywhere. Can we add something that will allow me to say "This command's file output is suspect, ignore it because I know better?" As it stands, I cannot use Tup for my project and have had to revert to Cmake. Thank you Microsoft. If this exists and is undocumented, then this issue is for that instead.

Hi Austin,

Can you clarify what command-line you're trying to run that causes this file to be created? I just installed vs2012 and tried to compile a simple file using cl, but no such file was created.

There isn't currently the ability in a Tupfile to list a file that should be ignored, though we do ignore other files similar to this automatically. I think I'd prefer to add this to the list of automatically ignored files, rather than force the user to specify them (otherwise everyone who uses vs2012 would presumably have to have the same list in their Tupfiles).

Thanks, -Mike

ahicks92 commented 10 years ago

Actually, looking at it, it appears I was being an idiot when I wrote it-it's vs2013 professional, not 2012. My bad. I'll have to see if I still have the tup build file, but don't think so; I hadn't yet pushed the commits and, after an afternoon of trying, I'm pretty sure I locally reverted them. I'll have to hunt and see if they've been garbage collected yet for the exact command, but it wasn't anything super tricky. If I recall, not even optimizations. I think the issue is more general, however; I need to do other things in my build file for this project, most notably call a cmake project if it isn't built. This causes even more unpredictable outputs, or at least a rather large list of outputs I really don't care about. Really. Forever. I think that this option makes the tool more generally useful, because end-users can get around these kind of problems. I also think that it's my responsibility to tell you about my code's dependencies, not your tool's responsibility to say "You might want to, but can't"-but I think that is part of why Tup was written in the first place, so perhaps don't mind me. Note: I'm not 100% sure I can get Tup and CMake happily interfacing anyway, and it might just be better to give up and use CMake as I care about windows more than anything and consequently am including and building dependencies myself. I'm not sure I can give up CMake and recompile Portaudio and possibly a few other things with Tup, nor am I sure I want to as that makes tracking upstream harder. It still looks like a very nice tool just, now that I've started taking the project somewhere, perhaps not the right one for the job.

distee commented 10 years ago

Just started on using 'tup' and experienced the thing camlorn describes here. Feeding the following command line to the Visual Studio 2013 compiler:

cl /GS /TC /FC /EHsc /Fo"doc_try.o" ..\src\doc_try.c produces tup error: File 'C:\Users\dirk\AppData\Local\Microsoft\VSCommon\12.0\SQM\sqmcpp.log' was written to, but is not in .tup/db. ...

This makes the tup job fail.

gittup commented 10 years ago

I just installed vs2013 and tried to reproduce the issue, but I was unable to. It sounds like this may have been fixed in "Update 2" - is it possible that I got that since I just installed it? Can you try to upgrade VS and see if it resolves the issue?

Thanks, -Mike

On Fri, Jun 6, 2014 at 1:15 PM, Dirk Steenpass notifications@github.com wrote:

Just started on using 'tup' and experienced the thing camlorn describes here. Feeding the following command line to the Visual Studio 2013 compiler:

cl /GS /TC /FC /EHsc /Fo"doc_try.o" ..\src\doc_try.c produces tup error: File 'C:\Users\dirk\AppData\Local\Microsoft\VSCommon\12.0\SQM\sqmcpp.log' was written to, but is not in .tup/db. ...

This makes the tup job fail.

— Reply to this email directly or view it on GitHub https://github.com/gittup/tup/issues/182#issuecomment-45361500.

gittup commented 10 years ago

On Tue, Apr 29, 2014 at 3:08 PM, Austin Hicks notifications@github.com wrote:

I think the issue is more general, however; I need to do other things in my build file for this project, most notably call a cmake project if it isn't built. This causes even more unpredictable outputs, or at least a rather large list of outputs I really don't care about. Really. Forever. I think that this option makes the tool more generally useful, because end-users can get around these kind of problems. I also think that it's my responsibility to tell you about my code's dependencies, not your tool's responsibility to say "You might want to, but can't"-but I think that is part of why Tup was written in the first place, so perhaps don't mind me.

I don't think you'll have much luck trying to run one build system from inside another. If you need to build things with two build systems, you might be better off having a top-level build script that does something like:

cd cmake-proj; cmake cd tup-proj; tup

Note: I'm not 100% sure I can get Tup and CMake happily interfacing anyway, and it might just be better to give up and use CMake as I care about windows more than anything and consequently am including and building dependencies myself. I'm not sure I can give up CMake and recompile Portaudio and possibly a few other things with Tup, nor am I sure I want to as that makes tracking upstream harder. It still looks like a very nice tool just, now that I've started taking the project somewhere, perhaps not the right one for the job.

If you manage dependencies yourself, you're pretty much guaranteed to get them wrong at some point and waste your time. That's the second rule of build systems; I'm not compromising on it.

Thanks, -Mike

distee commented 10 years ago

I was already running on VS2013 Update 2, so it's likely unrelated to "prevent-new-local-user-accounts-to-login-to-windows" (I like that one :).

I looked around a bit and learned that SQM stands for Software Quality Metrics. This data is send to MS if you opt into the "Customer Experience Improvement Program", which I did. I opted out, but the data is still generated, it is just not send.

Right now, I have no idea how to turn writing off. Opting into the "Customer Experience Improvement Program" may trigger the writing for you ... or not.

I used procmon (process monitor, sysinternals) to follow the activity of cl.exe and discovered that link.exe also writes to 'sqmcpp.log'. Sadly, the event log did not convey any clue about the cause for this annoyance to me.

Maybe somebody else has an idea?

ahicks92 commented 10 years ago

I have moved on to cmake because I needed to make progress. That is not to say I couldn't move back, however, and I kind of want to simply because cmake is monolithic, archaic, and just painful. I've done the build script thing before for another project.

It is nice that Tup manages dependencies. Software may eventually become as smart as a human, but it isn't currently. Tup protects me, but if I know that I am safe and that the tool is wrong, it should be my option to tell it so. I have had to deal with dependency problems before and get where you're coming from, but being able to say "This command makes a bunch of extra output that I don't use" would fix this case and any cases in future, as well as allowing for stuff like building documentation (the output of doxygen comes to mind here). If we start implementing special rules for specific tools, it starts making the build system specific to those tools-instead, let me loosen the requirements as I need to. I may waste time when I get it wrong, but I did waste time figuring out that Tup wouldn't work on my setup because Tup insists on telling me that I'm wrong even when I know I'm not. This could be fixed in the specific case of visual studio, but I think that a more general fix is applicable; if I'm warned not to use it in the documentation unless I really need it, it's my fault for wasting my time and not the fault of the tool.

Anyhow, that's my two cents and tup isn't my project.

greneholt commented 10 years ago

I am very interested in using tup to manage the build process for extremely large Verilog ASIC designs at my company. Unfortunately, the compilers and toolchains involved in this process (mostly Cadence and Synopsys software) are very guilty of creating lots of extraneous or unpredictably named outputs. I understand that this is an extremely uncommon use case for build systems, and that probably no-one else using tup even touches these tools. As such, I would never expect support for ignoring the unnecessary outputs of these fringe programs to be built into tup, so the ability to manually configure what is ignored is critical.

I love the design of tup, especially the way it will refuse to run if you have implicit dependencies on generated files. That feature combined with the automatic dependency detection would save us so many headaches and bugs in our thousands of lines of makefiles. But the sad fact is that closed-source commercial tools can be stupid and/or do unpredictable things, and without the ability to manually compensate for these idiosyncrasies, many people will never be able to use tup.

gittup commented 10 years ago

On Sun, Jun 8, 2014 at 11:08 AM, Dirk Steenpass notifications@github.com wrote:

I was already running on VS2013 Update 2, so it's likely unrelated to "prevent-new-local-user-accounts-to-login-to-windows" (I like that one :).

I looked araound a bit and learned that SQM stands for Software Quality Metrics. This data is send to MS if you opt into the "Customer Experience Improvement Program", which I did. I tried opting out, but the data is still generated it is just not send.

Right now, I have no idea how to turn writing off. Opting into the "Customer Experience Improvement Program" may trigger the writing for you ... or not.

Ahh, that was indeed it. I had that turned off, and when I flipped it on I could reproduce it. I've just pushed a patch to ignore this particular file, since I think all normal compilers should work out of the box.

-Mike

gittup commented 10 years ago

On Mon, Jun 9, 2014 at 12:31 AM, Connor McKay notifications@github.com wrote:

I am very interested in using tup to manage the build process for extremely large Verilog ASIC designs at my company. Unfortunately, the compilers and toolchains involved in this process (mostly Cadence and Synopsys software) are very guilty of creating lots of extraneous or unpredictably named outputs. I understand that this is an extremely uncommon use case for build systems, and that probably no-one else using tup even touches these tools. As such, I would never expect support for ignoring the unnecessary outputs of these fringe programs to be built into tup, so the ability to manually configure what is ignored is critical.

I love the design of tup, especially the way it will refuse to run if you have implicit dependencies on generated files. That feature combined with the automatic dependency detection would save us so many headaches and bugs in our thousands of lines of makefiles. But the sad fact is that closed-source commercial tools can be stupid and/or do unpredictable things, and without the ability to manually compensate for these idiosyncrasies, many people will never be able to use tup.

Connor & Austin - thanks for your feedback. I agree that tup is not as usable in the cases that you describe as it should be. I think these are slightly different from the sqmcpp.log issue - for something common like that, I think it makes sense to ignore it in tup so that we can support standard compilers & tools by default. In other words, no-one would want to track writing to that file, and it's a little silly to require everyone to add it to an "ignore list" in their Tupfile.

As for the more general case that you are describing, where more rarely used or specialist tools do funky things, we definitely need a better way to support that, since obviously I can't keep track of them all inside the tup code itself. However, I'm not entirely sure that just ignoring the files outright is the way to go. Instead, we may want to consider allowing unspecified outputs as long as the command outputs to a group, though we'd need to figure out how to properly handle things like wildcards or overwriting ghost nodes, which are probably solvable but not entirely trivial.

My concern with adding an ignore feature is that rather than properly accounting for dependencies, we are allowing the user to compromise the integrity of the build. As such, users are much more likely to have to clean everything in order to get rid of the "ignored" files and return to an uncorrupted build state.

Thanks, -Mike

distee commented 10 years ago

Just fetched the latest build and run it: the issue is gone.

Thanks, Dirk

gittup commented 10 years ago

On Wed, Jun 11, 2014 at 10:24 AM, Dirk Steenpass notifications@github.com wrote:

Just fetched the latest build and run it: the issue is gone.

Thanks, Dirk

Cool! Thanks for digging into what SQM was - I wouldn't have figured that out on my own :)

-Mike

distee commented 10 years ago

Well, ..., figuring out something like tup is much more useful!

Kudos. Dirk

xwizard commented 9 years ago

Hi. tup error: File 'C:\Users\Mateusz\AppData\Local\Microsoft\VSCommon\14.0\SQM\VCToolsTelemetry.dat' was written to, but is not in .tup/db. You probably should specify it as an output

New version, new files :(

petervaro commented 8 years ago

@gittup actually @xwizard is right, for example on Windows 10, with the latest VS (2015) it is not working because of the above mentioned problem. (Even if "Customer Experience Improvement Program" is turned off! It looks like, you can't really do anything about this, unless you use something else than cl.exe) Gosh.. I'm so sick of supporting this fudging OS.. miserable :(

xwizard commented 8 years ago

@petervaro Tup works fine with Mingw64 on Windows 10. It's cl.exe's fault in this case.

petervaro commented 8 years ago

@xwizard I know, and I already switched to mingw-w64 as my default windows-compiler, I just wanted to make sure my code-base will also compile via cl.exe as well -- and I know, it's definitely not tup's fault, never mentioned it though :P tup's only fault is the absence of the run directive on windows.. but it's another story, isn't it? :)

cfillion commented 8 years ago

For anyone having the same issue @xwizard had with VS2015's cl.exe writing to VCToolsTelemetry.dat: There's a way to disable this unwanted data collection with a registry edit: https://msdn.microsoft.com/en-us/library/ee225238.aspx#Anchor_5 On my computer the key was in HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VSCommon\14.0\SQM.

mattiasflodin commented 8 years ago

I just ran into this issue too, with the VCToolsTelemetry.dat file. In my humble opinion, hard-coding specific file names is the wrong way to go about this because it will lead to new problems with every new version and compiler brand. It's annoying but possibly solvable when the developer hits it, but what if my users who are already struggling to build my software run into this? What if they use some antivirus or similar that hooks into processes and happen to write to some other file that has nothing to do with the build? It's just too much of a usability problem to be worth it. Besides, from a design perspective tup should be compiler agnostic; hardcoding behavior for specific compiler brands and versions is an ugly hack.

I would suggest that anything that is written outside the source tree is ignored unless a warning is explicitly enabled.

ahicks92 commented 8 years ago

I've long since left Tup, but am still subscribed to the thread. I doubt I'll come back as I now have a huge amount of CMake and Python automation (maybe 500-1000 lines and growing) for stuff like automated releasing to Pypi and handling continuous integration and finding libraries. I'm not saying this isn't a good tool, just that I would now have to spend very significant effort to get back to it. But it doesn't matter if I want to, as Tup cannot handle my case.

The scenario in my original comment is no longer hypothetical. I now have code that calls python setup.py bdist_wheel as part of the build process for a primarily C++ project. It also calls sphinx to generate docs. These two tools create hundreds of files at least in the build directory, and I don't care about 99% of them.

Saying "Ignore everything outside the build directory" fixes VC++, but it doesn't fix the actual problem. This is like the unsafe keyword in Rust. Our tools are not smarter than us. It is essential that I am able to tell my tools that I know better than them on issues like this. Otherwise you just keep running up against these cases.

mattiasflodin commented 8 years ago

Agreed, but I think both solutions can and should coexist.

ahicks92 commented 8 years ago

yes, both is certainly workable. My point is merely that "ignore everything outside the build directory" is fixing a set of symptoms, not an underlying problem.

mattiasflodin commented 8 years ago

Well, I see it more as implementing what you suggested but with a convenient shortcut for a common use case.

Edit: And, in the short run, it will work as an initial stopgap for a problem that everyone who uses Visual C++ will run into.

xmclark commented 6 years ago

I'm trying tup with cl.exe, visual studio 2017 on my windows 10 x64 machine. I've tried setting the "OptIn" flag in various registry string keys, but I still get the error:

tup error: File 'C:\Users\Me\AppData\Local\Microsoft\VSCommon\15.0\SQM\VCToolsTelemetry.dat' was written to, but is not in .tup/db.

I've tried setting the value "OptIn" string value in:

\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Policies\Microsoft\VSCommon\15.0\SQM
\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Policies\Microsoft\VSCommon\14.0\SQM
\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Policies\Microsoft\VisualStudio\SQM
\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Policies\Microsoft\SQMClient
\HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\VSCommon\15.0\SQM
\HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\VSCommon\14.0\SQM
\HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\VisualStudio\SQM
\HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\SQMClient

No success. I'm hoping somebody else has a work around for this. This is a pretty big blocker for me.

kalrish commented 6 years ago

Try with \HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VSCommon\14.0\SQM or \HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VSCommon\15.0\SQM.

xmclark commented 6 years ago

\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VSCommon\15.0\SQM worked! Thanks @kalrish.

datgame commented 6 years ago

Updated Visual Studio 2017 yesterday, and now this problem is back. The setting for feedback in the GUI is off, and the registry key above is set to 0. Seems like they don't listen to either anymore. \HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VSCommon\15.0\SQM is what exists here. I don't have the \HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\VSCommon\15.0\SQM entry in my registry, or any of the other examples above. One question is if any of them should be created, by making a new folder and key called "OptIn". (rebooted after modified to verify) Setting works in VS 15.5.6 but not in 15.6.0 and 15.6.1.

datgame commented 6 years ago

I have sent a report via their feedback button in MSVS now, with a link to this thread. Asking for them to fix this regression or provide an alternative.

Vote it up: https://developercommunity.visualstudio.com/content/problem/210627/vctoolstelemetrydat-is-being-written-to-when-not-n.html

datgame commented 6 years ago

I downgraded to 15.5.7 and now it's working again: https://docs.microsoft.com/en-us/visualstudio/productinfo/installing-an-earlier-release-of-vs2017 I have no plans to upgrade for a few years now, until really required. Will try to create an offline installer too, just in case, since old installers are removed after a while from the site.

manbearian commented 6 years ago

Hi, @datgame i'm replying here and I also replied to the dev-community issue you opened. First, thank you for brining this too my attention and second, i'm sorry for the inconvenience this change has caused you.

I have a fix in 15.7 for this issue which will be in 15.7 preview 3 when it's released imminently.

You can immediately unblock usage of 15.6 via this workaround: set VCToolsTelemetry.dat to be read-only so that the VC++ compiler and friends cannot write to it.

Below is a simple CMD script that accomplishes this; feel free to cut-and-paste it into a CMD shell on your machine.

:: first kill any process that might be accessing or will access VCToolsTelemetry.dat
taskkill /f /t /IM vctip.exe /IM cl.exe /IM link.exe /IM msbuild.exe

:: second, create a new, empty VCToolsTelemetry.dat and lock it
set file=%USERPROFILE%\AppData\Local\Microsoft\VisualStudio\15.0\VCToolsTelemetry.dat
echo.>%file%
attrib +r %file%

Please let me know if you have any more issues or have any questions or concerns regarding this.

ian Bearman Principal Software Engineering Manager Microsoft Compiler and Languages Platform Group
/ Making your code faster, smaller, smarter! /

gittup commented 6 years ago

There is some preliminary support for ignoring files in v0.7.8. You can specify files to be ignored in the output section by prefixing it with a '^' character. Eg:

: |> cmd... |> actualoutput.o ^ignoredoutput.txt

The "ignoredoutput.txt" here is a regular expression used to match against filenames picked up during dependency detection, but just note that the initial carat is not part of the regular expression. The path matching is done against a canonicalized path, so I believe it is always done against a fullpath version of the file access. If you need to match against a path separator, always use '/', even on Windows.

Let me know if this helps with any of these issues, and be mindful that it can be easy to shoot yourself in the foot with it by ignoring too much if your pattern is too general.

DerelictHelmsman commented 5 years ago

Microsoft strikes again, in the latest build of Visual Studio 2019 they generate a different filename per compile in the temp folder: tup error: File 'C:/Users/ADMINI~1/AppData/Local/Temp/lnk{532BAC1E-C44B-4420-BED4-667060A9A2BB}.tmp' was written to, but is not in .tup/db. You probably should specify it as an output -- Delete: C:/Users/ADMINI~1/AppData/Local/Temp/lnk{532BAC1E-C44B-4420-BED4-667060A9A2BB}.tmp

datgame commented 5 years ago

Microsoft strikes again, in the latest build of Visual Studio 2019 they generate a different filename per compile in the temp folder:

report it to the microsoft site i linked above. we should get a microsoft developer to monitor this github thread, so they can learn more about their own code.

mattiasflodin commented 4 years ago

I'd say Microsoft is well in their right to produce any temp files they want during compilation. It is tup that is making unwarranted assumptions about the behavior of a compiler.

alaestor commented 2 years ago

It would be nice if it could be done the same way wildcard inputs are.

Take the following hypothetical rule syntax, similar to "order only" inputs

: something |> log_generating_program %f -o %o |> output.file | log/* dependency based the output.file but tup would ignore everything generated in the log/ directory

: something |> a thing that must run and exit successfully |> {empty_bin_or_group_for_ordering} | * wildcard ignores all generated outputs, with no expected outputs named: it only cares about whether or not the the program exited successfully.

In my opinion, something like this would make tup scale better, be more user-friendly, and make it easier to integrate with complex environments.

Currently, the work around I've been using is a custom program which acts as a buffer between tup and executing shell commands, combined with empty bins to direct rule ordering. This way, tup tells the program the command and the program executes it outside of tup's purview, only reporting back either success or failure. I've had good results with this work-around, even using tup to integrate with and compile cmake git submodules. But it's less than ideal and adds another custom tool as a project dependency. An enhancement to this work-around would be to tell the program the expected results and it would touch them so they could be tracked by tup rather than using magic empty bins or groups; but I haven't gotten around to that yet.

Anyways; with python cache, C++20 modules, log files, atypical and complex tool-dependent workflows.... This feature is becoming ever more important. I love the low-level power and flexibility of tup; I've even been using tup for lite-weight unit testing in some projects. But this has been a stubborn blemish. I understand not wanting to give the user a gun to shoot themselves in the foot with, but having the option would be nice.

gittup commented 2 years ago

It would be nice if it could be done the same way wildcard inputs are.

Take the following hypothetical rule syntax, similar to "order only" inputs

: inputs | order-only inputs |> cmd |> thing i want | things i dont care about everything after the output | would be ignored by the DAG as an unimportant side-effect.

Currently tup uses this syntax for "extra outputs", which are files created by the program that you don't want included in the %o flag. Sometimes programs write to multiple files, but only specify one on the command-line. The other files that are expected to be created would go after the | in the output section, but aren't passed into the command.

: something |> log_generating_program %f -o %o |> output.file | log/* dependency based the output.file but tup would ignore everything generated in the log/ directory

This should work today with the ^ syntax in the output section (see the man page under TUPFILES, :-rule, outputs). It would look something like this:

: something |> log_generating_program %f -o %o |> output.file ^log/

(The ^ flag denotes a regular expression, so the * can be dropped. All dependency tracking on files that match the pattern is skipped).

Somewhat confusingly, the ^ character has a different behavior in the input section where it is used to skip files in glob matching.

: something |> a thing that must run and exit successfully |> {empty_bin_or_group_for_ordering} | * wildcard ignores all generated outputs, with no expected outputs named: it only cares about whether or not the the program exited successfully.

This is an interesting idea, and something I've thought about before. I think it would be possible with <groups> (since {bins} are just temporary lists during Tupfile parsing and discarded after), but there are a lot of tricky cases that can pop up with ghost files and such that would need to be handled correctly. If it could be made to work, I agree it should make some build structures easier to create. I don't know if that would help with all the use-cases you mention, though. There are definitely some build system anti-patterns that just aren't going to scale well in any build system.

AndydeCleyre commented 10 months ago

I'm not sure if I'm using the ignore syntax correctly.

I'm starting to try tup for generated files from templates but am having two difficulties.

One is that I have a bunch of commands that require (and generate if needed) a Python venv, and I'd like for separate rules to all be able to use the same .venv folder, rather than cleaning it up and recreating it for each rule.

tup v0.7.11-120-g8739d024
A rule like: : requirements.in |> mk/reqs.sh |> %B.txt ^\.venv

* 0) mk/reqs.sh
mk/reqs.sh: 15: .: cannot open ./.venv/bin/activate: No such file
 *** tup messages ***
 *** Command ID=3488 failed with return value 2
 [ ] 100%
 *** tup: 1 job failed.

Of if I manually delete .venv and try again:

 [ ]   0%/home/andy/.../.venv/lib/python3.10/site-packages/pip/_internal: No such file or directory
* 0) mk/reqs.sh
...
 *** tup messages ***
tup error: Unable to create directory '/home/andy/.../.venv/lib/python3.10/site-packages/pip/_internal'
 [ ] 100%
 *** tup: 1 job failed.

gittup / tup

VS2012: Allow For the Ignoring of unpredictable and Unnecessary Outputs #182