Specify directories as sources and targets

[arosenbe]

It can simplify dependency tracking if SCons allows directories to be tracked as sources and targets, not just files within directories. In this issue, I'll explore how feasible a setup like this is.

Followup task to the lunch discussion with @gentzkow on January 24, 2017.

[arosenbe]

To start, I check if an empty directory created before the first scons run can be specified as a source or target by following these steps.

1. Make a new empty directory in source/paper.
2. Specify that directory as a source in a build command in source/paper/SConscript.
3. Run scons

Click for the error

``` (master) template$ scons scons: Reading SConscript files ... TypeError: Directory /Users/arosenbe/Desktop/template/source/paper/test found where file expected.: File "/Users/arosenbe/Desktop/template/SConstruct", line 23: SConscript('source/paper/SConscript') File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Script/SConscript.py", line 604: return method(*args, **kw) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Script/SConscript.py", line 541: return _SConscript(self.fs, *files, **subst_kw) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Script/SConscript.py", line 250: exec _file_ in call_stack[-1].globals File "/Users/arosenbe/Desktop/template/source/paper/SConscript", line 7: env.Lyx(target, source) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Environment.py", line 260: return MethodWrapper.__call__(self, target, source, *args, **kw) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Environment.py", line 224: return self.method(*nargs, **kwargs) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Builder.py", line 626: return self._execute(env, target, source, OverrideWarner(kw), ekw) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Builder.py", line 547: tlist, slist = self._create_nodes(env, target, source) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Builder.py", line 477: slist = env.arg2nodes(source, source_factory) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Environment.py", line 483: v = node_factory(self.subst(v, **kw)) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Node/FS.py", line 1404: return self._lookup(name, directory, File, create) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Node/FS.py", line 1383: return root._lookup_abs(p, fsclass, create) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Node/FS.py", line 2395: result.diskcheck_match() File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Node/FS.py", line 2593: "Directory %s found where file expected.") File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Node/FS.py", line 398: return self.func(*args, **kw) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Node/FS.py", line 419: raise TypeError(errorfmt % node.get_abspath()) ```

I repeat but specify the directory as a target in step 2. SCons throws a nearly identical error.

Click for error

``` (master) template$ scons scons: Reading SConscript files ... TypeError: Directory /Users/arosenbe/Desktop/template/source/paper/test found where file expected.: File "/Users/arosenbe/Desktop/template/SConstruct", line 23: SConscript('source/paper/SConscript') File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Script/SConscript.py", line 604: return method(*args, **kw) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Script/SConscript.py", line 541: return _SConscript(self.fs, *files, **subst_kw) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Script/SConscript.py", line 250: exec _file_ in call_stack[-1].globals File "/Users/arosenbe/Desktop/template/source/paper/SConscript", line 6: env.Lyx(target, source) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Environment.py", line 260: return MethodWrapper.__call__(self, target, source, *args, **kw) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Environment.py", line 224: return self.method(*nargs, **kwargs) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Builder.py", line 626: return self._execute(env, target, source, OverrideWarner(kw), ekw) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Builder.py", line 547: tlist, slist = self._create_nodes(env, target, source) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Builder.py", line 494: tlist = env.arg2nodes(target, target_factory, target=target, source=source) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Environment.py", line 483: v = node_factory(self.subst(v, **kw)) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Node/FS.py", line 1404: return self._lookup(name, directory, File, create) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Node/FS.py", line 1383: return root._lookup_abs(p, fsclass, create) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Node/FS.py", line 2395: result.diskcheck_match() File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Node/FS.py", line 2593: "Directory %s found where file expected.") File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Node/FS.py", line 398: return self.func(*args, **kw) File "/usr/local/Cellar/scons/2.5.1/libexec/scons-local/SCons/Node/FS.py", line 419: raise TypeError(errorfmt % node.get_abspath()) ```

I got the same errors after putting an empty file in source/paper/test and rerunning scons.

[arosenbe]

I'll pass this off to you @yuchuan2016. It will be good to check how scons handles full directories as sources and targets. Also feel free to check anything else you can think of!

[arosenbe]

@gentzkow, I have a quick follow-up (two comments above) to our conversation at lunch today about specifying directories as sources and targets in the SCons framework. It seems that SCons' default position is that directories should not be sources or targets. I didn't go through the documents or code to try to figure out why this is the case or to look for some possible workaround. We can invest the time if you think it's something valuable to explore further.

[yuchuan2016]

@gentzkow , a simple test uses the following structure:

source/data/build.R generates 10 files data*.txt and stored in build/data/ source/analysis/analysis.do uses 10 files in build/data/

If we change:

SConscript in source/data/:

• target = ['#build/data/'] -> error as above
• target = ['#build/data/*'] or target = ['#build/data/*.txt'] or target = ['#build/data/data*.txt'] -> rebuild 10 files in each run

SConscript in source/analysis/:

• source = ['#source/analysis/analysis.do', '#build/data/'] -> error as above
• source = ['#source/analysis/analysis.do', '#build/data/*'] or source = ['#source/analysis/analysis.do', '#build/data/*.txt']or source = ['#source/analysis/analysis.do', '#build/data/data*.txt'] -> Error

  scons: ** [build/analysis/plot.eps] Source `build/data/' not found, needed by target build/analysis/plot.eps'.

Let me know if you think @arosenbe 's suggestion to go through the documents or code is worth doing.

[gentzkow]

Thanks guys. I think this is enough investigation on this for now. We can revisit when we wrap up other template tasks and take an overall look at the system.

[yuchuan2016]

Summary: Currently we have not found a good way to specify directories as sources or targets in SCons.