A library+tool to get all source files for a crate

[jsgf]

It would he useful to have a library which can produce a list of all source files used to build a crate. Currently this can be done with rustc --emit dep-info, but that requires a full build, including all dependencies to have already been built. I'm thinking of a tool which can be applied in isolation to any top-level source file, much like rustfmt can be.

Given a top-level module source file, it should find:

• the top level module (given)
• all other module sources
• all included files (whether they're Rust source or not)

It should also have the option to take a set of cfg options so that the list of files can be limited to conditionally compiled code. But it should also have an option to emit all files, regardless of conditions.

For more general use, each file should also have associated with it whether its 1) a Rust module, 2) included Rust code, 3) included binary/text.

I can think of two awkward cases:

1. a procedural macro generates an include! variant or a mod name; reference which would be missed without running the procmacro
2. a path isn't a string literal, but composed by concatenation with env vars or other macros

I'm OK with leaving those out for now (the latter could be a best-effort representation, or just a warning that there's an unrepresentable name).

This would be useful as both a library and a CLI tool wrapping the library. The CLI tool should have the option to emit plaintext and json for easy scripting integration.

[basil-cow]

I'll soon push first version of this tool to github (any thoughts on a name?), maybe you know an include and/or #[path]-heavy library to test on?

[dtolnay]

name

srcfiles would be straightforward and simple.

include and/or #[path]-heavy library

https://github.com/dtolnay/syn uses both include! and #[path]. https://github.com/dtolnay/watt is another tricky one involving #[path].

[jsgf]

Hi! Thanks for working on this!

I don't have any particular target crates in mind, and I'm fine with handling the 90% case at first and iterating thereafter (ie, if it can completely identify sources for 90% of crates, and needs some handholding for the rest, then that's a great start).

[basil-cow]

I pushed first version at https://github.com/Areredify/srcfiles. I didn't test it thoroughly (I'll add tests ASAP), so it's probably not worth filing bug issues yet. Right now it should work on code that uses mod a; and #[path] only (no #[cfg_attr] and stuff).

[jsgf]

It also seems to have trouble with cfg_if! selection - I'm seeing libc missing some sources for example.

[basil-cow]

@jsgf Would you mind describing your usecase and/or possible usecases so I can write up a README? I'll open cfg_if! issue and I suggest to do all crate-related things in that repo's issues. Also please comment there on what kind of info you would like to see in json and/or plaintext output. I also added some tests so now feel free to file bug issues.

[jsgf]

I'm writing a tool to generate Buck build rules from Cargo.toml. Part of that is getting a complete list of all the sources from the crate root that the Cargo metadata exposes. Right now it is all based on globs, but with srcfiles I can generate a much more precise source file list. I'm using it as a library only right now.

Once comment on the crate_srcfiles API - returning the tuple of success/failure isn't very useful for me. If it returns failure then I don't really care about the successes. A Result<Vec<SourceFileDesc>, Error> would be more useful (with the Error type perhaps containing success/failure lists for things that care). (This isn't a huge point, but it makes the API feel a bit odd.)

It's also worth noting that this only supports Rust 2018 (because that's all syn supports). My tool tries srcfiles first, and if it fails for any reason it falls back to globs.

I'm currently running it over a few hundred crates from crates.io, but I haven't systematically looked at how accurate it has been (ie, are the generated rules buildable, or do they fail from missing sources?).

Another style note: it would be good if the naming consistently either used "source" or "src" (eg SourceFileDesc vs SrcError).

[dtolnay]

I've closed this out with a link to https://github.com/Areredify/srcfiles. Thanks all! We can follow up with any remaining feature requests in the issue tracker there.

In particular, @Areredify it would be good to get the tool published to crates.io at this point (https://github.com/Areredify/srcfiles/issues/6).