request: truffle dump out commands given to solc

[fxfactorial]

I'm trying to use abigen against truffle compiled contracts but had some trouble because solc doesn't know about JS style imports - so would have been much easier if could see the module remappings that truffle does in its invocation of solc

would be great if truffle could accept some env var that allows dumping out to screen the commands it passes to solc - I looked in source and currently it doesn't do that, or have a hook for that.

[eggplantzzz]

Hey @fxfactorial! Are you interested in just getting the raw input that Truffle gives to the Solidity compiler? If so, Truffle doesn't currently expose that information. Something that you may find helpful is the metadata field in the built artifacts (the JSON files). This information is complete in the sense that you can accurately re-create the compilation. You can check out the docs from Solidity here. Hopefully this helps!

[fxfactorial]

@eggplantzzz ya exactly - the raw command truffle calls to solc - I see in the source that it doesn't expose that, so I'm asking if could have something like

if (process.env.TRUFFLE_DUMP_SOLC_CMD) {

}

right before the compliation

[haltman-at]

Note that Truffle generally doesn't invoke solc by way of a command, unless you're using version: "native" in your Truffle config. So the question then becomes whether you would want just the JSON input, or whether you would want it converted to the form of a command. I imagine the former makes more sense, as it's much more generally usable than the latter (e.g., for verifying on Etherscan). The latter seems kind of abstraction-breaking.

[cameel]

Another use case: having easy access to the JSON input would be very helpful when debugging compiler crashes (like in #2994 or #3419). Admittedly, that's not typical usage for contract developers but currently the only way I know to get this info when someone reports a bug is to download Truffle source and modify it (by inserting console.log(inputString) in invokeCompiler() in packages/compile-solidity/run.js). It would be much better to be able to get that info out of a built version downloaded via npm or system package manager.

I can get it from the metadata but that's obviously not available if the contract fails to compile due to an internal compiler error. Also, the JSON stored in the metadata is not the exact copy of the input. It has been read, analyzed and then serialized again by the compiler from its internal data structure that stores settings - the difference can sometimes matter when trying to analyze buggy behavior.

This info is also directly usable with solc --standard-json - can be supplied on standard input or stored in a file if you specify the name. Currently this does not always map easily to the standard use with command-line flags but this is likely to change in the future (https://github.com/ethereum/solidity/issues/9583).

[gnidan]

Hm, some questions:

1. Is this part of truffle compile or is it a separate command? a. If it's part of truffle compile, what's a good name for a CLI flag? truffle compile --save-compiler-inputs=<filename>? b. If it's part of truffle compile, should it print this info in addition to doing the normal compilation, or instead of doing that? c. If it's a separate command, what do we call it?
2. Since truffle compile handles more than just solc, how does this work for Vyper, etc.? a. If we don't want to support non-solc compilers, can we restrict this to just solc? So truffle compile --compiler=solc --save-compiler-inputs=<filename>? This might get tricky in implementation. b. If we support this for all compilers, what's the aggregate result data look like? We'd need to figure that out c. What do we do about @truffle/external-compile? (see external compiler docs)

Anyway, I'm into this. I'll move this to our backlog with a low priority, but I'll mark it help wanted if anyone wants to open a PR for this, once we get these requirements nailed down.

This would require changes in a few places:

• packages/core/lib/commands/compile to add the CLI flag and hook that up (unless we make it a totally separate command)
• packages/workflow-compile to accept the new option and forward it to the relevant compiler integrations
• packages/compile-solidity for solc
• packages/compile-vyper for vyper
• packages/external-compile for external compilers (this might not need to change much, not sure)
• packages/compile-common, since that defines the common interface across all the compiler packages

If anyone's interested in pursuing this, let us know! Happy to work with you to figure out the unknowns and make this happen. Thanks!

[cameel]

Hm, some questions:

1. Is this part of truffle compile or is it a separate command?

I see it just as an option to enable more verbose output, possibly usable also with truffle test and maybe other commands if they invoke the compiler, so a CLI flag sounds like the best fit.

a. If it's part of truffle compile, what's a good name for a CLI flag? truffle compile --save-compiler-inputs=<filename>?

How about --show-compiler-input? --show-invocations? Just --verbose might be good too - this does not really have to be limited to showing just the compiler input, though that's the part that's useful to me at the moment.

Being able to save the input to a file sounds convenient but only if it's one file per invocation. If it's just a big dump of all invocations, then I can't feed that directly to solc and I think it's better to just print it to stdout so that I can copy it and paste into a file.

b. If it's part of truffle compile, should it print this info in addition to doing the normal compilation, or instead of doing that?

I'd say that printing it to stdout along with everything else would be fine for me. It would just be helpful if that output was printed prettified (i.e. with indentation) to make it readable.

2. Since truffle compile handles more than just solc, how does this work for Vyper, etc.?

I'm not sure how Truffle communicates with Vyper but I'd say that I would just expect it to tell me that it's invoking Vyper and show me whatever information it passes to it. Might be JSON, TOML, CLI commands, a dump of RPC commands if it's running like a language server or any other info that it receives.

a. If we don't want to support non-solc compilers, can we restrict this to just solc? So truffle compile --compiler=solc --save-compiler-inputs=<filename>? This might get tricky in implementation.

Personally I'd be fine even with it working only for Solidity but I don't think it would be hard to add logging for other compilers too as long as you treat it just as log output and not something that has to adhere to the same format in all cases.

b. If we support this for all compilers, what's the aggregate result data look like? We'd need to figure that out

Just a free-form log of all the stuff that was invoked.

c. What do we do about @truffle/external-compile? (see external compiler docs)

It actually sounds like this feature would be helpful when trying to hook up a custom compilation script. While doing stuff like that for the first time there's usually some trial and error involved to figure out how exactly to specify it and what exactly the command is getting as arguments.

Does Truffle execute it as a process (or a shell command)? Then just seeing the command, its arguments and the working directory would be the information I'd want to see.

Anyway, I'm into this. I'll move this to our backlog with a low priority, but I'll mark it help wanted if anyone wants to open a PR for this, once we get these requirements nailed down.

Thanks a lot! I hope the answers above help.

[gnidan]

I see it just as an option to enable more verbose output, possibly usable also with truffle test and maybe other commands if they invoke the compiler, so a CLI flag sounds like the best fit.

I concur with this.

I'd say that printing it to stdout along with everything else would be fine for me. It would just be helpful if that output was printed prettified (i.e. with indentation) to make it readable.

The problem with stdout is that it makes it harder to extract programmatically, since there's other stdout that might get in the way. We faced this problem recently and decided to go with a similar solution with the (undocumented) truffle compile --save-intermediate flag.

Being able to save the input to a file sounds convenient but only if it's one file per invocation. If it's just a big dump of all invocations, then I can't feed that directly to solc and I think it's better to just print it to stdout so that I can copy it and paste into a file.

I'm imagining that we'd save a JSON file that'd be organized by compilation - so you could go into that file and grab whatever compilation you need specifically.

Alternatively, we could create a directory and store everything there, but I think that's a bit uglier.

Personally I'd be fine even with it working only for Solidity but I don't think it would be hard to add logging for other compilers too as long as you treat it just as log output and not something that has to adhere to the same format in all cases.

The implementation concern I have with this approach is that, right now, there's no way to define CLI options that only apply to one compiler. We could add that, and then have (e.g.) truffle compile --solc-save-standard-json, but I think that'd be more work than defining something in common for everything.

Does Truffle execute it as a process (or a shell command)? Then just seeing the command, its arguments and the working directory would be the information I'd want to see.

Yep, it executes as a child process in the shell. So yeah, seems pretty easy.

The next tricky thing, I think, will be to create unified type that works across compilers. Note that Truffle currently invokes solc in a number of ways (and similarly for Vyper): we either do docker run or we call a native solc binary through the shell, or we invoke a wrapped soljson via JS. Would we use a type union to represent the different possibilities here?

Seems like the first step would be to start spec'ing out what the TS type would look like.

Thanks a lot! I hope the answers above help.

Definitely! Thank you!

[cameel]

I'm suggesting just printing to stdout because my and @fxfactorial's use cases do not really require it to be easy to process programmatically and I'm not sure if there's really a good use case for that. If it needs to be processed like that then some way to hook into the compilation process in JS would be more a general solution anyway. Trying to solve it here might mean dealing with problems that don't really need to be solved :)

I'd do it in the simplest possible way now and refactor into a more robust feature once a real use case appears. The nice thing about this being just optional, free-form log output meant for human consumption is that the backwards-compatibility is not really a concern. So there's no need to make it perfect from the start, it can be minimal and improved when the need comes. Maybe I'd add a randomly generated content boundary (like in MIME multipart messages) to make it easy to cut that part out programmatically but that's basically it.

[eggplantzzz]

@gnidan This is a perfect task for the event system if we go ahead and implement something for this :)