Closed vMeph closed 6 years ago
Summary of the issue:
The cause of the compilation failure, as found by @bobc in the mentioned issue report, is that the large number of filenames in {object_files}
when compiling the Marlin sketch causes the command to exceed the maximum command length of 32767 characters on Windows.
The Marlin sketch folder structure was recently reorganized to put most of the source files in subfolders and this caused the command length to significantly increase. I believe this version is found in the bugfix-2.0.x-new-layout branch.
When I compile the tip of that branch using the hourly build of the Arduino IDE, Windows 7, for Arduino/Genuino Mega I still only end up with a recipe.c.combine
command length of 21370 so it may be that a non-default configuration of Marlin leads to a longer command.
I believe that, on Windows, the total length of the command (not the number of arguments, like on Linux) is relevant. This means that the length of the path to the sketchbook might be relevant, but more importantly the length of the path to the build / temp folder. I think this might normally be in the "user" directory, so that would suggest the username length is relevant too.
As a workaround, you could probably set build.path
to "C:\build" or something similarly short in preferences.txt
to do all builds in that same short directory.
Setting build.path
appears to result in all sketches compiling to same folder (tested with 1.8.3), is there a way to build to different folders for each sketch, e.g. build.path=c:\temp\{sketch}
or something?
But should the user be required to manually set a temp directory in order to build a sketch ? Am I the only one who sees this as a flaw ?
Setting build.path appears to result in all sketches compiling to same folder (tested with 1.8.3), is there a way to build to different folders for each sketch, e.g. build.path=c:\temp{sketch} or something?
Nope, it either just uses that path if it is set, or generates a new path for each sketch when it is empty.
But should the user be required to manually set a temp directory in order to build a sketch ? Am I the only one who sees this as a flaw ?
It's certainly not a proper fix, but it is useful to mention the workaround, to at least get people unstuck.
The actual problem here is the limited length of a commandline, combined with a sketch (and library) combined with a lot of files. I can't see a perfect solution for this problem, unfortunately. I do have some ideas.
You could link together groups of .o files, and then link the resulting groups together into the final binary. A common approach for this would be to use archives (.a files), which contain multiple .o files. This already happens for the core, and libraries that have enabled this in their libraries.properties file.
It would be obvious to create an archive for the core, sketch and each library separately. In practice this would probably work, but in theory, this could still break when a single library or sketch contains enough files to trip the limit. To really solve this, the final linker command should be automatically split, based on the actual commandline length.
Another downside of using .a files is that it slightly changes the linking strategy: .o files specified on the command file will be included in the link, while .o files inside .a files are only included when they are needed. Due to subsequent optimization steps, the difference between these is minimal, but it might break some corner cases.
An alternative would be to merge .o files into a new .o file. Apparently the -r option can be used for this, though I have not tried this.
One caveat is that splitting the linker command will very likely need additions to the recipes in platform.txt
, since those only specify how to perform linking with a single linker step. This makes it harder to implement a fix for this.
One option that would be perfect is if you could specify the list of files to compile through stdin or through a tempory file instead of on the commandline. Looking at the gcc commandline, it supports specifying @file
on the commandline, which will read commandline options and arguments from file
. Looking at the linker recipe, this could be implemented by simply generating a tempfile containing .o filenames and filling {object_files}
with @/path/to/tempfile
. The downside is that targets that somehow do not support the @file
syntax would break. An alternative would be to create a second variable to be used in platform.txt
, {object_list_file}
or something like that, making this an opt-in for cores (but that does need manual action for all cores, and breaks compatibility with older Arduino versions).
One point of attention is that the contents of the @file
needs proper escaping of whitespace, which again adds complexity. Fortunately the escaping is fairly simple, without the full complexity of shell escaping.
Summarizing: There are solutions, but nothing that is so simple it could be very quickly implemented. The @file
approach seems promising, but needs more thought, especially wrt compatibility.
Nope, it either just uses that path if it is set,
Exactly, it uses the same path for all sketches you might compile IF it is set. There is no per-sketch setting.
Using @file seems the simplest and most scalable fix. Clearly this needs to be an option, if there are targets that don't use gcc. If I knew Go, I would propose a patch.
I suspect though, anyone who encounters this issue has outgrown the Arduino IDE, which is only really usable for small sketches, and should be using a more professional tool.
There could be a third option:
buildpath
cmd.Dir
with buildpath
This should shorten the commandline A LOT, without any drawback (if the split doesn't recognize a filename, using the absolute path makes no harm)
A test commit is here (https://github.com/arduino/arduino-builder/compare/master...facchinm:relative_paths?expand=1)
I'm also attaching a Windows build of the builder, if anyone at Marlin still has a copy of the problematic branch it would be nice to test it live :smile: arduino-builder.zip
@facchinm - dropping in your arduino-builder debug build with your Windows command-line length work-workaround and previously-failing build now succeeds :+1:
However, it's unclear how because the output seems to still include the full paths (avr-gcc link command truncated at 32k in the build window):
Hi @fiveangle , thanks for testing! I kept the output "as-is" to double check the fix; it gets printed here but the actual commandline is being modified in an internal routine.
Also, I don't know if it's a good idea to print the mangled commandline since you need to cd
into the build folder to launch it correctly; the other command, on the other hand, would work from anywhere in the filesystem.
@facchinm, sounds like a practical approach, though it won't really solve the problem (just raise the bar for making the problem to occur).
What command to show is indeed an issue: I would argue that showing a different command from what's actually being run is problematic when debugging some problems. The instant-copy-pasteability of full paths is an advantage, though, so I'm not sure what side to pick (probably show the actual command being run, especially if this happens only for long commandlines).
One caveat is that there might be targets that do not like relative paths (on some of its commandline options perhaps). I don't have any examples ready, though. OTOH, if this fix is applied only for too-long commands, then these targets will only break when the build would be broken anyway.
Trying to download the arduino-builder.zip fails via antivirus https://www.microsoft.com/en-us/wdsi/threats/malware-encyclopedia-description?Name=Trojan:Win32/Fuerboos.A!cl
I got the same so downloaded on my Mac and scanned with clamd and determined it was just a heuristics based false positive. Regardless, after applying the debug it resolved the path length issues but then later broke building for Uno target (I suspect boards support in debug files isn’t complete).
Looking forward to a long term solution. Curious, does Arduino 1.9.0 beta include the proposed Windows command length work-around ?
Hi @fiveangle , 1.9.0 beta contains the fix with 30000 characters limit (https://github.com/arduino/arduino-builder/pull/250/commits/f2cd784acb723e829c538ec19b6d4affc348dfe1) AND the arduino-preprocessor
integration, thus it could expose other kind of problems (like missing prototype declaration in case of non-standard cores); it would be supercool if you could test it with difficult sketches and report any problem you may encounter!
For anyone curious, I also had sucess fixing this problem by upgrading to 1.9.0 beta Arduino IDE.
Fixed in 1.8.9.
You could link together groups of .o files, and then link the resulting groups together into the final binary. A common approach for this would be to use archives (.a files), which contain multiple .o files. This already happens for the core, and libraries that have enabled this in their libraries.properties file.
Thanks a lot, this suggestion allowed me to finally compile latest Marlin firmware on Windows with Arduino IDE! For those stuck as well with Marlin, edit the file %USERPROFILE%\Documents\Arduino\libraries\U8glib\library.properties
and add the line dot_a_linkage=true
in the end. The command line is still pretty close to the limit though (31718 characters).
The @file approach seems promising, but needs more thought, especially wrt compatibility.
It does not work: {compiler.c.elf.cmd}
is set to gcc
most of the time, so gcc
parses the command line from given file, and builds a new command line to invoke the real linker program (ld
I guess) which fails because the command line is too big then.
i im having a issue with arduino paths being to long and dont let me compile Marlin-bugfix-2.0.x-new-layout firmware there was new arrangments on marlin and seems that now there is a problem in windows and arduino paths being to long this issue seems to afect alot people
the issue discution is on this link https://github.com/MarlinFirmware/Marlin/issues/7707#issuecomment-331312824
is there any way to fix this problem?