Open paniag opened 4 years ago
@paniag thank you for the the proposal/suggestion. We've been following your project :) I'll have to check with @cmaglie . You seem to have a specific and detailed case and it sounds like something others might need. I like how some of our users push the boundaries :D
Interesting project. I happen to be looking at a similar thing for a customer (running firmware on a regular PC using mocks for the Arduino API / hardware), so I'm much interested in this as well. Here's some misc thoughts:
main.cpp.o
. But if you want to always omit this file, and have a custom Arduino core, why not just omit main.cpp
from the core? I'm not sure I understand your need, though, since your build still needs a main function, right? Or maybe you want arduino-cli to build just the library, and later include it in a bigger build system? Then I wonder, why not just let the bigger system just build the arduino library and mock core as well?#if !defined(OMIT_MAIN)
around your entire main.cpp
file, and then run arduino-cli --build-properties build.extra_flags=-DOMIT_MAIN
or so (but also see #846 for some caveats about the property name, though with a custom core, you can use whatever property you want).setup/loop
, provided your customized core does not need those.Thanks for your comments!
tl;dr: Your comments led me to try another configuration method, and it worked! While I was able to achieve the desired result (building a library from a sketch to link into a larger binary), personally I did not find this method to be easily discovered. So I would like to keep some form of this proposal open. Read on for details.
Highlights
Detailed response @matthijskooijman Very cool to find that we're not alone in wanting this kind of setup! We should chat; we might well be able to learn from one another. See https://github.com/keyboardio/Kaleidoscope/wiki#community for how to get in touch with us. Please find point by point responses below.
Summary
Conclusion: I believe this proposal is still useful and relevant. Please let me know your thoughts.
Thanks for your additional clarifications. I believe I agree with pretty much everything you said :-)
I initially thought that your suggestion was to add a command that only omits the main.cpp.o
file, but now it is clear that another (possibly more important) part of your suggestion is that no executable will be produced, but an object file (or whatever the format is exactly) that can be later included in another build.
Thanks for sharing your current virtual core, that might indeed be useful inspiration. One more thought I want to share is that if such a core is further developed, it should probably have a flexible and powerful way to mock Arduino API behaviour. arduino_ci has a "Godmode" class for this, that allows e.g. setting pin values, but I've recently wrote down some thoughts about making this more configurable, that you might find interesting (though it could very well be that the googletest framework solves some or all of this, maybe even in a more powerful way?
I have some more thoughts below. It has become a bit of a braindump, apologies for the lack of structure, but hopefully it will be useful nonetheless.
I do wonder how much of this and how it should be included in arduino-cli directly. I.e. it would seem wrong if arduino-cli would hardcode the "main.cpp.o" filename and omit that from the file list, so that should probably be somehow configured in platform.txt
. And it would be wrong if arduino-cli would somehow change the linker commandline to not generate an executable, for that platform.txt
would need to offer a different linker command.
So platform.txt
would need to modified in any case, so do we really need this option (I'm not arguing we don't, I'm just asking)? I can imagine it can make things easier, but it would have to be used in tandem with modifications in platform.txt
. In other words, this option will not allow compiling into non-executable object for arbitrary for arbitrary cores. I initially thought that this would only be useful for such "virtual" cores, but thinking on this more, I can also see a usecase where you would build a non-executable object for an actual microcontroller, which you then link with a custom main function, or maybe include in a bigger application, maybe even using an embedded OS of some sort.
So then the question is: Is a single "output executable or linkable library" flag sufficient, or does that assume too much? I can, for example, imagine that a core could define multiple modes. The default would of course be an executable, but I can imagine that a core could operate in a mode where it omits main()
, but still exposes an arduino_init()
function that initializes the hardware needed (e.g. timers), or another mode where it does no initialization at all? Maybe this is could of course just be some extra defines or build properties, but maybe all of this could be build properties?
Maybe the question is: If this option is added, how should it influence the build process exactly (in terms of omitting which filenames from compilation, replacing which recipes from platform.txt which which other ones, etc.)? How can this be done in a way that's easy to use, but also remains flexible for the platform developer to implement what they need?
@matthijskooijman I think your comments are entirely fair.
If the core symbols (like main
) in main.cpp
are weak, then actually excluding main.cpp.o
would be unnecessary. I don't believe this is the case for the standard arduino core though.
I'm not sure I agree that specifically excluding main.cpp.o
is an unnecessarily special case. Any program's main.cpp
is a special file already, responsible for providing the entry point to an executable. On those grounds alone, I think it makes sense to single out main.cpp.o
. That said, I agree it could be a convenient feature to drop object files from core.a
in a configurable way if a use case arises for having a partially-virtual core. Maybe something like that could be useful in porting Arduino to new embedded platforms. I'm not sure. I'm also not sure that the standard Arduino core is designed with this level of modularity in mind in the first place, so it may be a moot point. It would be good to hear from someone from Arduino (or at least someone more knowledgeable than I am) about this.
Ultimately, I agree with your point that if someone is doing sufficiently tricky things to need to modify or swap out the standard core then they are already modifying platform.txt
and the value in having a specific option to control that behavior outside of platform.txt
has value primarily in terms of convenience.
That said, if the above mentioned case of selectively dropping out pieces of core.a
is useful, I don't believe that functionality is currently configurable even via platform.txt
. My understanding is that arduino-builder/arduino-cli has full control over how the core is built. Configuring platform.txt
just enables you to drop or swap out the whole thing. Feel free to correct me if I'm mistaken.
To summarize:
platform.txt
does already provide sufficient configuration to achieve my original goal stated in this issue.What do you think?
If the core symbols (like main) in main.cpp are weak, then actually excluding main.cpp.o would be unnecessary. I don't believe this is the case for the standard arduino core though.
This is not the case for the Arduino core, but that could be changed (
I'm not sure I agree that specifically excluding main.cpp.o is an unnecessarily special case. Any program's main.cpp is a special file already, responsible for providing the entry point to an executable. On those grounds alone, I think it makes sense to single out main.cpp.o.
You have a point, but I'm just saying that from the arduino-cli perspective, it is not special. It's just the file that happens to contain the main()
function, but a core could also put the main()
function in a NotMain.cpp
, or even leave it out entirely if the particular architecture does not need it (i.e. has another entrypoint). The latter is a bit far-fetched, but it shows that there aren't much assumptions arduino-cli can really do here.
My understanding is that arduino-builder/arduino-cli has full control over how the core is built.
This is partially true: arduino-cli handles which files are included in compilation, which are then compiled using the normal c/cpp compiler recipes in platform.txt and combined into core.a using another recipe from platform.txt IIRC. But indeed, there is no way for a platform.txt to omit specific files from the core (other than using hacks like wrapping the compiler with a script that drops things from the commandline, but I'm going to ignore that).
What do you think?
Agreed, getting the usecases clear is probably needed first.
Since my last post, I've been working more on my project and I'm also feeling the need for a compilelib
option a bit more. I'm now relying on a custom Makefile that sets up all include paths to libraries similarly to what Arduino does, which works, but it would be nicer if I could just pass my sketch to arduino-cli and have it output some .o file that I could then link against my virtual code (i.e. simulator main function, or unit tests) to produce the final executable.
However, writing this, I realize that my simulator/unittest code actually references types and code from the sketch and libraries, so my unittest code should probably be compiled by arduino-cli as well (or at least with the same or similar compiler flags and include paths). That could significantly change the requirements: Instead of producing a .o file to be linked externally, I would need to externally insert code into the build process (which could be a matter of a conditional include in the sketch, with the unittest code in an Arduino library, but I'm not sure that seems right).
That's a significantly different usecase from the "embed into other OS", maybe, since that would typically just need a few functions defined by the Arduino sketch to be called by the OS. Though maybe tighter integration could also make sense here, but then you might have code that relies on the (include paths for) the Arduino sketch and the external OS, so then who should compile this code without duplicating flags and include paths? Should Arduino-cli maybe just allow exporting include flags? Hm.
arduino_ci owner here, subscribing for interest. (I moved the project and your GitHub links to it in this thread are broken, s/ianfixes/Arduino-CI/
)
I haven't looked at arduino_ci personally. I think it may have been considered in another context, but I'm not sure. However, I would strongly prefer a solution using an official (rather than custom) Arduino build system. The reasons are quite simple: custom systems may drift from the spec over time (or even just from the very beginning)
arduino_ci
began as a prototype to see whether hardware-less CI testing was even possible. At the time, arduino-cli
was very limited in its functionality so I had to wrangle the IDE in some very roundabout ways to get it working. The drift you were concerned about is/was very real and very frustrating. (The lack of official unit testing capability was really my only motivation for the project in the first place.)
Now that arduino-cli
covers most of my use cases (in wonderful JSON format) I'm in the process of adopting it as ardunio_ci
's backend: https://github.com/Arduino-CI/arduino_ci/pull/218
When that's complete, I'll be very curious to see how much of my codebase is useful in support of this issue.
I've completed the backend swap of arduino_ci
from the graphical IDE to arduino-cli
version 0.13.0
. I love it! Plus, it can now support a GitHub Action for testing Arduino.
From here, I'd like to look more closely at coordinating with this project, to hopefully get backend support for some of the values I'm simply guessing at right now (e.g. the proper compiler flags for various platforms). I'm sure this issue thread isn't the right forum for this; can you point me to the right one?
I'm sure this issue thread isn't the right forum for this; can you point me to the right one?
If there is no existing issue that fits the bill, I suspect the best approach is to just create a new issue to describe what you need, and we can always point to another issue, split into multiple issues, etc. from there.
@ianfixes we're very happy you like what we're doing here, it really means a lot for the team. As @matthijskooijman said, feel free to post an issue and one of us will definitely go through it. We tend keep an eye notifications on a few repos, this being one of them ;)
The github actions bot closed a lot of issues that I had submitted, with no explanation. Was this by design? I'm still interested in getting official "will consider this" / "wontfix" on any of those unceremoniously-closed issues, and I apologize for asking here but I don't know where else to direct the question.
Hi @ianfixes. I apologize for the confusion. I seem to have broken the "stale-bot" workflow somehow. This was never intended to be closed.
Thanks for clarifying -- it looks like you restored a lot of issues. I didn't realize how many things I was subscribed to in this repo until that happened :)
The name of the command isn't important, but the functionality is. Our use case is for testing large and complicated sketches as native applications on the host.
Desired functionality: build a library as a normal Arduino build that omits only main.cpp.o from core.a. Toolchain configuration to build for the host would be provided through a user's custom platform.txt and board.txt.
Use case: keyboard.io develops keyboard firmware, and we would love to have a native environment where we can do sophisticated testing involving simulation of the hardware. The firmware is highly customizable and extensible, but testing it is currently a highly manual endeavor involving flashing firmware onto a physical keyboard and actually typing. In particular this would allow us to incorporate arduino-cli into a larger build system and link against test and simulation libraries not built by any form of the Arduino build process. Concretely, we'd like to link against googletest and we have already put together a minimalist "mock" arduino core for native compilation.
Forking: We'd like to avoid forking arduino-cli due to the maintenance cost of keeping it up to date. We'd be happy to do the development - it should be a relatively simple change to make, could even be a flag passed to the existing compile command. We think this would be helpful to the community as well since it provides greater flexibility, so we're hoping you would be willing to take on maintenance of this feature.
We'd love to hear your thoughts on this!