Allow configuring board for IDE to select by default when sketch is opened

matsekberg commented 11 years ago

Enable the use of some meta-comment in the code so that the appropriate board is automatically selected for that code.

Additional context

Additional requests

If there is a preferences file, then it doesn't "travel" with the INO file. Better to use, as originally suggested, "meta-comments" in the INO file. Yes, one can have multiple files in a folder, includes, data, etc... but many users find a single, simple, INO file posted on the web and copy it in. Yes, Arduino code is very portable, but it's best for the IDE to at least try to select the right one for the sketch when the author bothered to specify it. Keep in mind, the vast majority of INO files will not specify the chip. This is primary for those poor souls who are maintaining a large set of programs over a range of devices and are tired of switching. And also to help out new users who don't understand that a sketch might be for a specific device when the author is kind enough to hint that in the INO file.
In any case, the IDE should never assume an action. Upon opening the file, and parsing meta-comments, it should offer to make those changes for the user. The user can accept or reject the offer. If the user rejects the offer, no changes are made. The user may just be browsing the code.
Keeping a simple format in the INO file means it also works as documentation for the human. E.g. // Tools / Board / Arduino Nano can be read by the IDE and also by a human.
As one menu change may affect the availability of others, they should be processed in order. e.g. // Tools / Board / Node MCU 1.0 (ESP-12E Module) // Tools / Flash Size / 4M (1M SPIFFS)
As humans hate to type, it might be good to have a menu option that records the current menu settings into the top of the .INO or updates the existing settings.

Things I'm less sure about:

When to trigger the parsing? File / open only? File / New, Edit / Paste? Upon any change to the file, but only if the current menu settings don't match the parsed meta-comments? I would lean towards the latter, but that does mean that will be parsed on every build. That isn't too much of a burden is it?
Which items should be included / excluded? Obviously, the File, Edit, Sketch, and Help menus should be excluded. Tools / Port and / Upload speed should probably be excluded. Perhaps just start with everything on the Tools menus and tune in later releases?
What to do if a board isn't available. Actually, that's pretty obvious, just error out. But the point is you have to look for that and interact with the user about it.

timkoers commented 6 years ago

Sure! I'll work in my own branch for now and place a pull request once the feature is finished

Avamander commented 6 years ago

@timkoers Just mentioning, compatibility with doxygen would be really nice (so that the generated documentation actually also mentions the sketchs' intended boards and etc.), it would be a good idea to even contact the doxygen developer if you aren't that familiar with doxygen yourself.

timkoers commented 6 years ago

Sure! What I could do is use the doxygen syntax formatting so that doxygen is able to generate the data out of the box

timkoers commented 6 years ago

Can someone reopen this issue and add the label in progress to this issue?

timkoers commented 6 years ago

Just a little update, I managed to get it working! The syntax is as following:

/** Arduino IDE Board Tool details
 * Board: Node32s
 * Flash Frequency: 80MHz
 * Upload Speed: 921600
*/

JamesNewton commented 6 years ago

Love it!

Is it looking for the exact text "Arduino IDE Board Tool details"? If so, may I suggest something simpler and less likely to be incorrectly entered? E.g. "Settings" or "Tools Menu" or even "Arduino Tools Menu Settings". I'm trying to come up with things that are unlikely to be misspelled or typoed. E.g. You can look up and see the word "Arduino" in the ide so that's a good choice. And it is the "Tools Menu" so that's easy to remember. Probably "Arduino Tools Menu" is best. It tells you exactly where it's going to have effect.

timkoers commented 6 years ago

Sure! It is a single line of code to edit :-) I'll add the header automatically when a new file is created and warn the user if there is a settings mismatch when loading the file.

Should I create a dialog that notifies the user that the settings have been loaded from the header?

Van: JamesNewton notifications@github.com verzonden: maandag 19 maart 2018 10:51 p.m. Aan: arduino/Arduino Cc: timkoers; Mention Onderwerp: Re: [arduino/Arduino] Autoselection of device (#1481)

Love it!

Is it looking for the exact text "Arduino IDE Board Tool details"? If so, may I suggest something simpler and less likely to be incorrectly entered? E.g. "Settings" or "Tools Menu" or even "Arduino Tools Menu Settings". I'm trying to come up with things that are unlikely to be misspelled or typoed. E.g. You can look up and see the word "Arduino" in the ide so that's a good choice. And it is the "Tools Menu" so that's easy to remember. Probably "Arduino Tools Menu" is best. It tells you exactly where it's going to have effect.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/arduino/arduino-ide/issues/2438, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEELRavnKUyZJ_LKKjXnUrZ9JC2rNr3Lks5tgCh3gaJpZM4AwkHs.

facchinm commented 6 years ago

@timkoers thanks for your effort! I took a look at https://github.com/timkoers/Arduino/tree/board-autoswitch and the method is totally convincing. My only concern is that it's totally tied to Java IDE, so if you compile the same sketch using plain arduino-builder it doesn't apply the in-sketch configuration. It could also be a feature, so the plain builder only considers the fqbn provided explicitly.

timkoers commented 6 years ago

No problems! I’ll try to convert the methods into the arduino-builder (that is the arduino-core right?). I’ll try to fix more issues like these, so it would be usefull if I can manage the issues of the Arduino repo too.

Tim

per1234 commented 6 years ago

arduino-builder is here: https://github.com/arduino/arduino-builder

per1234 commented 6 years ago

It would be nice if the Arduino Web Editor and the Arduino IDE could use the same system for this. I'd like to see the functionality of the two diverge the least amount possible. That could mean either implementing the system added to the IDE code in the web editor or vice versa. It's not clear what system the Arduino Web Editor uses to store the sketch metadata. Making the changes in arduino-builder rather than the IDE would facilitate that goal since that tool is also used by Arduino Web Editor.

It could also be a feature, so the plain builder only considers the fqbn provided explicitly.

I think if arduino-builder was made to use the sketch properties when no fqbn was specified but override those properties when one is specified then that would provide this feature and more.

Doing it in arduino-builder could lead to code duplication since the IDE must make the correct menu selections according to the sketch properties. The solution I see is for the IDE to run arduino-builder -dump-prefs on the sketch then make the correct menu selections according to arduino-builder's output.

timkoers commented 6 years ago

You mean code duplication as in, the Arduino IDE reads and parses the settings and the arduino-builder reads and parses the settings?

I tried to execute the arduino-builder command, but it is not letting me. Is the output of the -dump-prefs command documented somewhere?

So to get confusion out of the way, I need to change the arduino-builder to do the things that I've programmed the IDE to do (read the header and select the right boards), and I need to reprogram the IDE to read the settings from the arduino-builder -dump-prefs command.

How would we handle the opening/reloading/closing of the file, should I execute the arduino-builder everytime?

Many thanks

per1234 commented 6 years ago

You mean code duplication as in, the Arduino IDE reads and parses the settings and the arduino-builder reads and parses the settings?

Exactly. It seems silly to have that same code in two places and need to maintain it.

Is the output of the -dump-prefs command documented somewhere?

Well, somewhat: https://github.com/arduino/arduino-builder/blob/master/README.md#usage Or just run arduino-builder with no options for the same output.

So to get confusion out of the way, I need to change the arduino-builder to do the things that I've programmed the IDE to do (read the header and select the right boards), and I need to reprogram the IDE to read the settings from the arduino-builder -dump-prefs command.

How would we handle the opening/reloading/closing of the file, should I execute the arduino-builder everytime?

That was my suggestion. You should be aware that I don't really know what I'm talking about and have no official status in making decisions. So don't misinterpret what I say as coming from a member of the Arduino development team (such as facchinm).

facchinm commented 6 years ago

The direction is to remove as much logic as possible from the Java IDE, making it more or less an interface to cli tools. This will make it easier to share the tools between all "unofficial" IDEs (like VSCode/Atom) so the resulting behaviour is the same across all platforms. In this vision, "Add build settings to .INO file" JMenuItem would call an "endpoint" provided by the builder that takes care adding the settings to the actual file. Of course there are problems there too, because the builder is only aware of files already written on disk. I don't have a real solution about this, but we can discuss freely, starting from @timkoers code. The feature appears to be much needed these days so I'll support every effort for fixing it in a sane way.

timkoers commented 6 years ago

I took a look at the android-builder source code, I don't understand a single line of the Go language. I have no problems learning another programing language, but I don't think this is the right way (as I'll probably make the code very inefficient) @facchinm do you have more experience coding in Go?

JamesNewton commented 6 years ago

I think it's important to see that this feature is an IDE, and NOT a compiler feature. It is NOT for CLI users, but for those who have little experience, to help them use the IDE and understand which board / settings the INO file is for, and for people who use the IDE all the time and so don't enjoy the nice features of CLI make files, scripting, etc...

Should I create a dialog that notifies the user that the settings have been loaded from the header?

More than that, I would strongly suggest adding a dialog that asks the user if they wish to have the settings loaded. Actually, it should first check to see if the settings need to be loaded (e.g. if the current settings are any different from those in the meta-comment) and then check to see if they can be loaded (e.g. if those options are available) and then ask the user if it should be loaded. Once the user agrees, having a confirmation dialog is probably unnecessary.

timkoers commented 6 years ago

I am doing the last part in my code (check if they can be loaded) first, because it makes no sense to ask the user if the IDE should load settings and then notify that the settings are not found xD I haven't implemented the dialog yet, if you'd like to do it, please make an preference option that will allow the more advanced user to disable the IDE to ask if it should load the settings everytime

adwsystems commented 6 years ago

As I am working with several different Arduino boards (Pro Mini 3.3V and 5V, UNO, MEGA, Nano, and possibly Micro. This is a feature that would be GREATLY useful. It would make sure that when I open the sketch the board selected is aligned with the board being used. Currently I am working on a project that uses 1 Mega, 1 UNO, and several Nanos.

timkoers commented 6 years ago

I made the code to integrate this into the IDE, I've created a pull request that they haven't accepted yet. Right now a merge conflict has occurred, due to them not merging my code 🤣 and changing part of the original code, that I haven't edited in my files.

AndreKR commented 6 years ago

Seems that branch needs a rebase.

lmihalkovic commented 6 years ago

I think I will do the following

create arduino.json

have the following contents

{
environments:[
"__all__": {
   "config": {
       "shared1": "valShared1"
   }
   , "define": { "DEBUG": 0 }
}
"my wet dream 1": {
   "config": {
       "var1": "val1",  "var2": "val2",  "var3": "val3",  "var4": "val4"
   }
   , "define": { "BOARD_XXXXXXX": 1 }
}, 
"but this can also do": {
   "config": {
       "bar1": "bal1",  "bar2": "bal2",  "bar3": "bal3"
   }
   , "define": { "BOARD_YYYYYY": 1 }
}
]
}

add a dropdown list (on the editor toolbar) with "my wet dream 1" and "but this can also do" and I can choose between the two
add a editor for arduino.json (doing the same that atmel had done to Visual Studio)

THE DROPDOWN IS A MOCK (the rest is not)

adding a -env "my wet dream 1" to the command line of builder would make it do the right thing ... the gods helping..

PS: I added the 'define' section. These are #define XXXXX to be passed on to the. I use these #define BOARD_XXXXX_AAAAA or __#define BOARD_YYYYYY__ to control a number of features that are not equivalent depending on which board I am running on.

PPS: to be fancy, it needs also needs a 'targets' section with the same contents as the 'define' section, but for special purposes (like targets: { debug Light': { ... }, 'full debug': { ... }, release: {} }__)

It would be real nice if arduino-builder would know how to define them for the command line of the compiler

Again... these are just my personal thoughts

lmihalkovic commented 6 years ago

This is a more realistic example that I will be using for my project. I added UUIDs so that even if the conf gets renamed builder can use these names as a basis for the cache name it generates (syntax highlighting is all wrong because I havent tried to changed it yet - but the editor understands what that file is)

mirage335 commented 6 years ago

Solved.

ArduinoUbiquitous looks in the sketch directory for an "ops" file, importing it into its environment. Any shell commands, including preferences configuration, such as board type, can be hooked and forced from there. https://github.com/mirage335/arduinoUbiquitous https://github.com/mirage335/arduinoUbiquitous/blob/master/_local/ops https://github.com/mirage335/arduinoUbiquitous/blob/master/_prog/core.sh

The framework also contains applications, like Arduino, to use arbitrary home folders, and provides a structure to maintain 'splice' instructions against existing software. Such functionality can be used to maintain a a statically compiled recent OpenOCD build, a custom Arduino hardware core, or a custom blockly visual programming environment. https://github.com/mirage335/openocd-static https://github.com/mirage335/SigBlockly-mod https://github.com/mirage335/SigBlockly

Extremely flexible virtualization features allow the entire development toolchain to be launched, from the host filesystem, in a virtual machine image which can simultaneously be run under ChRoot, Docker, QEMU, and VirtualBox. Multiple toolchains can be contained to a "lab" set of directories, all Ubiquitous Bash forks using git submodules. https://rawgit.com/mirage335/ubiquitous_bash/master/USAGE.html https://github.com/mirage335/ubiquitous_bash/blob/master/lab

Atom is integrated. A general purpose Arduino debugging IDE is on the way.

AndreKR commented 6 years ago

I fail to see how this arduinoUbiquitous solves this problem. In fact I don't even understand what it is and how to use it. Am I supposed to replace some components of the Arduino IDE with parts from that repository or what?

mirage335 commented 6 years ago

It is a huge wrapper bash shell script for ArduinoIDE, which includes functionality to open a text file named "ops" in the same directory as the sketch, and run commands from that file before launching the IDE. This wrapper script is also extensively broken down into functions, some of which are meant to be overridden by re-defining with this "ops" file, effectively 'hooking' some code to be run at various times in the launch sequence.

So you could put this, in a file named "ops", in the same directory as "Blink.ino".

#Uncomment to force. Consider configuring through GUI instead.
_arduino_prepare_board() {
    #true
    _arduino_set_board_zero_native
}

Which sets the "_arduino_set_board_zero_native" function, included with the wrapper, to run before the IDE, forcing the board type to be set to Arduino Zero (Native). Then you would run something like this. "../arduinoUbiquitous/_arduino ./Blink.ino"

Arduino itself is just installed to "_local/h/arduino-1.8.5" or similar. This is already included in the repository. The "_local/h" folder is used as the user's home folder when launching the IDE, though symlinks are used to allow the ArduinoIDE to use its own internal "portable" mode with the same files.

JamesNewton commented 6 years ago

@mirage335 a couple of questions:

"bash shell script" so that means it only works on Linux systems?
Can you put a script somewhere other than the project directory, which will run when a new .ino files is opened, which will look through the commends in the .ino file to see if there are comments which direct the IDE to select a specific board / board settings / etc..., popup a dialog to confirm that with the user and then finally, actually make those setting changes?

I ask 2 because having a second file in each project folder is exactly what I don't want. Instead: https://github.com/arduino/arduino-ide/issues/2438

mirage335 commented 6 years ago

#1 No. Technically Linux is most supported, because it is what I am using, but the underlying scripts have been used under Cygwin, autodetection of MSW executables is possible with the architecture, and OSX support in theory partially works already. That said, Linux is both open-source and by far the most convenient platform for development, so I expect the most serious developers to use it when possible. Also, because Linux is free, it is the easiest operating system to include as a virtual machine fallback, which the parent project, Ubiquitous Bash, makes exceptionally useful for integrated applications.

#2 Yes. This could actually be done on a per-project basis. You would only need to place an "ops" file in the wrapper script's own directory, overloading the "_arduino_prepare_board" function with an implementation to read the sketch file for this information, and install it. Best way to do that might be to allow raw shell commands to be included in the file. Such an implementation could also replace the current placeholder under "core.sh".

per1234 commented 6 years ago

This issue tracker is for bug reports or feature requests for official Arduino software. I'm sure arduinoUbiquitous is really great but this is not an appropriate place to promote or discuss a 3rd party project.

mirage335 commented 6 years ago

Correct. The discussion as to how to use arduinoUbiquitous, or any issues it has, should be taken to its issue tracker.

However, my intent in mentioning it here was to point out a solution - specific to Arduino itself - to an essential feature request left unaddressed since 2013. Having done so with an external framework may also imply that the original feature request is not within scope of the Arduino project itself.

lmihalkovic commented 6 years ago

IMHO this is just one of the many hackish way of addressing the question of board selection. I have no doubt that it would work, but there are times when no solution is better than a poorly devised one, because it leaves the hope alive that there might one day be a good one. And designing a good solution is not difficult either.. there are plenty of good solutions out there to base one on.. it is mostly a matter of will and resources.

mirage335 commented 6 years ago

UNIX philosophy. Do One Thing and Do It Well.

Arduino can be used to develop stand-alone interactive objects or can be connected to software on your computer (e.g. Flash, Processing and MaxMSP).

ArduinoIDE compiles/uploads code, and includes some libraries. Beyond that, it is the growing ecosystem that makes it powerful. Shields, machine control panels, g-code generators, web interfaces, etc.

For the Arduino project itself, I wonder if embedding a specialized, probably java based, "meta-comment" processor is considered gold plating. Some challenging extensibility features could be required to address user's needs. Also, a debugger would seem to be in a similar category of desired functionality, as an example of how difficult this entire category of functionality could be to integrate directly into Arduino.

Meanwhile, there seems to be plenty of ongoing bug fixing and release cycle work going on, directly related to what Arduino does really well. Focusing on stabilizing Arduino itself, maybe making it even more modular (Arduino Builder looks cool by the way), and maybe supporting interfaces third party software can use, might do much more for power users in the long term.

Avamander commented 6 years ago

@mirage335 It's an IDE, the whole point is to gather up a lot of tools and make them usable.

timkoers commented 6 years ago

How would a strong and robust board header/board definition look like when it's up to you? I think we should discuss about how we are going to implement a solution instead of what we shouldn't do?

By the way, I already coded a solution that looks like this in your Arduino file

Issue 1481

mirage335 commented 6 years ago

At least arbitrary setting of any Arduino preference, including programmer. Overriding the libraries directory would be useful too. Maybe some sort of arbitrary code execution, search and replace, as well.

@Avamander An IDE of highly limited scope. Just on account of not having a debugger, forcing specific libraries for various projects, and keeping a complete compiler with git submodules for reference, Arduino by itself still will not be usable for me.

timkoers commented 6 years ago

@mirage335 can you be more specific when it comes to:

"At least arbitrary setting of any Arduino preference, including programmer. Overriding the libraries directory would be useful too. Maybe some sort of arbitrary code execution, search and replace, as well."

mirage335 commented 6 years ago

Everything in preferences.txt . Equivalent to running this from the meta-comment. --save-prefs --pref programmer=arduino:sam_ice

mirage335 commented 6 years ago

Also, while I don't want to derail the issue tracker, I think it is worth mentioning that with an external tool approach, it is possible to automatically symlink a "libraries" directory, neighboring the sketch itself, into a temporary home directory. Doing so makes it much easier to force Arduino to compile firmware with a specific set of library requirements.

Avamander commented 6 years ago

@timkoers Your suggestion is exactly what I'd like to see.

@mirage335 You have the C(++) preprocessor for most of what you've described. Board selection as such can't be done by it and that's why it's discussed here. Editing an additional file loses in both clarity, UX and usability and thus in my opinion can't be the solution.

Maybe some sort of arbitrary code execution, search and replace, as well.

That's a really horrible idea from security perspective and must not be implemented.

mirage335 commented 6 years ago

C Preprocessor cannot set the programmer type. When dealing with custom cores, having a programmer set to something from another core seems to cause build failures in the ArduinoIDE. Moreover, setting board type cannot force a huge 3d printer firmware codebase to be compiled against libraries in the local directory, which is my biggest nuisance with Arduino. Asking me to install anything to a global directory because some specific microcontroller firmware version needs it is completely unacceptable. I would rather set up a dedicated virtual machine, but should not have to.

That's a really horrible idea from security perspective and must not be implemented. To be clear, I am suggesting the ability to manipulate variables in the ArduinoIDE, not to run code on the host computer. But while even that might be unwise to enable by default, "must not exist' is a strong position to take. Arduino firmware developers are working on code that will at least have root access to the microcontroller itself. In the context of Internet of Things, machine control, and industrial automation, the security issues are already there.

matthijskooijman commented 6 years ago

Folks, even though I think this is interesting discussion, most of it is getting off-topic for this isse. Great that there is an tool by @mirage335 that can solve this issue externally, but that does not remove the wish for solving this internally in the IDE as well. Let's keep discussion on that topic (and if anyone wants to pursue the other topics, the developers mailing list might be a better place).

Thanks!

timkoers commented 6 years ago

@Avamander it's here!

arduino / arduino-ide

Allow configuring board for IDE to select by default when sketch is opened #2438

Additional context

Additional requests

Related