IDE/Eclipse integration?

[ilg-ul]

I am considering using yotta modules for my own projects (GNU ARM Eclipse and XCDL), but, in addition to command line support (which I fully agree to be mandatory), I would also need IDE integration (more specifically Eclipse integration).

Unfortunately, I think yotta has some portability problems, which makes this integration difficult:

• the metadata is/cannot be fully represented in an easy to parse, language neutral, format, like JSON (or even XML); the CMake files cannot be edited programatically as a result of some GUI actions - I think this is the major weakness of yotta
• it makes heavy use of Python
  • in the command line program
  • in scripts running from the modules
• it is too CMake centric
• it uses less common build tools (like ninja)
• (maybe others that I did not identify)

These days I've been thinking on these issues and here are some thoughts:

• ideally, the entire metadata associated to a module would be represented in JSON; the command line utility (and possibly the Eclipse plug-ins) would read/update the JSON files and generate the required files to conduct the build (by default it would generate make files, but any other format should be posible). unfortunately this is not true, since yotta modules include CMake files, which are hard to parse and even harder to update programatically
• ideally, the entire functionality would be available within the existing Eclipse build environment (which is a set of Java plug-ins and the make command line program); this is far from possible now;
• the second best solution would be to have yotta as a single executable, statically linked, easily to build for Windows, OS X and GNU/Linux, and distributed with Eclipse; this is also not very realistic, due to modules dependencies on CMake and Python;

As possible compromises:

• the ninja-build can be avoided by changing the CMake config to generate make files (obviously with speed penalties);
• the command line program can be reimplemented in a compiled language; for my build system I plan to use an utility (named xcdl) written in Go; one possible solution would be to reimplement part of the yotta functionality and use the same utility for both the standard yotta and my custom purposes;
• the Eclipse plug-ins would generally call the xcdl utility, but also reimplement some functionality natively (for example searching/browsing the repository);
• although I don't feel very comfortable with this, CMake could also be packed and distributed in the same package as xcdl, but preferably only to maintain compatibility, the XCDL build system would better not depend on CMake (to be decided);

As far as I can tell, the only missing thing from this image is the possibility to run Python scripts from build scripts.

I do not know how many modules would be affected by not having Python available and if it is realistic to imagine if a version of yotta without Python can be functional.

In case there is anyone else interested in such an Eclipse/other IDE integration, I'd be glad to discuss details.

[autopulated]

Thanks for the detailed feedback, easy things first :grinning:

• yup, ninja can be easily switched for Makefiles, to do this just run yotta build -G "Unix Makefiles" instead of yotta build.
• while rebuilding yotta in an easier-to-bundle/compiled language is definitely an option in the long run, for now we have python. It isn't inconceivable that using py2exe/similar will be possible soon however, which might give many of the same advantages. It looks like the cryptography module for example has made some big steps towards supporting this: https://github.com/pyca/cryptography/issues/2039 I'm tracking the need for this in #607

And the hard things:

• Re-implementing yotta's logic separately: (e.g. as an eclipse plug-in, or as part of $OtherIDE, instead of calling the yotta command line program from the IDE): I don't think this is a good idea, not least because yotta is still young. Having multiple separate implementations would make it much harder to make big improvements in the future, and even to make small changes in a backwards compatible way.
• Bundling external programs into a single binary: this would be much harder than just containing all yotta's own dependencies. Conceivably it could be possible with a lot of work for CMake, although I believe CMake calls itself, and it certainly uses the filesystem for its internal configuration, so those files would need to be installed somewhere or CMake would need to be patched.
• The use of CMake at all: I still believe it's the best compromise. Building software, especially embedded software, can get very gnarly around the edges. I like that CMake has a well-defined, and well-document (if not the prettiest) language for defining all the rules that you would want to use to do that. It's certainly favourable over devising another new language to do the same. This only leaves other options like scons, jam, gyp, plain make, etc. ... these would still be an external dependency that needs to be run and supported across all host operating systems.

I'm sure that there are other people interested in IDE integration, for both Eclipse and others, here ;) I hope this can be a good place for further discussion.

[ilg-ul]

CMake ... Building software, especially embedded software, can get very gnarly around the edges.

yes, building embedded software is not easy, and if you want to support all fantasies that people can come up when using a command line approach and manually configured build systems, it is close to 'mission impossible'; but if you use a managed build environment like Eclipse CDT you do not have to deal with these details, you just throw in sources, headers, linker script, compile options, defines, libraries, everything, and on the other side you get the binary. the CDT build handles all details, generates simple make files, and the build is quite fast. I never felt the need for more, so... I cannot confirm that CMake is really required.

[Timmmm]

I strongly agree with the original comment. I've tried to integrate yotta with Qt Creator, which actually supports CMake. It sort-of works, but you have to re-run yt build to add/remove files, and the linking step doesn't work (Error: you do not have 'intelhex' installed). CMake and Python are fine if you do all your editing from the command line, but they're about the worse choices possible for IDE integration.

It's too late to change in the short term, but in the medium term I would:

1. Rewrite the yotta program itself in Go. It has a nice large standard library to simplify things like HTTP. There's even a git library you can use (rather than distributing the git binaries - I must have 5 copies of them on my machine by now). It also has the huge advantage of being trivial to cross-compile statically linked binaries. Good luck doing that with Python or C++.
2. Support more build systems. Unfortunately there still isn't a standard IDE-friendly build system so I can't see any way around it other than just supporting several of them. Some support file globbing so at least you won't have to re-run yt build just to add/remove files. I'd recommend:
   • QBS
   • Eclipse
   • Visual Studio

I don't think you should do this by using CMake to generate the build files.

[ilg-ul]

thank you @Timmmm for your comments. unfortunately I doubt they will make any difference for yotta.

but in xPack/XCDL I plan to correct all these issues, so if you are interested, I can keep you posted on the progress.