search

platformio / platformio-core

Your Gateway to Embedded Software Development Excellence :alien:
https://platformio.org
Apache License 2.0
7.81k stars 787 forks source link

ZephyrProject integration #1613

Closed leodesigner closed 4 years ago

leodesigner commented 6 years ago

Hi,

I have been playing with https://www.zephyrproject.org/ recently. Seems like they are developing really fast. A lot of SOC supported, especially bluetooth / bluetooth mesh is very promising. It would be great addition to Platformio. Please consider adding Zephyr support.

Thanks.

jracle commented 6 years ago

Same comment as @leodesigner.

borud commented 5 years ago

I talk a bit with various chipset vendors and it seems that Zephyr is the platform they are interested in supporting. So I would expect growth in the number of people interested in Zephyr. So I think it would be well worth it.

borud commented 5 years ago

To be a bit more specific: we work a lot with Nordic Semiconductor and their new nRF91 chipset for LTE-M and NB-IoT. Their recommended RTOS is Zephyr. In the telco industry there is a lot of interest in this chipset. Both because it is fairly powerful (dual core, includes a GPS, TrustZone etc) and because Nordic has an approach to developers that is much friendlier than most chipset makers.

Managing a Zephyr toolchain leaves something to be desired in terms of ease of setup and use, and I really like how Platformio has solved this for the platforms and frameworks I have tested.

ivankravets commented 5 years ago

I had a good talk at Embedded World 2019 with @nashif, who leads Zephyr project. We plan to back to this issue in the next weeks.

borud commented 5 years ago

@ivankravets @nashif This is really good news! (We're in releasing a NB-IoT / LTE-M module soon and we really want to use Zephyr. The awkward setup of Zephyr is going to be a major stumbling block for our customers, so it would be extremely useful to have something like Platformio to manage the framework)

carlescufi commented 5 years ago

@ivankravets Glad to hear you are moving forward with Zephyr integration. Let us know if you have any questions! Off-topic (sorry for hijacking the thread): @borud Curious as to what you find "awkward" about Zephyr and its build system (no sarcasm here, genuinely interested). We've made great efforts to ensure that the build and configuration system is completely cross-platform (Windows, macOS, Linux) and to rely only on CMake, Python and a few other tools. We are always looking to make things simpler for the users (out new tool, "west" is one step towards a better experience for example). Since this is rather off topic here feel free to open an enhancement issue in our GitHub repo. All feedback welcome.

borud commented 5 years ago

@carlescufi Hi! Thanks for the question. I'll try to clarify.

Making it cross platform is just the first step towards making a piece of software easy to approach. To make it palatable for a larger audience it has to be simple to set up and then maintain as well. Embedded programming has been its own special place for decades, but it is about to get overrun by more mainstream developers that have been taught to have high expectations. If Zephyr is going to keep its promising position just as the embedded development scene is exploding, I think it has to understand these new users.

Traditionally anything that involves cross compilers tends to be pretty horrific to set up and maintain. It isn't unusual to find developers who have set up multiple cross compilers that get into weird conflicts with each other and then you just have to help them clean out everything and start over. Or that developers try multiple recipes for setting up their development environment before finding one that actually works on their machine.

We develop MCU + NB-IoT/LTE-M modules that go into other companies' products. Since we need to support users that are new to Zephyr I actually asked a few developers on my team who are not experienced embedded developers to write some tutorials to familiarize themselves with Zephyr.

It turned out that just getting it set up was a problem so we needed to do a tutorial for that. (Since very few of the recipes that could be found on the net didn't work for one reason or another. Including the official guide). This isn't what users expect. They expect to be able to say brew install zephyr or apt install zephyr and the toolchain is installed. Developers then expect that whatever gets installed contains tools to manage cross compiler toolchains and even libraries etc.

Arduino was pretty early on to adopt the idea that you install the dependencies in a somewhat sandboxed fashion that doesn't interfere with the software installed on the system. And while Arduino isn't a professional tool, this is something they did better than most of the professionals: it requires no effort to install and fire up. It requires extremely little effort to maintain both new cross compilation toolchains and libraries. The way Arduino does this is somewhat akin to the way MacOS manages applications as well: put everything under an application directory so that it becomes trivial to manage and doesn't interfere with the rest of the system. (Being an old-fashioned UNIX-geek, I can see how this might rub purists the wrong way, but after 30 years of trying lots of ways I've become old and pragmatic and less patient with purists :-)).

Now, I would imagine that you want to mostly focus on getting things to just work across platforms and not worry too much about what you might see as wrapping or spit and polish. Which is why I think Platformio might be a good match for Zephyr. It lets you focus on what you care about, and it offers developers the experience they have come to expect.

Even the way Platformio is installed makes it easy for users. Usually I avoid Python-based applications because it is the versioning gift that keeps on giving. To quote George in John Steinbeck's novel "Of Mice and Men": "Whatever we ain't got, that's what you want." However, Platformio is no further away than a simple brew install platformio – and that makes me forget that it is written in the most application user-hostile language I know of :-)

marc-hb commented 5 years ago

@borud Curious as to what you find "awkward" about Zephyr and its build system

It turned out that just getting it set up was a problem so we needed to do a tutorial for that. (Since very few of the recipes that could be found on the net didn't work for one reason or another. Including the official guide).

Maybe sharing this tutorial is the simplest way to answer @carlescufi 's question and help the Zephyr project by reporting the issues.

I don't know about macOS and Windows but for Linux users the official instructions seem to work pretty seamlessly

The way Arduino does this is somewhat akin to the way MacOS manages applications as well: put everything under an application directory

I'm pretty sure the main reason Arduino is easy to use out of the box is not so much about where its installer copies its files but more about the number of them it bundles - not the least its own Java runtime.

Drawing the line between what's bundled versus what's expected is a question as old as ~DLL hell~ shared libraries and has never been a simple one. Bundling reduces the number of things that can go wrong and definitely helps with the "out of the box experience" but it also increases the maintenance burden (e.g.: what do you mean by "urgent security fix"?) and generally reduces flexibility. Software as usual.

ivankravets commented 5 years ago

@marc-hb you look from SDK's provider point of view. Try to be in end developer role where you have 10 boards, different vendors, different IDEs and different SDKs, different build systems, different configurations. It's very difficult to manage multi-board projects. Also, you lose a lot of time switching between different development environments (IDE, CLI, etc).

Let's take a look at docs for ESP32 => https://docs.zephyrproject.org/latest/boards/xtensa/esp32/doc/esp32.html A developer should manually install some detached ESP-IDF b2ff235bd00 version. What is more, need to export it globally in the system which conflicts with other ESP components which depends on ESP_IDF_PATH. This is just a small example.

I agree, using long installation tutorials is not bad for developers who finally stopped on some specific framework/OS and plan to use it for the next months/years. Nevertheless, for quick prototyping or just to "plug & play", these long steps tutorials are not the best candidates.

but it also increases the maintenance burden (e.g.: what do you mean by "urgent security fix"?) and generally reduces flexibility.

SDK providers can maintain PlatformIO integration in pair with their own build tools. In this case, the end SDK's developers can use any version of the SDK with confidence. See example how Espressif team maintains integration of their Arduino cores with PlatformIO Build System:

It can look so funny but this 1 platformio-build.py file is actually the whole integration bridge :)

it would be good to have the same "bridge" for Zephyr.

marc-hb commented 5 years ago

@marc-hb you look from SDK's provider point of view.

No I'm not. I'm comparing two types of developers/projects: 1. Just like you, the developer busy getting 10 completely different projects up and running as fast as possible without any idea of what (obsolete/buggy/insecure/etc.) software is involved behind the scenes cause life is too short. 2. The opposite type of high-quality project with a list of reputable, tracked, maintainable and maintained dependencies totally under the developer's control. The sort of project approach I wish were used to pilot planes, trains or cars.

If platformio can help with both ends of the spectrum then awesome! Most solutions seem to be much narrower.

@borud Curious as to what you find "awkward" about Zephyr and its build system

borud commented 5 years ago

I think you could say that an awkward setup where you will have to make lots of workarounds for conflicts that might arise from having tools installed globally is more tolerable for developers that can amortise the cost – but it still isn't ideal. It is still cost.

Not too long ago some people from my team visited a vendor who has chosen Zephyr as their primary framework for their chipset. It was supposed to be a quick workshop on how to use their chipset, but in reality it ended up being limited to "how to get Zephyr up and running". Some of the participants had prepared by trying to install Zephyr ahead of time, only to find that it was somehow installed wrong (I didn't get the details on what happened there), so they had to do it over again.

So what does this mean? Is the chipset vendor useless? Are the people who took part in the workshop useless? Is this common?

What I admire about Arduino is that they did what was necessary to solve the problem without getting too dogmatic about it: if the runtime is a pain in the ass: embed it. If system wide cross compiler toolchains tend to screw each other: isolate them and bring your own – or even, be able to fetch them when needed (as you can with the espressif SDK and compiler).

Now, is it the role of Zephyr to care about the developers? Perhaps not. I'd love to see more love for the developer by reducing pages and pages of install instructions reduced down to a single install command that installs something that acts like a good neighbor and keeps to itself and has tools to manage itself.

I think Platformio could save the Zephyr project a lot of work. And it would make Zephyr a lot more pleasant to work with.

marc-hb commented 5 years ago

So what does this mean? Is the chipset vendor useless? Are the people who took part in the workshop useless? Is this common?

No one can tell until specific facts are shared.

If no detail is shared back we can just tell open-source is probably not the right fit for these people.

jcw commented 5 years ago

Firm platformio supporter here. The reason: 1) clone a project, 2) type pio run, 3) done. Toolchains get installed as needed, libraries get installed as needed, and pio tells me when there are updates. It fits a pattern which suits me more and more: {apt,brew,pio} {update,upgrade}. I don't know in which category this puts me, all I know is it lets me focus all my energy on the project itself, not the tooling.

I hope that Zephyr will be available in the same way one day. It looks like a very interesting project.

borud commented 5 years ago

@marc-hb I'm not going to ask people to detail every step they needed to go through to get a toolchain for zephyr set up on their computers, including every gotcha they had to resolve along the way to get a functioning setup. The fact that this process took several hours (the better part of a day) to do should at least suggest that you may want to go out into the field, do some user research and figure out what developers have to go through to get an environment set up by quietly observing them.

I doubt anyone telling you about this will have any productive effect when your firmly held belief is that it's not a problem. Observing new users in the wild is really the only viable option - and before you dismiss them as idiots: they are the very people you depend on for success and growth. It isn't a good idea to call them idiots.

If that sounds harsh, that's sad, but in open source the key to success is to ensure you don't get your head too far up your own ass. If you harbour illusions that it is okay to be a primadonna I have a friend that did a lot of this type of work for one of the world's most popular open source projects who might be able to coach you a bit on what it takes to reach large audiences. Zephyr is still a project that addresses a tiny audience. Stop being too full of yourself.

marc-hb commented 5 years ago

The reason: 1) clone a project, 2) type pio run, 3) done

"Done", really? How about applying a security or bug fix without upgrading everything? Using a different, vendor toolchain?

marc-hb commented 5 years ago

you may want to go out into the field, do some user research and figure out what developers have to go through to get an environment set up by quietly observing them.

This is of course on the table and I hoped and literally asked you above to contribute.

your firmly held belief is that it's not a problem. ... It isn't a good idea to call them idiots.

I don't know who has the firmest held beliefs but I know who is writing without reading. One of the most amusing beliefs is that I'm more than just a Zephyr luser.

... idiots ... up your ass ... harbour illusions... primadonna ... coach you a bit ... tiny audience ... full of yourself

Wow, having a bad day? If not then sorry but I'll unfortunately have to pass on your coaching offer; for non-technical reasons. There seems to be very different ways to do open-source. But thanks! It's been a pleasure.

ivankravets commented 5 years ago

How about applying a security or bug fix without upgrading everything?

This is not a problem of end developer. PlatformIO has decentralized architecture and in theory, semiconductor companies and SDK provides should be responsible for maintaining development platforms. This is actually where we move now. PlatformIO provides interfaces for both:

For example, Espressif team personally maintains integration with PlatformIO. So, their customers can use not only stable versions of their SDKs, but also staging => http://docs.platformio.org/en/latest/platforms/espressif32.html#using-arduino-framework-with-staging-version

Using a different, vendor toolchain?

Do you have any issues with that?

marc-hb commented 5 years ago

Do you have any issues with that?

No, I'm just asking you to detail and confirm how awesome is PlatformIO. I thought it was clear I don't know it.

ivankravets commented 5 years ago

No, I'm just asking you to detail and confirm how awesome is PlatformIO. I thought it was clear I don't know it.

Thanks! Currently, we use GCC in all open source development platforms. We would like to an option with IAR or ARM compiler. The problem here that these companies don't allow 3rd parties to redistribute their tools. So, the only solution is to pay them for their IDE even if you don't use it and later override in PlatformIO Buil System toolchain binaries.

We had a similar issue with Segger J-Link. Previously, developers manually installed their GDB server and did configuration. However, a few months ago we received permissions from Segger company and PlatformIO Package Manager automatically installs J-Link GDB Server and it is very easy to do 1-click debugging or programming. No need to worry with extra steps or if something is not well configured.

I hope we will see official support for IAR and ARM compiler within PlatformIO soon.

Sorry for these issues.

jcw commented 5 years ago

(I was merely trying to say that convenience sells - even with OSS, where the currency is eyeballs)

ntavish commented 5 years ago

Hi, is there anyone working on this at the moment?

ivankravets commented 4 years ago

Hi, is there anyone working on this at the moment?

Yes, something great is coming in the next days 😊

nashif commented 4 years ago

Yes, something great is coming in the next days

keep us posted :)

ivankravets commented 4 years ago

Hey all,

We have a piece of good news. We've finally added support for Zephyr RTOS and would be glad to get some feedback. It was the most complicated integration for the last 6 years :) We even did some changes to our PlatformIO Core Build API. So, the only latest development version of PlatformIO Core will be compatible with Zephyr. All our efforts were directed on keeping the same workflow which we have with other frameworks (ARM mbed, ESP-IDF, Arduino, etc.). It means that we are still fully independent from operating system and IDE. No need to install any software, make, cmake, ninja, toolchains, different Python packages, other system-related tools.

Integration

We did pure native integration and PlatformIO build system will be used. It's based on top of SCons and is super fast and smart. You can use Python language to control and extend the build process using Advanced Scripting or quick shortcuts with build_*** options for platformio.ini.

The source code is here https://github.com/platformio/zephyr/tree/v2.0-branch/scripts/platformio It actually parses Zephyr CMake configuration and constructs PlatformIO build environment. This is a good solution because of no need to do any changes to default Zephyr tools, scripts, and source code.

P.S: Thanks, @valeros, for the great professional work on this! 👍

Supported IDE and OS

You have a choice between 10+ different IDEs and all popular operating systems (Windows, macOS, Linux 32/64, Linux ARMv6+). Any combination of IDE+OS provides the same reproducibility with ZERO changes.

Supported dev/platforms

Currently, 3 development platforms are Zephyr-enabled:

Support for other dev/platforms and boards will be added next week.

How to try it?

  1. You need to have the latest development version of PlatformIO Core. If you use CLI, just run pio upgrade --dev. Or, open PlatformIO IDE for VSCode, click on "terminal icon" on the bottom toolbar and run the same command pio upgrade --dev. It should be 4.1.1-b1 or above.
  2. You will need to use a development version of dev/platform mentioned above.

Project Examples

See projects with zephyr- prefix:

You can open/import any project and try it. The only change which you need to do is switching platform to development version:


[env:nrf52_board]
platform = https://github.com/platformio/platform-nordicnrf52.git
framework = zephyr
board = ...

[env:stm32_board]
platform = https://github.com/platformio/platform-ststm32.git
framework = zephyr
board = ...

[env:sifive_board]
platform = https://github.com/platformio/platform-sifive.git
framework = zephyr
board = ...

Configuration

We will provide soon extra information in our documentation on how to configure Zephyr RTOS within the PlatformIO project. In short, you can use any example from Zephyr repository with small changes:

  1. Create zephyr folder in the root of PlatformIO project and move here all related Zephyr's files (CMakeLists.txt, prj.conf, sample.yaml)
  2. Edit relative source paths in CMakeLists.txt, see example here https://github.com/platformio/platform-nordicnrf52/blob/develop/examples/zephyr-ble-beacon/zephyr/CMakeLists.txt#L7

The rest configuration should be done in CMakeLists.txt. It's the same when you use the official Zephyr build tools.

Demo

Below are a few demos from different IDEs with PIO Unified Debugger and SiFive HiFive1 Rev B board:

PlatformIO IDE for VSCode

Screen Shot 2019-11-17 at 00 31 44

CLion with PlatformIO plugin

Screen Shot 2019-11-17 at 00 27 34

Old-school Eclipse

Screen Shot 2019-11-17 at 00 43 04

Your feedback is very important to us.

ntavish commented 4 years ago

This is great! I'll test with https://docs.electronut.in/ (nRF52840)

marc-hb commented 4 years ago

It was the most complicated integration for the last 6 years :) We even did some changes to our PlatformIO Core Build API.

Can you summarize what differences made it the "most complicated for the last 6 years" and why it was easier to change some PIO API(s) than fix Zephyr directly? Thanks!

me21 commented 4 years ago

Platform ST STM32 is mentioned twice in the list of supported platforms. One link is to PlatformIO site, one to GitHub repo.

martinjaeger commented 4 years ago

Thanks a lot for your great work, @valeros and @ivankravets. It's really awesome that Zephyr support is now there.

I tested it in a project that's currently being migrated from mbed to zephyr, so it has files for both frameworks in the src directory (might be a bit confusing). My tests for the Zephyr integration can be found in the following branch: https://github.com/LibreSolar/bms-firmware/tree/pio-zephyr

As I have custom board definitions for the Zephyr project, I generated the .json file needed by platformio in boards subdirectory in addition to the Zephyr board definition. This worked very well.

Below some findings and issues I could not solve yet:

1. Libary dependencies

It seems like PlatformIO performs some checks in the Zephyr Makefile before installing library dependencies. In my case, these tests fail if the external library ThingSet was not yet downloaded. I might be doing it wrong, but I added the files from .pio/libdeps/my-lib as include_directories and target_sources to CMakeLists.txt.

Solution: Build once using mbed (different pio env) and include the libraries downloaded during mbed build.

I'm sure there is a more elegant way. Is it possible to integrate files from libraries automatically to the build without adding it to CMakeLists.txt?

2. Newlib nano

It seems like PlatformIO uses newlib nano as default, but Zephyr integrates other libc files in addition to that. I got following compilation error: 

Compiling .pio/build/bms-8s50-ic/zephyr/arch__arm__core__cortex_m/cortex_m/reset.S.o
arm-none-eabi-gcc: fatal error: /home/martin/.platformio/packages/toolchain-gccarmnoneeabi/bin/../lib/gcc/arm-none-eabi/8.2.1/../../../../arm-none-eabi/lib/nano.specs: attempt to rename spec 'link' to already defined spec 'nano_link'
compilation terminated.

Tried to deactivate LIBC in my CMakeLists.txt, but it didn't make a difference.

A solution was to add build_unflags = -specs=nano.specs to platformio.ini.

Now it compiles, but gives linker errors that e.g. printf and malloc are not found.

3. Compiler warnings

I got compiler warnings "cc1plus: warning: command line option '-Wno-pointer-sign' is valid for C/ObjC but not for C++". Adding also -Wno-pointer-sign to build_unflags solved this.

Instructions to reproduce described issues:

Build environment bms-8s50-ic in branch linked above.

ivankravets commented 4 years ago

Can you summarize what differences made it the "most complicated for the last 6 years" and why it was easier to change some PIO API(s) than fix Zephyr directly? Thanks!

@marc-hb Yes, I can. Actually, there is no problem with Zephyr. This was the first CMake-based integration for us. We support over 20 different frameworks and even if they use Cmake, we just do native integration using a build script with hardcoded source paths, flags, and options. The Zephyr is a more complicated project. It's not about a few source files or a 1-stage build process (which is classic). It is based on a 2-stage build process where we build pre-firmware and later using it generate other source files and build the final firmware. Again, no problem here, just an interesting use case for us. Here is a source code how does it look with SCons.

We didn't want to do "hard-coded" integration which very often requires significant modification between framework minor releases. In a case with Zephyr and CMake we decided to use CMake File API. It's an excellent solution for us which gives us a clear picture of the build files and flags related to them. However, there is a big problem here. Cmake does not provide information about custom targets declared via add_custom_target.

This took us some time. We reverse-engineered ninja build process and replicated the same with SCons. This is not a good idea but is not a bad as a temporary solution. Maybe it makes sense to contact @Kitware CMake team and ask them if they can add information about custom targets to File API.

Another solution is to have an intermediary "protocol/format" between Zephyr and PlatformIO where we will know more about custom targets and how to run them.

P.S: I'm not an electronic or an embedded engineer. I came to "embedded" from the Web world. This is actually why I founded PlatformIO. 6 years ago I was shocked at how is complex the Embedded world. That is why I tried to fix it a little bit. Today, we have PlatformIO. What do I mean here...

Can someone explain to me, why do we need YET ANOTHER build system with YET ANOTHER syntax and interpreter? I understand that embedded engineers like complex things (make, cmake, ninja, etc, the same philosophy with declared nodes and trees). How about other developers? I see a lot of complaints about CMake and their syntax. They even do not allow to change CMakeList.txt to something another. Why? We have CLIon IDE which depends on CMakeList.txt and Zephyr which also uses it via CMake. As a workaround, we moved Zephyr's configuration files to zephyr folder in the PlatformIO project.

Now I understand why SCons is still popular. Over 15 years! Very powerful API, super fast, no need to learn yet another interpreter! Combines generator and build system. No extra dependencies to make, cmake or system operating software. Just amazing Python! Python is very easy for using. A lot of over users, who are high-level developers (Java, C#, Python, JS), do interesting things with PlatformIO Advanced Scripting. They don't learn "yet another complex syntax, they just use PlatformIO, resolve own tasks and have MVP in the days, not weeks or months.

No need to go so far, even Zephyr uses Python internally and CMake is used just like a caller.

P.S.S: This just my own opinion and I'm not sure that it has any value. I don't like complex things. I used a lot of them for the last 15 years. The success is in simplicity, in approach where we can save time and other valuable resources (money, efforts, nerves, etc.). This is a small part of why PlatformIO is popular today.

I don't know all the technical details about Zephyr and how it is cool. What I actually like is its unified approach to everyone, transparency, a truly open source where everyone can contribute (the same Linux philosophy which has great adoption). This is actually what we miss today. Mbed, FreeRTOS, etc are fully proprietary and try to promote their products/services. They never think about developers/customers.

We would be glad to help Zephyr Project with more simplified approach to how to use its power and build amazing products in a short time using any OS, IDE, and do not worry with Cmake, Ninja, and other system-related things. If developers are limited with PlatformIO, they can easily migrate to native tools including "west-cli" or extend functionality using CMake declaration (we support this way, it the same for us and "west").

marc-hb commented 4 years ago

Thanks @ivankravets for sharing the additional details, experience and perspective, really appreciated.

Can someone explain to me, why do we need YET ANOTHER build system with YET ANOTHER syntax and interpreter?

Because they all suck. From https://mesonbuild.com/Design-rationale.html (and others)

A software developer's most important tool is the editor. If you talk to coders about the editors they use, you are usually met with massive enthusiasm and praise. [...] The second most important tool, even more important than the compiler, is the build system. Those are pretty much universally despised.

But the main issue with build systems is actually... C.

Even though build systems are "universally despised", language designers eventually realized their importance and many recent languages provide a build system "for free". However it was too late for C.

The "translation unit" is another, old age problem unique to C. It forces every build system that want to support C well to care about weird complexities like precompiled headers or LTO.

Last and main build issue with C: fragmentation. Nothing comes close. Partly because of its age again, mainly because it's low-level. Because it's low-level, C generally can't rely on a layer of indirection abstracting portability issues away like most other languages can. In fact C is typically central to that abstraction layer itself.

https://dwheeler.com/autotools/

Note, however, that the autotools don't just build software, they detect various features to make it much easier to build software that ports to a wide variety of systems. Sure, you should use standard interfaces, but sometimes there's no standard interface or the interface has serious bugs/limitations. [...] Some of the autotools issues are fundamental to their goals. Their goal is to make it easy to build a system, using the native tools, and there are variations in the native tools. In the longer term, forcing native tools to get fixed (so that people don’t have to work around their bugs) and getting their functionality updated (so that people don’t have to work around their limitations) would help best in the long term.

From https://mesonbuild.com/Design-rationale.html again:

Must not add complexity due to obsolete platforms

https://youtu.be/7THzO-D0ta4?t=1465 (slides unfortunately missing from https://github.com/CppCon/CppCon2017/tree/master/Presentations )

ivankravets commented 4 years ago

@marc-hb Thanks for the answer. I do not have so much experience with CMake. What is the reason that people do business logic using normal programming languages? For example, you use Python, Greenwaves (GAP) also use Python. So, CMake is again useless and a limited tool that can actually be used as a declarative instrument for code building? You can't resolve here business tasks? I still do not understand its benefit over make or ninja for the which CMake generates project. Is the main reason just to use it as a unified declarative bridge to other real build systems (make/ninja)?

carlescufi commented 4 years ago

@ivankravets

This took us some time. We reverse-engineered ninja build process and replicated the same with SCons. This is not a good idea but is not a bad as a temporary solution. Maybe it makes sense to contact @Kitware CMake team and ask them if they can add information about custom targets to File API.

Both ninja and CMake are open source, so you shouldn't need to reverse engineer, the source code is available for you to inspect. Regarding changes to the File API, that's also something that anyone can submit a patch for.

ivankravets commented 4 years ago

@carlescufi "We reverse-engineered ninja build process", I mean "build process", not ninja engine.

Regarding changes to the File API, that's also something that anyone can submit a patch for.

Good, we hope someone will make it. We can actually leave the current implementation with hard-coded SCons hooks for CMake's "add_custom_target" and gather more feedback.

/cc @bradking , @billhoffman

marc-hb commented 4 years ago

I'm not sure what you mean by "business logic". This answer maybe? https://mesonbuild.com/FAQ.html#why-is-meson-not-just-a-python-module-so-i-could-code-my-build-setup-in-python

The other question is easy to answer:

I still do not understand its benefit over make or ninja for the which CMake generates project.

https://ninja-build.org/manual.html#_philosophical_overview

Where other build systems are high-level languages, Ninja aims to be an assembler...

Build systems have two very distinct functions and phases: 1. System discovery and system-specific, once-off build configuration. 2. Dependency graph, recurring incremental build. Ninja focuses on 2 and 2 only. Unless something went wrong, humans never look at ninja code.

Make can do 1 only in the simplest cases, so autotools and CMake only use it for 2.

ivankravets commented 4 years ago

I see, thanks! Now I understand why you need these 2 tools. Also, understand very well why we don't use other tools and only SCons. It seems it covers 1, 2 and gives developers freedom where they can do what they want using Python language.

This is actually what we did. We do not use SCons as a build system via their SConscript mechanism. Instead of it, we just build our own build system for our needs reusing SCons Build API which internally resolves 1+2 which you described above.

Could you try this PlatformIO + Zephyr Integration? I would be thankful for your feedback. I see you are so experienced in these build process.

valeros commented 4 years ago

Hi @ martinjaeger! Thanks for the report!

It seems like PlatformIO performs some checks in the Zephyr Makefile before installing library dependencies. In my case, these tests fail if the external library ThingSet was not yet downloaded. I might be doing it wrong, but I added the files from .pio/libdeps/my-lib as include_directories and target_sources to CMakeLists.txt.

Indeed, CMake is launched before resolving dependencies and therefore if you specify library sources in CMakeLists.txt file CMake won't be able to find them. You'd better specify only sources that are located in src folder and then libraries will be added automatically.

It seems like PlatformIO uses newlib nano as default, but Zephyr integrates other libc files in addition to that. I got following compilation error:

Please run pio update, it should be fixed in the latest zephyr package.

I got compiler warnings "cc1plus: warning: command line option '-Wno-pointer-sign' is valid for C/ObjC but not for C++". Adding also -Wno-pointer-sign to build_unflags solved this.

That's also should be fixed in the latest zephyr package.

martinjaeger commented 4 years ago

Thanks! Tried it out, but building fails immediately with following error message:

Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/ststm32/bms8s50ic_l452cx.html
PLATFORM: ST STM32 5.7.0 #fa55b07 > Libre Solar BMS8S50IC L452Cx
HARDWARE: STM32L452RET6 80MHz, 64KB RAM, 256KB Flash
DEBUG: Current (stlink) On-board (stlink) External (blackmagic, jlink)
PACKAGES: toolchain-gccarmnoneeabi 1.80201.181220 (8.2.1), framework-zephyr 1.20000.191120 (2.0.0), framework-zephyr-hal-ststm32 0.0.190619, tool-cmake 3.15.5, tool-dtc 1.4.7, tool-ninja 1.7.1, tool-gperf 3.0.4
Reading CMake configuration...
KeyError: 'framework-zephyr-hal-stm32':
  File "/home/martin/.platformio/penv/lib/python3.7/site-packages/platformio/builder/main.py", line 151:
    env.SConscript("$BUILD_SCRIPT")
  File "/home/martin/.platformio/packages/tool-scons/script/../engine/SCons/Script/SConscript.py", line 605:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/martin/.platformio/packages/tool-scons/script/../engine/SCons/Script/SConscript.py", line 286:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/martin/.platformio/platforms/ststm32@src-15e0e87f41a00aa2f9602f71367ac29c/builder/main.py", line 98:
    target_elf = env.BuildProgram()
  File "/home/martin/.platformio/packages/tool-scons/script/../engine/SCons/Environment.py", line 224:
    return self.method(*nargs, **kwargs)
  File "/home/martin/.platformio/packages/framework-zephyr/scripts/platformio/platformio-build-pre.py", line 23:
    env.ProcessProgramDeps()
  File "/home/martin/.platformio/packages/tool-scons/script/../engine/SCons/Environment.py", line 224:
    return self.method(*nargs, **kwargs)
  File "/home/martin/.platformio/penv/lib/python3.7/site-packages/platformio/builder/tools/platformio.py", line 107:
    env.BuildFrameworks(env.get("PIOFRAMEWORK"))
  File "/home/martin/.platformio/packages/tool-scons/script/../engine/SCons/Environment.py", line 224:
    return self.method(*nargs, **kwargs)
  File "/home/martin/.platformio/penv/lib/python3.7/site-packages/platformio/builder/tools/platformio.py", line 328:
    SConscript(env.GetFrameworkScript(f), exports="env")
  File "/home/martin/.platformio/packages/tool-scons/script/../engine/SCons/Script/SConscript.py", line 668:
    return method(*args, **kw)
  File "/home/martin/.platformio/packages/tool-scons/script/../engine/SCons/Script/SConscript.py", line 605:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/martin/.platformio/packages/tool-scons/script/../engine/SCons/Script/SConscript.py", line 286:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/martin/.platformio/platforms/ststm32@src-15e0e87f41a00aa2f9602f71367ac29c/builder/frameworks/zephyr.py", line 30:
    "platformio", "platformio-build.py"), exports="env")
  File "/home/martin/.platformio/packages/tool-scons/script/../engine/SCons/Script/SConscript.py", line 668:
    return method(*args, **kw)
  File "/home/martin/.platformio/packages/tool-scons/script/../engine/SCons/Script/SConscript.py", line 605:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/martin/.platformio/packages/tool-scons/script/../engine/SCons/Script/SConscript.py", line 286:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/martin/.platformio/packages/framework-zephyr/scripts/platformio/platformio-build.py", line 746:
    codemodel = get_cmake_code_model()
  File "/home/martin/.platformio/packages/framework-zephyr/scripts/platformio/platformio-build.py", line 206:
    run_cmake()
  File "/home/martin/.platformio/packages/framework-zephyr/scripts/platformio/platformio-build.py", line 179:
    "framework-zephyr-hal-" + PLATFORMS_WITH_EXTERNAL_HAL[platform_name]),
  File "/home/martin/.platformio/penv/lib/python3.7/site-packages/platformio/managers/platform.py", line 345:
    version = self.packages[name].get("version", "")

I saw during update (assuming you meant pio update and not pio run update) that some packages including above mentioned framework-zephyr-hal-stm32 are "Detached". Is this expected?

valeros commented 4 years ago

@martinjaeger Please run pio update again to pull the latest changes of ststm32 platform. Thanks!

martinjaeger commented 4 years ago

Great, it compiles and is linked fine now :)

Question for my understanding: Do you link against newlib-nano by default? If I keep the previously added build_unflags for newlib-nano, the generated binary is larger than with the default settings. If yes, is this only valid for zephyr or also for other frameworks? I'm asking as we are using this custom script as a workaround to enable newlib-nano for mbed.

valeros commented 4 years ago

@martinjaeger We use flags based on CMake project configuration that CMake generates for us. If newlib-nano is enabled in Zephyr project then it will be used in PlatformIO project as well. It's only valid for Zephyr, so other frameworks are responsible for their own flags.

n3rd4i commented 4 years ago

Since now there is [...]\.platformio\packages\framework-zephyr as ZEPHYR_BASE, the fact that ZephyrProject needs what they call west-installation, only having framework-zephyr is not enough since there is an west init -l framework-zephyr and then west update need before being able to properly use it from PlatformIO, details here, building a project which uses zephyr modules, spoiler: part1:

[...] fatal error: stat_mgmt/stat_mgmt.h: No such file or directory
 #include "stat_mgmt/stat_mgmt.h"
          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
Compiling .pio\build\development\[...]]

part2:

framework-zephyr\include/mgmt/buf.h:11:10: fatal error: cbor_encoder_writer.h: No such file or directory

*****************************************************************************
* Looking for cbor_encoder_writer.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:cbor_encoder_writer.h"
* Web  > https://platformio.org/lib/search?query=header:cbor_encoder_writer.h
*
*****************************************************************************

 #include "cbor_encoder_writer.h"
          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

Folder structure needs to be something like:

.west\
modules\
tools\
zephyr\
  1. How will this be handled, since currently only framework-zephyr exists (efectively ZEPHYR_BASE without west installation around it)?
  2. Additionally assuming that one wants to use different than original zephyr remote URL, could that be configurable from within PlatformIO or? e.g.: west init -m https://github.com/zephyrproject-rtos/zephyr --mr v1.15.0
martinjaeger commented 4 years ago

Zephyr can be built/used without west as far as I know, see also here. I guess PlatformIO does what west would normally do in the background.

n3rd4i commented 4 years ago

So then how do I update the zephyr "modules" which are needed for my project to compile in PlatformIO?

As previously stated, this was done by simply doing: west init, west update, then I could build my project normally.

n3rd4i commented 4 years ago

This kind of works, there are some open-questions that I've found so far:

ivankravets commented 4 years ago

could the version of the zephyr platform be specified, say v2.0.1-rc.1 ?

There is no difference for PlatformIO Zephyr or Mbed. They are frameworks and are packaged with versions. If there is a pre-built package in our CDN, you can use it with http://docs.platformio.org/en/latest/projectconf/section_env_platform.html#platform-packages

Please note that we don't support beta software. Only, stable releases. If merge our scripts with Zephyr main code, it will be possible and you could use as:

[env:blackpill_f103c8]
platform = ststm32
framework = zephyr
board = blackpill_f103c8
platform_packages =
  framework-zephyr @ https://github.com/zephyrproject-rtos/zephyr.git#v2.0.1-rc.1
ivankravets commented 4 years ago

So then how do I update the zephyr "modules" which are needed for my project to compile in PlatformIO?

We work on adding support for this. @valeros will back here soon with updates.

n3rd4i commented 4 years ago

platform_packages = framework-zephyr @ https://github.com/zephyrproject-rtos/zephyr.git#v2.0.1-rc.1

Question about the platform_packages url type, both of the following are not currently supported? e.g.:

This could be useful to use the git installed [ssh] keys, instead of providing user/password combination. Will this be a feature in the future?

LMESTM commented 4 years ago

@ivankravets I have already Zephyr native environment up and running on my windows machine. Now I'm testing this new Zephyr integration into platformIO with VScode plugin (BTW thanks a lot for trying to make our life easier !).

When trying to build the zephyr example project, I run into this: 

PackageManager: Installing tool-ninja @ ~1.7.0
Error: Could not find a version that satisfies the requirement '~1.7.0' for your system 'windows_amd64'
The terminal process terminated with exit code: 1

Even though ninja is already supported. From a PIO terminal:

$ ninja --version
1.9.0

Any idea what I have missed ?

valeros commented 4 years ago

Hi @LMESTM! That's probably a system-wide program. I assume you're using ststm32 platform? Could you please try to update your platform via pio update? Thanks!

LMESTM commented 4 years ago

@valeros

I assume you're using ststm32 platform?

yes indeed

Could you please try to update your platform via pio update? Thanks!

Done and I confirm this worked fine ! I thought I had done this already previously but well ... ! Thank you :-)