search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.29k stars 985 forks source link

suggestions for extra compiler / arch settings #86

Closed weatherhead99 closed 6 years ago

weatherhead99 commented 8 years ago

Hi, I started using conan last week to package some of our internal stuff, and am really enjoying it so far!

A few of things have popped up for me, both relating to the values available for compiler & arch settings. Both of these might be solved by some hierarchy of settings.

1) when using LLVM/clang, it is possible to link against different standard libraries. This is basically only really a problem (I believe) with c++ builds on Linux, where builds may be against libc++ or libstdc++ and both are valid choices. In princple though, this functionality can be used on other platforms. A build of e.g. "macos" + "libstdc++" doesn't make sense.

2) having one arch called "arm" really isn't enough for any serious ARM work. For hosted ARM environments, I'd suggest at least armv6, armv7, armv7hf and armv8 would be needed.

3)In the non-hosted (bare metal ) ARM environments on the other hand (which I'm actually building some conan packages for one of our projects here), basically there are so many options that it seems only sensible to have many, many settings... not sure what the best idea is here. Probably a generic option something like "targetboard", where the exact target system is specified as a string.

4) again on ARM, there are many compilers (only the popular ones: gcc, clang, iar, keil), but for example it doesn't make sense for a package to be "x86" + "keil", because that compiler is only for ARM. On x86, the intel compiler (icc) is probably popular enough to be included in the compiler possibilities.

conan is of course already flexible enough that I can implement all this in house here for our packages - but if there is any thought to include any more of these distinctions upstream then I would of course rather use those conventions.

Thanks very much for all your work, DW

memsharded commented 8 years ago

Hi @weatherhead99 , Thanks for using conan and your feedback!

The current version of "settings.yml", you can see it in https://github.com/conan-io/conan/blob/develop/conans/client/conf/__init__.py or in your home ~/.conan/settings.yml, is just a proposal for our users, as it is important to have a consensus about typical "names" for compilers, versions, etc. So we are willing to accept proposals about those settings from users that actually use often those settings and think they are common enough to be useful for other users.

About the proposal for ARM, I agree, but possible we can take advantage that all the settings can be nested, like:

arm: {version: [v6, v7, v7hf, v8]}

About the bare-metal, I am really interested in the use case scenarios. Would you be interested in storing pre-compiled packages for a specific board? All embedded dev tools I know (bu I am not an expert) build libs from source. Building from source is not a problem, the problems obviously come with the compatibility of binary packages. The targetboard might not solve the problem, maybe it is better to add a file with hundreds of boards to ensure everybody refer to the boards the same, and let the community contribute to that file with hundreds of boards.

About excluding non-sense configurations, it is quite hard to "declare" it with conan right now in a general case. But on the other hand it is very simple to explicitely restrict compilers, architectures or combinations in a certain package. You can check: http://docs.conan.io/en/latest/reference/conanfile.html#settings It is also possible to exclude certain combinations, just with python logic in the config() method, for the package users convenience.

We initially thought to include the architecture "arch" as a sub-property of the "compiler", similarly to the "version" one, but we finally decided it was general enough to be on its own. Maybe it was a mistake; we can always add new "arch" properties inside "compiler", it is not incompatible (and it would be a necessary step to maintain backwards compatibility with all existing packages), and deprecate the top level "arch" in the future.

To summarize: yes, there is thought to include these distinctions upstream. We should be careful to try not breaking existing packages, but yes, we are willing to evolve the settings.yml upstream definition. From your issue, I'd say that the needed settings are:

The libstd++/libc++ is actually a problem not enough specified by conan, probably deserves its own issue. What do you think?

And I'd say that incompatible/non-sense configurations as x86+keil are not really a problem, and it is responsibility of users to provide correct settings. In the worst case, the conanfile.py can explicitely restrict settings or raise errors for non-sensical combination of settings.

I'd need more info about the embedded "targetboard" idea. Is it possible to have a list of most boards? Are binaries really used?

Thanks a lot to you for using conan and providing such feedback, it is really useful and necessary to improve!

weatherhead99 commented 8 years ago

@memsharded, thanks very much for the detailed reply, I'll try to be brief about each point to avoid anyone having to read too much!

In everything I originally suggested, I do think that the libstdc++/libc++ thing is as you say a genuine problem and some new issue needs to be opened. In our setup here, the linux builds are mostly clang & libstdc++ (i.e. not the default), and it can cause quite a few issues.

Agree the nonsense configs aren't really an issue as the package would be unbuildable anyway pretty much.

The arm arch idea is pretty much exactly what I'm using here right now, but as it happens we only currently build stuff for v7 and v7hf. So, I agree this is probably the best way to go. Though, incidentally, if the plan is to "nest" these arm architectures, for consistency, the same might be said about x86.

About the ARM bare-metal, it's true that a most times developers will build everything from source, but there are certainly libraries that are distributed closed source as pre-built archives, and I don't know about many setups but conan is proving useful here to build platform libraries and things which people can share. The problem is, there are just so many possible compiler options to the compiler that fundamentally change the build (e.g. no-exceptions, no-stack-protect, -mthumb, -mcpu=cortex-m3 and similar) that it's probably not worth trying to support. Maybe once I have got a bit further through using conan in this case the best idea is just to write some documentation about it. I think it's unlikely that many people will share libraries for anything but the most popular dev boards. That said, there is a sort of similar project to conan from the folks at ARM mbed (http://yottadocs.mbed.com/), and possibly the Arduino community might be interested in using something similar.

TLDR: it's probably the case that anyone actually wanting to pass around pre-compiled bare metal libraries will want to do it all from scratch on their own, conan is already very useful already, maybe some support for the huge community boards like arduino, mbed etc might be nice.

As a side issue, things like no-exceptions, no-rtti, etc, may be common enough to include as possibilities in the "gold master" settings.yml.

Thanks, DW

memsharded commented 8 years ago

Hi again!

I have just submitted a PR with the arm changes, as you said, to be consistent it would require to nest x86, and that setting is already heavily used, so I will attach to your initial proposal of armv6, etc.

I've had a look at iar, keil compilers, to check for a reasonable default configuration for settings of versions, archs, etc, but not clear to me. If you have any specific suggestion for including those in our default settings.yml, that could be great.

Also, I have created a separate issue for libstdc++ management https://github.com/conan-io/conan/issues/124

Thanks a lot

nlbutts commented 8 years ago

I am also looking for a way to specify build options for bare-metal ARM systems. From this thread, it sounds like that feature is not currently supported. But it seems to alude to the fact that I could define my own os, arch, compiler in the settings.yml file. Is that correct?

memsharded commented 8 years ago

Yes, the file ~/.conan/settings.yml is the one containing the "default" settings. You can edit it, add as many configurations (also remove), compilers and architectures as you want, basically they are some nested dictionaries with lists of possible values. The only known current restriction is that it doesn't allow "any" values, or not having to specify a set of possible values for a certain setting.

As long as you and your collaborators use the same settings.yml, everything should work for you. For the rest of the users, it could be nice if you contribute your settings, to try to build some community consensus about them. There is an issue where you can enter such discussion: https://github.com/conan-io/conan/issues/86

astralien3000 commented 8 years ago

Hello !

I was discussing in issue #160, and found out this issue was more relevant for my interest. I am interested into microcontroller programming.

First of all, let's try to add some data to the discussion. The microcontroller world is a big mess, so that their classification is hard. I don't know every family, so, I will mainly talk about the ones I know. You can :

In the other way, you need to enumerate :

If you are under an operating system, It seems to make sense to separate by architecture and version, as you are doing now.

But for microcontrollers, it makes more sense to be able to specify the exact hardware, and even the board.

I don't know if it is relevant to gather both in the same settings name "arch". But I think they are kind of mutually exclusive (since the microcontroller or the board should tell you the architecture).

weatherhead99 commented 8 years ago

As far as microcontroller builds using conan go, i am doing this internally on about 5 different ARM chips at the moment. Note that when running bare-metal (without runtime), it is likely that all code is statically linked. So, actually, you generally don't need to know the board, because likely you are adding the linker script/bsp/startup vector code at the last stage of linking the executable code.

I think (if we're only talking about "library" type code) it's sufficient to know e.g. ARMv7/cortex-m3/thumb because that defines the important compiler options.

The situation is different if you're running on e.g. uclinux. I don't have so much experience in that area, so don't really know what the complications would be.

memsharded commented 8 years ago

I have played with arduinos and RPI, and some other micros before, but not such an expert. I wonder, would it be enough to enumerate by board? would it be a desired way to do it? Even if we have many more settings, but all of them grouped in a single setting, rather than several more complex settings (as with nested versions). I am thinking if the Arduino Uno is always exactly built with the same Microcontroller, and the other boards as well. If that holds, specifying the boards will be fine. Also the opposite must hold, would someone would like to build the same "library" for many different boards, and such library will be the same for all boards? So they dont have to build/create a different binary for each board.

astralien3000 commented 8 years ago

@weatherhead99 I agree that a lot of libraries are independent of the board, but a lot of others libraries (generally OS-like, as you said) need to know the board.

And about knowing ARMv7/cortex-m3/thumb, I think it is not enough. Since you are on a microcontroller, you need some drivers for the hardware modules. The libraries will not be able to provide any driver without the microcontroller reference.

@memsharded Indeed, the board is enough information for any library. It gives the microcontroller, as well as the architecture, mcu family, vendor, etc... And it also give enough information for an OS-like library to give standards drivers, etc...

But the problem is that only being able to specify the board is not sufficent since there is a lot (I would say, the majority) of microcontoller that are not used on a "well-known" board.

memsharded commented 8 years ago

Well, about the discussion of knowing just the architecture for some libraries, it is true, if it is a pure "logic" library, it is enough to know the architecture. And in the general case, other libraries need to know the compiler. But this is not a problem at all, the package creator chooses. If the package is board/micro-independent, then, they will just check the architecture for the cross-build.

So, from what I understand, then it would make more sense to:

  1. Add more architectures to the current "arch" setting. This is not a problem, and architecture number are not as large as micros
  2. Add a microcontroller or "target" setting, but not board one, as the relation seem to be in that direction. Knowing the microcontroller is enough to build an application for any board in which they are used? I'd say yes, but please confirm.
  3. If it turns that the usability needs improvement, we could introduce something as settings "alias", where we could define mappings from boards to microcontrollers, and users could use that as input to settings too.
  4. The open issue would be the classification of the microcontrollers. I'd say that the constructor is not so relevant, I'd probably try some nested settings as:
target: { family1(ATMega):  {version: XX-1, XY-2...},
             family2(CortexM): {version: ZZ1, ZZ2....,
                                            other: val1, val2}
            ...
            }

Probably that would account for a large number of use cases, many build options would be decided on the family (also the arch is properly defined), and the exact microcontroller could be achieved for the build, just by composing target + target.version

In any case, I would propose to move this forward by doing. Why don't both of you propose something based on your current needs or current settings, and we start integrating and iterating it? If you pass me the info, I will manage to merge it and propose a conan PR. No problem if it is not perfect from the beginning, we will learn in the process.

Thanks very much for all this feedback, very valuable!

astralien3000 commented 8 years ago

Ok, I will try to do something.

I don't really agree about 2. Indeed, if the application supports multiple boards, the board information may be mandatory. Indeed, either the application uses a package that manage several boards (in this case, you need the board information) or the application manages itself the boards. In the second case, the management can be done by modifying a define in a configuration header (dirty way, but conan don't need to handle the board), or it can be done by passing some argument in the build chain (conan can either manage it or not, but I think it is better if it is managed by conan).

That being said, I think the system of alias is not really good, since I guess the final information given to the packages will be the micro and not the board. The problem is that several boards can have the same micro, and different boards versions can have different micro. The first way cannot be resolved. To resolve the second way you may just add the version to the board.

Just one question, I have searched the settings.yml file in the sources, it seems that it is given in this file, isn't it ?

weatherhead99 commented 8 years ago

I think that including information to the level of the board ( option 2) might be a limiting decision. I can't speak for other use cases of course, but the way I'm using conan is to build library code for ARM platforms. The individual firmware packages for each board are just built using a cmake toolchain, no conan integration at all, and I don't actually see a use for that in my situation - we are just outputting hex files from the build process which get flashed onto the boards. The real usefulness of conan is that library code which is common between boards can be packaged up and re-used. It's not that the compile time is arduous, it's simply that getting the compiler options exactly right can be difficult.

memsharded commented 8 years ago

Yes, that file right.

I see your points. Then I think that we might need both of them, and rely on settings management. This is a feature that allows a package to disregard a provided settings. So if your package works in a micro, but doesn't require board details at all, you could just declare that you don't need that setting and you will be fine. You will get the same binary for all boards with that micro.

memsharded commented 8 years ago

@weatherhead99 was faster than me! :) An example:

class ConanPackage(Conanfile):
      settings = "compiler", "arch",  "micro"

That will basically ignore board settings at all in that package, and it will build based on compiler, arch and microcontroller. All builds and consumers that provide a specific board -s board=ArduinoUno will get exactly the same binary package, as the board is not being hashed for that package. Does this make sense? Thanks!

astralien3000 commented 8 years ago

@memsharded Yes, it makes sense. I will try to do something this way.

@weatherhead99 I kind of see what is your use-case. You compiles general libraries for ARM microcontrollers with conan, and use a vendor library (STM32Cube, Kinetis, etc..) in your application to compile and link (with the provided linker script), right ? Why not considering the vendor library as a package ?

astralien3000 commented 8 years ago

Hey ! I just wanted you to know how yotta (an other package manager designed for microcontrollers) manages the cross compilation : http://yottadocs.mbed.com/tutorial/targets.html

What I propose in the light of this

salkinium commented 8 years ago

and then you can add new microcontrollers in target (in a flat way, no group)

Though it may be of advantage to do this in a hierarchical way, otherwise you end up with hundreds of targets from a vendor that only differ in flash/ram size and pin count.

There was some work done on yotta to reduce duplicate configuration and code in targets (Target Inheritance), but not finished.

astralien3000 commented 8 years ago

Yeah, I agree that having a hierarchical way is better to reduce duplicate (that is what I do with my toolchains), but I am not sure it is useful for the package system. Indeed, as a user, when I want to compile for a particular microcontroller, I just want to type the reference in the settings, and not decomposing it in arch/family/vendor/group/reference. It does not prevent the package developer to use a hierarchical representation, I think.

EDIT : One good idea may be to let the user specify the mcu reference (flat way), and conan could provide a hierarchical representation in a python module for conanfile.py.

Something like : 

 from conans import ConanFile, CMake, TargetHierarchy
...
htarget = TargetHierarchy(self.settings.target)
htarget.arch == "arm"
htarget.arm_family == "Cortex-M4"
htarget.vendor == "ST"
htarget.vendor_family == "STM32F4"
...

What do you think of that feature ?

salkinium commented 8 years ago

let the user specify the mcu reference (flat way)

👍

Since I've already been through this thought process before, here is my two cents. Note that I'm blissfully unaware how Conan works, so this may not be implementable.

There are two views to be balanced:

Taking the STM32F407VG as an example, it looks like something like this:

v- Build system view

Base (ARM Cortex-M)
^
|
Family (STM32)
^
|
Series (STM32F4)
^
|
Chip (STM32F407VG)
^
|
Board (STM32F4 Discovery)

^- User view

This is a pyramid of course, so it gets wider the lower you get. The whole point is to move common code and data as far up as possible. So for example, STM32F4 all have the same CPU core (ARM Cortex-M4), all have the same LSI and HSI clock frequencies (32kHz and 16MHz), have the same HAL peripheral implementations, etc…

The Chip target should ideally only consist out of configuration data, like what GPIO pins are available, which Peripherals are available. Take a look at the incredible similarities between certain STM chips of one series. The Board target would declare additional information such as renaming/aliasing the pins to the PCB silk screen pins, external connections like crystals/clocks, communication connections (like STLinkv2-1 serial) and external chips (Discovery boards usually have some MEMS sensors on them).

Looking forward, it is essential to distinguish between Chip and Board targets. The internal hierarchy of family and series is not strictly necessary, as they are "internal inheritance targets" anyways, but the Chip and Board targets is what users interact with. They are separated, so that users can build their own custom board and reuse the chip target for it.

Also note, that a board may declare a footprint, like an Arduino header, or in the case of the ST Nucleo boards, their Morpho header. These are not targets in the traditional sense, but protocols. If these were C++ classes (multi-inheritance), you'd inherit the ST Nucleo board target from both the chip target as well as the Arduino footprint target, as well as the Morpho footprint target. If these were Obj-C classes (single inheritance), you'd create a Protocol target for the Arduino and Morpho footprints and make the ST Nucleo board target "implement" those. (No clue how to make that work in Conan, btw).

Note that Shields are explicitly not a specialization of a board target, instead, they are unified with a board target that implements its footprint protocol. For the user this would somehow look like this: ST Nucleo + Arduino Shield. You could make this a new target that uses a robotic motor shield on a Nucleo board as a "robot base target". (Again, no clue how to make that work in Conan).

This is a relatively complex view on the matter, but I've looked at Freescale, ST, NXP and AVR targets and this is the most common and universal structure I could come up with that reduced both duplicated code as well as configuration data. Ideally you'd couple it with configuring/generating parts of the HAL, like xpcc does (just think about the pin alternate functions on STM32).

I haven't seen such a hierarchical approach to the matter anywhere before (except for Target Inheritance in yotta), so I'm sure there will be some unknowns to be solved.

astralien3000 commented 8 years ago

Ok, I see better your point. You want conan to have a hierarchical point of view to compile as few versions of a package as possible. A package that is common to all Cortex-M4 would be compiled once instead of several times in the case of a flat setting. That would make a bigger task that I thought.

astralien3000 commented 8 years ago

/!\ WARNING /!\ very long post, sorry !

Hey ! I am back with some propositions that I tested. I can't do it in a PR because It would take 1 PR for each of my propositions.

@salkinium, about conan settings, (@memsharded, can tell if I'm wrong) the aim is to recompile the package when a setting changed. For example, if you compiled the SDL package with settings.compiler == Visual Studio, it need to be recompiled if you use settings.compiler == gcc. But the package can choose whether if a setting is relevant or not. For example, if a library package is header-only, it can say that it won't change because of the compiler, so that conan will not have to "build" it several times.

[1] Full hierarchical

Settings :

target:
    native:
    mcu:
        base: 
            cortex-m:
                family:
                    stm32:
                        series:
                            f4:
                                chip:
                                    [stm32f407vg]
                            l1:
                                chip:
                                    [stm32l151re]
            avr:
                family:
                    atmega:
                        series:
                            atmegaxx0_1:
                                chip:
                                    [atmega2560]
                            atmegax8:
                                chip:
                                    [atmega328p]

Example of install command : 

conan install -s compiler=arm-none-eabi-gcc -s compiler.version=5.2 -s target=mcu -s target.base=cortex-m -s target.base.family=stm32 -s target.base.family.series=f4 -s target.base.family.series.chip=stm32f407vg ..

As you can see, I did not added Boards, but you can add it as a leaf of chip.

Advantages :

Drawbacks:

[2] Hierarchical, flat settings

Settings : 

target: [native, mcu]
base: [None, cortex-m, avr]
family: [None, stm32, atmega]
series: [None, stm32f4, stm32l1, atmegaxx0_1, atmegax8]
chip: [None, stm32f407vg, stm32l151re, atmega2560, atmega328p]
board: [None, "STM32F4DISCOVERY", "Arduino Uno", "Arduino Mega"]

Example of install command : 

conan install -s compiler=arm-none-eabi-gcc -s compiler.version=5.2 -s target=mcu -s base=cortex-m -s family=stm32 -s series=stm32f4 -s chip=stm32f407vg -s board="STM32F4DISCOVERY" ..

Advantages :

Drawbacks :

[3] Mixed hierarchical

target: 
    native:
    mcu:
        base: [cortex-m, avr]
        family: [stm32, atmega]
        series: [stm32f4, stm32l1, atmegaxx0_1, atmegax8]
        chip: [stm32f407vg, stm32l151re, atmega2560, atmega328p]
        board: ["STM32F4DISCOVERY", "Arduino Uno", "Arduino Mega"]

Example of install command : 

conan install -s co"mpiler=arm-none-eabi-gcc -s compiler.version=5.2 -s target=mcu -s target.base=cortex-m -s target.family=stm32 -s target.series=stm32f4 -s target.chip=stm32f407vg -s target.board=STM32F4DISCOVERY ..

Advantages :

Drawbacks:

[4] Flat

target: [native, stm32f407vg, stm32l151re, atmega2560, atmega328p]
board: [None, "STM32F4DISCOVERY", "Arduino Uno", "Arduino Mega"]

Example of install command : 

conan install -s compiler=arm-none-eabi-gcc -s compiler.version=5.2 -s target=stm32f407vg -s board=STM32F4DISCOVERY ..

Advantages:

Drawbacks:

Conclusion

As you may imagine, there is a lot more mix of these solutions that has their advantages and drawbacks. If you have a better one, feel free to propose it. I tried to give solution that implies the lesser modification of conan, but maybe a deeper modification could be better ?

Anyway, my favourites solutions are [4] "Flat" and [3] "Mixed hierarchical" (or a mix of both ?).

EDIT: I added numbers to propositions to make identification easier.

astralien3000 commented 8 years ago

A new Idea ! (And my new favorite)

[5] Improved flat

Settings :

target:
    native:
    mcu:
        mcu: [
            stm32f407vg,
            stm32l151re,
            atmega2560,
            atmega328p,
        ]
    board:
        board: [
            "STM32F4-Discovery",
            "Arduino Mega",
            "Arduino Uno",
        ]

Example of install command : 

conan install -s target=native ..
conan install -s target=mcu -s target.mcu=atmega2560 ..
conan install -s target=board -s target.board="Arduino Mega" ..

Advantages:

Drawbacks:

I think it is the best proposition, since there is no pollution of the previous behavior (only target=native is added, which is not so much). The problem of trying to group chips into base/family/vendor/series is that it need big changes, and is really hard to manage.

memsharded commented 8 years ago

What an interesting discussion here, thanks very much @salkinium also for your contributions. Trying to update myself, while checking PR https://github.com/conan-io/conan/pull/333/files, thanks very much @astralien3000 for your good job.

Let's have a look at the package-creator point of view with this latter proposal. I have the concern about the duplicity of settings that the package creator should handle. If a certain target.board, like Arduino Mega, has the mcu atmega2560, then, the package creator should write something as:

def build(self):
    if self.settings.target.board == "Arduino Mega" or self.settings.target.mcu == "atmega2560":
        # do some stuff for this setting

Also I am not sure about having two different options for the user. Actually, as a user, why I should use either of? How do I know which is the correct one? Both are? 

$ conan install -s target=mcu -s target.mcu=atmega2560 ..
$ conan install -s target=board -s target.board="Arduino Mega" ..

Regarding the topic of build<->user views, I am not afraid of balancing either one, and the other can be made easier with helpers. So if we go with an approach close to the user and specify the final, complete target, it is very easy to abstract that with helpers for the package creator, so the user view:

$ conan install -s target=mcu -s target.mcu=stm32f407vg

And the package creator view:

def build(self):
    if "stm32" in self.settings.target.mcu:
        # do some stuff for this family

We can also very easily abstract away those details in a helper, very similar to the current CMake one, so it would look like:

def build(self):
    emb = Embedded(self.settings)
    if emb.base == "Cortex":  # Or emb.is_cortex
        # do some stuff for it
    if emb.family == "stm32" and "Arduino" in emb.board:
        # do other thing

Just some ideas from my first read of the thread, wanted to share with you. I keep reading and working on it.

astralien3000 commented 8 years ago

Trying to update myself, while checking PR https://github.com/conan-io/conan/pull/333/files, thanks very much @astralien3000 for your good job.

Thank you ^^ !

Also I am not sure about having two different options for the user. Actually, as a user, why I should use either of? How do I know which is the correct one? Both are?

There is two use case in embedded systems : either you develop for a well-known development board (Arduino, Discovery, Nucleo, Teensy, ...) or you develop for your own custom board.

We can also very easily abstract away those details in a helper, very similar to the current CMake one, so it would look like:

def build(self):
    emb = Embedded(self.settings)
    if emb.base == "Cortex":  # Or emb.is_cortex
        # do some stuff for it
    if emb.family == "stm32" and "Arduino" in emb.board:
        # do other thing

:+1: ! It seems like an idea I made earlier, and I agree it would be the simplest way to get these details. I think I will try to make a beginning of helper like this in the pull request.

Now I also need your feedback about naming conventions :

For microcontrollers :

I think the lowercase is the best because you don't need to remember the naming convention of vendors.

For boards :

The naming convention for the board is even more difficult !

lasote commented 8 years ago

I'll definitely dedicate more time to this proposals and your PR #333 . You are doing a GREAT work with this, it's a very hard feature, really.

Just give us some time, it's VERY important to us to get a good solution for cross compiling etc. I'm prioritizing it now. Thanks!

lasote commented 8 years ago

It's hard to assimilate all this information for me, I'm far of having your knowledge about the world of microcontrollers, but I have some points to share more related to the implementation in conan.

  1. I think the approach of board and mcu 5[improved flat] is good enough and should work well. But I think the library author will need to write good docs about what settings are supported or use the subsettings feature. If I'm writing a library that will only run in atmegas:
from conans import ConanFile, CMake, tools
import os

class mylibatmegaConan(ConanFile):
    name = "mylib_atmega"
    version = "1.0"
    settings = { "target": {
           "board": {"board": ["Arduino Mega","Arduino Uno"]},
                   "mcu": {"mcu": ["atmega2560", "atmega328p"]}
                 }
               }

It's a little tricky but proving an example like the above in the docs maybe it's ok. What do you think?

  1. I have some concerns about the thousand of devices and boards, I think it's too hard to provide a full settings.yml and have in conan codebase a helper with the logic for all those devices.
    • Maybe conan should provide some dynamic support to this like dynamic generators do?
    • Maybe (referring to PR #333) the helper source code should not have information about concrete devices and load the relations between them from another config file?
    • I think these file with all the mcu/board relations will change a lot and will be big. It sounds like we need to provide a way to update it and not harcode it in the conan codebase.
astralien3000 commented 8 years ago

It's a little tricky but proving an example like the above in the docs maybe it's ok. What do you think?

I didn't know you could do that, it's cool !

It may be useful in a lot of cases I think, but It also could be a tricky, for example : If the library is MCU specific and board agnostic, it seems useless to specify the boards. But you have no choice in this case.

I have some concerns about the thousand of devices and boards, I think it's too hard to provide a full settings.yml and have in conan codebase a helper with the logic for all those devices.

👍

  • Maybe conan should provide some dynamic support to this like dynamic generators do?

I don't understand this solution. Have you an example of "dynamic generator" in mind ?

  • Maybe (referring to PR #333) the helper source code should not have information about concrete devices and load the relations between them from another config file?

👍 It would be quite easy and may be necessary to move the relation tree to a config file. But be carefull, my PR si not complete. Indeed, it may be needed to add some information like the MCU's clock frequency or something. Actually I don't know if it would be a good or bad setting to add.

  • I think these file with all the mcu/board relations will change a lot and will be big. It sounds like we need to provide a way to update it and not harcode it in the conan codebase.

The relations may not change a lot if it is well designed in the first place ^^, but of course, a lot may appear and the need to update the relations. Should it be done in a separate "package" or something ? A different pip package ? A different git repository ?

lasote commented 8 years ago

About the concrete solution to update these config file, I don't know, we need to think about it. But it's good to know that you agree with not having hardcoded models/etc in the codebase and with the need of update this config file in an easy way. What do you think @memsharded?

So, I'll keep thinking! :)

astralien3000 commented 8 years ago

Hey ! I was just wondering something lately. Does adding a config file to the repository will not make the same problem as what @memsharded told me in #317 ?

lasote commented 8 years ago

Well, It could be auto-generated in runtime, so the pyinstaller issue shouldn't be a problem.

I've been trying to implement it and it's not easy at all. I tried something similar to conan dynamic generators with the "Embedded" helper. The idea was handle it as a conan package, so it could be updated, versioned, allow people to collaborate with PR in github... etc .But it's not possible because we also need this helper in conanfile.py "config" method to make decisions like "this Micro requires the same package than this board" and serve the same package for both of them. And the "config" method is executed before we have retrieved the requires (conan flow), so we cannot require the Embedded class and use it in config method.

I talked with @memsharded about the problem and we don't have a solution (yet). We will keep reporting progresses.

astralien3000 commented 8 years ago

Hello !

I'm beginning to implement the config file. I'm almost done.

I just have a question about what is better about the config file. Of course, as we said before, it is better to have it in a separate repository. I have 2 solution about this :

What do you think is the best ? Would you do it an other way ?

Cheers !

absassi commented 7 years ago

Hi, it looks like this issue have been abandoned since @astralien3000 stopped working on PR #333, but once this is still open and looks like here's where the main discussion about micro-controllers is, I'd like to contribute to this topic with my experience. Issue #675 is also related, but I feel here is more adequate.

Standard settings specific for micro-controllers

First, some philosophical questions. I'd like to point out that the standard settings that ships with Conan are pretty useful to share pre-built ABI-compatible packages with the world (conan.io), but not much for anything else, AFAIK. Do we want to do globally share binaries for the micro-controller or even the general embedded systems community? Even knowing there are almost infinite possible combinations for ABIs (or at least thousands if we take into account only the most common)? Or would sharing with the world source-only portable packages, leaving pre-built packages only for internal distribution among a team or company, suffice? Because Conan already do this job reasonably well (at least better than any other tool I've found so far). And it also does work for the pre-built case using options - far from perfect for lots of reasons, but works.

The main motivation for my choice to use Conan was that it solves the dependency problem with both source and binary packages. I believe no embedded developer would ever mind to build a package from source if that's as easy as using a pre-built one. Actually, many will prefer to do that, to specify custom flags.

Free binary-only distributions are somewhat rare. If it is still useful to have one of them in conan.io, the packager can source-only package (where the source are the binaries, perhaps to be downloaded or extracted) where its source() method succeed or fail depending if the binary is compatible with provided environment variables (see below about cross-compilation), using custom Python logic for that. It is a workaround, but I couldn't think anything better.

Non-free binary-only distributions are less rare, but uploading them to conan.io would violate its license.

tl;dr: My opinion is that Conan should not bother enumerating micro-controllers or other embedded targets, because it is not useful. Let's populate our local caches with cross-compiled binaries, not conan.io.

Cross-compilation

My understanding on how Conan deals with cross-compilation is that the user must create a profile specifying the compiler settings and then environment variables specifying how to invoke such compiler.

Where I think Conan is still weak for cross-compiling:

All that said, I could make Conan package an RTOS with a very complicated (and hard-coded) build system and I'm very satisfied, although it still requires some tweaks in consumer's conanfile.py for using the package.

memsharded commented 7 years ago

Hi @absassi,

I think you have done a very good and thorough analysis:

We are also happy that you were able to do such package, and you are satisfied with conan :). I suggest that we try to improve over those specific items, and I think the best approach would be to open specific issues for each one of them, or just the one that you think has largest priority (the smallest and more focused, the easiest to achieve progress), and start iterating and working on it.

Thanks very much. Great job.

memsharded commented 6 years ago

Reviewing this issue, it seems it has been holding for a long time:

So, it seems there is no action required on these items, so I am going to close it atm. Please feel free to re-open or comment if necessary.