OS X build

[ilg-ul]

I tried to build the current version on my Mac and I got:

ilg-mbp:distortos.git ilg$ tup
[ tup ] [0.000s] Scanning filesystem...
[ tup ] [0.004s] Reading in new environment variables...
[ tup ] [0.004s] Parsing Tupfiles...
* 1) [0.225s] .
tup error: Expected node 'output' to be in directory '.', but it is not there.
tup error: Failed to find directory ID for dir './output/distortos.elf' relative to 1
tup error [string "builtin"]:217: Error while parsing 'outputs'.
stack traceback:
    [C]: in function 'definerule'
    [string "builtin"]:217: in function <[string "builtin"]:61>
    (...tail calls...)
    [string "Tuprules.lua"]:134: in function 'link'
    [string "Tupfile.lua"]:12: in main chunk
 [          ETA~=10s Remaining=41           ]   2%
 *** tup: 1 job failed.

I installed tup from MacPorts, and added gcc-arm-none-eabi-4_9-2014q4/bin to the PATH.

Any suggestions?

Regards,

Liviu

[FreddieChopin]

Hmm... I must say that I don't know much about Mac OS X, but here are a couple of questions:

1. Could you tell me the exact version of tup (tup --version)? Generally you need a pretty recent version (0.7.3 - so to build distortos. Macports website lists version 0.7.1 - this is too old /;
2. Could you tell me whether you're using the most recent Mac OS X version?
3. Do you have FUSE installed too? tup needs it, for Mac the library is https://osxfuse.github.io/
4. Would it go any further if you'd create the "output" directory in the root of the project?

There seem to be many issues with tup and Mac OS X combination. Recently I was wondering about adding an alternative build system using make, because I expect tup may be seen as an obstacle for many potential users...

As a temporary solution you could build the project with your GNU ARM Eclipse plugin. The build is pretty simple, I can post output of compilation so that you'd see what flags are used.

BTW - with the toolchain you are using you'll get quite huge executable - due to exception's handling code - see the link from the README.md file. It's just the size - everything else should work fine (at least it did last time I checked).

[ilg-ul]

1. MacPorts 0.7.1; please update your docs to reflect this requirement, and possibly provide instructions (or link to instructions) on how to build tup if the MacPorts one is not usable.
2. OS X 10.9.5 (not the most recent, but I'll update as soon as 10.10.2 will be out).
3. under MacPorts, tup automatically installs FUSE, I checked and I found libosxfuse.dylib and libosxfuse.2.dylib; are these ok?
4. manually creating "output" made it went further (still with tup 0.7.1)

ilg-mbp:distortos.git ilg$ tup
[ tup ] [0.000s] Scanning filesystem...
[ tup ] [0.013s] Reading in new environment variables...
[ tup ] [0.014s] Parsing Tupfiles...
 1) [0.002s] output
 2) [0.012s] .
 3) [0.002s] documentation                                                                                                                                                                                                                        
 4) [0.002s] documentation/images                                                                                                                                                                                                                 
 5) [0.002s] documentation/images/source                                                                                                                                                                                                          
 6) [0.001s] include                                                                                                                                                                                                                              
 7) [0.001s] include/distortos                                                                                                                                                                                                                    
 8) [0.002s] include/distortos/allocators                                                                                                                                                                                                         
 9) [0.002s] include/distortos/architecture                                                                                                                                                                                                       
 10) [0.001s] include/distortos/containers                                                                                                                                                                                                        
 11) [0.002s] include/distortos/estd                                                                                                                                                                                                              
 12) [0.002s] include/distortos/scheduler                                                                                                                                                                                                         
 13) [0.001s] source                                                                                                                                                                                                                              
* 14) [0.005s] source/allocators                                                                                                                                                                                                                  
tup error: Expected node 'source' to be in directory 'output', but it is not there.
tup error: Failed to find directory ID for dir '../../output/source/allocators/SimpleFeedablePool.o' relative to 71
tup error [string "builtin"]:217: Error while parsing 'outputs'.
stack traceback:
    [C]: in function 'definerule'
    [string "builtin"]:217: in function <[string "builtin"]:61>
    (...tail calls...)
    [string "../../Tuprules.lua"]:128: in function 'cxx'
    [string "../../compile.lua"]:21: in main chunk
    [C]: in function 'include'
    [string "Tupfile.lua"]:14: in main chunk
tup error: Failed to parse included file '../../compile.lua'
 [          ETA~=<1s Remaining=29            ]  32%
 *** tup: 1 job failed.

• tup - indeed, it is not be very attractive, but not necessarily by itself, but because you are using the latest version, which is not available for direct install neither on my Mac nor on my friend's Linux; relaxing this might help; if this is not possible, then provide full details on how to build it, on all relevant platforms might help
• GNU ARM Eclipse - perhaps instead of providing an alternate make build system, you could just add the files required to make it an Eclipse project, with all options properly set.
• a friend did just that, created the managed Eclipse project and successfully built it, but when trying to run it on an STM32F407, it crashed during the first context switch; I blamed the different build environment and suggested him to first test the official version, built with tup; but I'm not convinced this is the problem
• toolchain - I know you have a custom toolchain, but, for a simple test, the huge executable should not be a problem

[ilg-ul]

we went further, my friend compiled the latest tup on Linux, changed the FUSE config file, the tup rights, the build went further, but is still does not pass, it crashes during compile :-(

tup is indeed a big obstacle...

probably you should work on the build details a little more and make it more robust.

[KamilSzczygiel]

MacPorts 0.7.1; please update your docs to reflect this requirement, and possibly provide instructions (or link to instructions) on how to build tup if the MacPorts one is not usable.

Please note, that this requirement is stated in the README.md file: "tup build system, version 0.7.3 or above".

Looking at the build file for tup from Macports ( https://trac.macports.org/browser/trunk/dports/devel/tup/Portfile ), I believe all that is required is changing "0.7.1" to "0.7.3" and rebuilding. Or at least I'm under the impression that Macports compiles the executable on your computer, not downloads the binary from central repository...

under MacPorts, tup automatically installs FUSE, I checked and I found libosxfuse.dylib and libosxfuse.2.dylib; are these ok?

I believe they are OK.

tup - indeed, it is not be very attractive, but not necessarily by itself, but because you are using the latest version, which is not available for direct install neither on my Mac nor on my friend's Linux; relaxing this might help; if this is not possible, then provide full details on how to build it, on all relevant platforms might help

Unfortunately, 0.7.3 fixes a bug that would make the build fail somewhere further, so relaxing this requirement is not possible ); Building tup is pretty simple - just download the sources and execute bootstrap.sh, and when it finishes you can copy (or not) the binary (install it) anywhere you want.

a friend did just that, created the managed Eclipse project and successfully built it, but when trying to run it on an STM32F407, it crashed during the first context switch; I blamed the different build environment and suggested him to first test the official version, built with tup; but I'm not convinced this is the problem

Probably this crash happened because one (or both) of two things:

1. You have to pass "-DUSES_TWO_STACKS -DUSES_CXX" to assembling of the source/architecture/ARM/ARMv7-M/ARMv7-M-startup.S file.
2. The tests are currently written for "-O2" optimization enabled and I believe your friend compiled that with -O0. In that case the stack sizes would be too small and the first context switch would cause a crash of the system due to stack overflow.

Here is a complete build output from tup - http://pastebin.com/8EGrEuiL . Almost all commands can be used as presented, only the last - linking - has to be actually edited. You need to replace %<objects> with the paths of all *.o created files.

we went further, my friend compiled the latest tup on Linux, changed the FUSE config file, the tup rights, the build went further, but is still does not pass, it crashes during compile :-(

Could you show me the exact error that you encountered?

[ilg-ul]

Could you show me the exact error that you encountered?

the build finally passed, we programmed the elf in a STM32F407 board and... nothing happened, apparently execution stopped in one of the fault handlers.

what is the expected behaviour?

[ilg-ul]

apparently execution stopped in one of the fault handlers.

fixed this too, it was probably due to not using -O2; now it terminates properly.

[KamilSzczygiel]

Good to hear that it works for you on Linux. Did you have any luck in getting recent version of tup on MacOS?

[ilg-ul]

not yet, for my projects I generally prefer to use stable MacPorts in favour of building from sources, not necessarily because this cannot be done, but because this is not something to recommend to the average user.

to be frank, I opened a ticket on MacPorts, asking for a tup update to 0.7.3...

[KamilSzczygiel]

I've written a post on tup-users mailing list (and CCed the maintainer of tup in Macports) asking about update to Mac OS X version from Macports.

https://groups.google.com/forum/#!topic/tup-users/Y-36_GT8kos

[ilg-ul]

thank you, I'll most probably wait for it and retest when available.

[KamilSzczygiel]

Have you considered using Homebrew instead of MacPorts? Homebrew has the most recent version of tup ( https://github.com/Homebrew/homebrew/blob/master/Library/Formula/tup.rb ), while the one in MacPorts is 2 versions behind. Your ticket with update request is not the only one - there is another one, posted few months ago and still not resolved, so I wouldn't hold high hopes for a quick update... );

This was suggested to me on the tup-users mailing list (link above).

[ilg-ul]

Have you considered using Homebrew instead of MacPorts? Homebrew has the most recent version of tup

I did; but have you considered also providing an Eclipse managed project for building the test application?

[KamilSzczygiel]

I haven't thought about that much... Don't take this personally, but providing such project is not a general solution - not everyone uses Eclipse, and of those who do, not everyone uses your plugin. On the other hand, make and/or tup can be used from Eclipse or directly from the shell, without Eclipse (or any other IDE)...

If you want, I could prepare and send you such preconfigured project so that you could evaluate distortos, but at this moment I'm a little reluctant to maintaining such project(s). (Unfortunately) I'm in the last group mentioned above - I use Eclipse, but prefer tup and/or make and don't use any plugin that manages the build, so maintaining that would be extra work for me...

It seems to me, that using Homebrew would solve your issue right away...

[ilg-ul]

ok, good luck with tup.

[KamilSzczygiel]

C'mon - I asked you not to take that personally (; Tup is a great tool - it's not just another variant of make - check its website ( http://gittup.org/tup/ ) and who knows, maybe you'll even like it (; . Sure - there are some rough edges, but Eclipse is not a perfect tool either...

I'll provide a make-base build system and I'll think about providing a managed project. There are still so many things to implement in distortos, that maintaining 3 different build system seems like a waste of time...

[ilg-ul]

Tup is a great tool

I can take your word for this, and you are free to use any tool you like. But, if I may a suggestion, when you choose a tool please be sure it is readily available on all relevant platforms, otherwise you may run into small acceptance problems.

I personally avoid to install libraries and applications that end up in the public locations (/usr/local included) so homebrew is out of luck. MacPorts is accepted since it uses a completely separate location (/opt/local).

I'll think about providing a managed project

if you need any help for this, or if you want to test your project, please let me know.

in my plug-ins I took a lot of special measures to allow for portable projects, it would be nice for your project to be portable too.

[KamilSzczygiel]

I'd be grateful for a test of the new make-based build system on OS X. Just execute make all (or just make). I'll test in on a nearby Windows too, so if it will work on OS X I'll add information about it to README.md file.

[ilg-ul]

a test of the new make-based build system on OS X

passed, without any problems:

arm-none-eabi-size -B output/distortos.elf
   text    data     bss     dec     hex filename
  50104    1076   14880   66060   1020c output/distortos.elf

[KamilSzczygiel]

Great - thanks!

[ilg-ul]

after a superficial look at your make/tup files, I would say that automatically generating the make files from the tup ones might be possible.

[KamilSzczygiel]

My initial plan was to make the build systems share some common description, something like a max-simple version of Rules.mk - with just the info about flags (maybe only variables like "INCLUDE_DIRECTORIES", "DEFINES" and stuff like that). But given that tup uses lua I think it's possible to actually parse the Rules.mk files in tup directly.

I'd prefer not to have anything "generated", as that's another step that has to be executed before building anything. Such generation would also put higher requirements on user's shell utilities (possible problems on Windows)...

The descriptions will definitely need to be shared in the (near) future... I assume you suggest using something like sed or awk (or maybe just shell?) and execute that from some configure.sh script, right? There are some minor nuisances to work out (especially in the "main" Rules.mk file in top-level directory + 2 Rules.mk files for architecture and chip [the issue with .elf dependency on linker scripts]), but it could possibly work...

I'm not committing too much time to the build system right now, as when I'll add support for different architectures, boards, chips and configurable applications, the whole thing will most likely get much more complicated... That's the thing you mentioned in your e-mail - currently I'm just ignoring these issues (;

In the future I'd like to support kconfig-frontends tool for configurations + a little bit of shell script for sth like configure.sh script. In this aspect I'd like to use some ideas of NuttX project, but with a lot of simplifications (NuttX is way too complicated)...