search

knurling-rs / defmt

Efficient, deferred formatting for logging on embedded systems
https://defmt.ferrous-systems.com/
Apache License 2.0
846 stars 76 forks source link

Support non-embedded targets #463

Open Dirbaio opened 3 years ago

Dirbaio commented 3 years ago

Currently defmt only works in ELF targets, and requires a custom linker script. Is supporting non-embedded targets in the radar?

I think the most portable way would be to not rely en ELF/linker tricks and have the proc macros hit flat files (or a sqlite DB?) to assign indexes to format strings.

This would be a big change, so if you don't want to do this because the only goal is supporting embedded devices, I understand :) I thought I'd open this issue to confirm.

Urhengulas commented 3 years ago

This certainly sounds like a desirable feature and feasible if someone likes to contribute or fund it.

DBLouis commented 3 years ago

Is it possible to make defmt works on standard x86 linux? It fails to link with the custom linker script. If I don't use the custom linker script readelf still shows a section that contains my log strings. However when running defmt-print on the ELF it fails with: Error: .defmt data not found.

japaric commented 3 years ago

Is it possible to make defmt works on standard x86 linux?

No OS is supported at the moment; that's what this issue is about. What you are describing are the symptoms of several unimplemented things.

It fails to link with the custom linker script.

sounds about right. wouldn't know the exact reason off the top of my head but may be related to Linux binaries being relocatable. Firmware images use absolute addresses for everything (relocation=static in linker/compiler lingo). This ("format string 'interning'" is probably the hardest thing to fix / design / implement. More so on platforms that don't use the ELF format: macOS and Windows.

If I don't use the custom linker script readelf still shows a section that contains my log strings.

(interesting that they "survive" (are not discarded by the linker) the linking process)

However when running defmt-print on the ELF it fails with: Error: .defmt data not found.

without the custom linker script the log strings won't end in the right place so the .defmt linker section is not created hence the error message

The other thing that's missing is a transport crate for Linux and other OSes. That crate defines how the encoded/compressed defmt data is output (e.g. framing) and to where (e.g. stdout, a file, a named pipe, a socket, etc.). You would then pass this defmt data to defmt-print

DBLouis commented 3 years ago

Is there any hope of making it work by setting the right compiler flags so that the log strings are "at the right place"? I don't know much about linking so maybe I am missing the point

japaric commented 3 years ago

maybe? I'm familiar with the details of static linking but not the details of dynamic / relocatable linking so I can't answer that

DBLouis commented 3 years ago

With the following parameters linking succeed but the program segfault immediately. I think the default defmt script might overwrite stuff.

  -C relocation-model=static  -C target-feature=+crt-static  -C link-arg=-Tdefmt.x

For what I understand, the INFO section should not interfere with the loaded program and only be present in the ELF file. If I remove only the last link argument, my program starts, if I put it back it crashes before main.

DBLouis commented 3 years ago

I had assume that the default linker script was always passed as a parameter. Looks like is it not. After adding it to .cargo/config my program compiles and runs, and defmt-print decode frames correctly.

[target.x86_64-unknown-linux-musl]
rustflags = [
  "-C", "target-feature=+crt-static",
  "-C", "relocation-model=static",
  "-C", "link-arg=-T/usr/x86_64-linux-musl/usr/lib/ldscripts/elf_x86_64.x",
  "-C", "link-arg=-Tdefmt.x"
]

It seems to works fine without +crt-static but not without relocation-model=static. For both musl and glibc.

Urhengulas commented 3 years ago

It seems to works fine without +crt-static but not without relocation-model=static. For both musl and glibc.

@DBLouis, very cool that you made it work! 🎉 What is your usecase that makes you want to use defmt over log?

andresv commented 2 years ago

Use case can be that there is a no_std lib that is primarily meant for embedded devices and it has bunch of defmt prints in there and you wanna use this lib also on x86.

It seems one way to do it is just to add different feature flags to the lib itself: https://github.com/embassy-rs/embassy/blob/master/embassy/src/fmt.rs

newAM commented 2 years ago

Use case can be that there is a no_std lib that is primarily meant for embedded devices and it has bunch of defmt prints in there and you wanna use this lib also on x86.

That would work, but I think the goal is to have binary logging on hosted targets, for things like embedded linux where there are storage limitations with normal logging.

gauteh commented 2 years ago

Came across this while working on a defmt-serial target: https://github.com/gauteh/defmt-serial. Wrote an example for the host environment, works with: https://github.com/knurling-rs/defmt/issues/463#issuecomment-894677833:

❯ DEFMT_LOG=debug cargo run
   Compiling defmt-serial v0.1.0 (/home/gauteh/dev/embedded/defmt-serial)
   Compiling example-std v0.1.0 (/home/gauteh/dev/embedded/defmt-serial/example-std)
    Finished dev [unoptimized + debuginfo] target(s) in 0.54s
     Running `bash defmt-print-runner.sh target/x86_64-unknown-linux-musl/debug/example-std`
Hello, world!
Logging to info with defmt..
Good bye.
INFO  Hello defmt-world!
└─ example_std::main @ src/main.rs:28
DEBUG Now at: 0
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 1
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 2
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 3
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 4
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 5
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 6
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 7
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 8
└─ example_std::main @ src/main.rs:31
...
DEBUG Now at: 45
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 46
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 47
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 48
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 49
└─ example_std::main @ src/main.rs:31
WARN  Done!
└─ example_std::main @ src/main.rs:34
^C
defmt-serial/example-std on  main via 🦀 v1.61.0-nightly
Urhengulas commented 2 years ago

That is great to see @gauteh! Is there anything we can do to support non-embedded targets better?

gauteh commented 2 years ago

@Urhengulas , I guess the biggest problem is the rustflags that are currently needed. Couldn't get it to work on -gnu target. The rest is solvable:

nathaniel-brough commented 2 years ago

If anyone is working on this, you might find the linker scripts and tooling in the google/pigweed/pw_tokeniser directory useful. It's all written in c++ but works for host/embedded-arm but uses similar mechanisms to what defmt uses. i.e. pw_tokeniser and defmt offer very similar functionality.

elpiel commented 1 year ago

Hello, I saw this issue and since I don't see much work around it after 2 years I was wondering if there's a way to circumvent defmt logging all together?

Right now https://github.com/knurling-rs/defmt/issues/463#issuecomment-894677833 seems to be working for linux -gnu as well, the only problem I'm encountering is that doc tests started failing. I'm using defmt_serial with an StdoutSerial impl from the std example.

note: /usr/bin/ld: /mnt/Disk/Projects/lechev.space/netbox/netbox-hardware/target/x86_64-unknown-linux-gnu/debug/deps/libnetbox_firmware-5b6d3b1c9ed15201.rlib(netbox_firmware-5b6d3b1c9ed15201.14bxe1s9urqapwuu.rcgu.o):
relocation R_X86_64_32 against symbol `rust_eh_personality' can not be used when making a PIE object;
recompile with -fPIE
          /usr/bin/ld: failed to set dynamic section sizes: bad value
          collect2: error: ld returned 1 exit status