Good ways to debug and emulate the firmware under Unix?

[maxieds]

I am running into a number of complications finely tuning and testing the DESFire emulation support mentioned in #218. All of the solutions I have seen running gdb with avr-gcc related tools are either dated at this point of do not have working integrated support for the ATXMega chips on the Chameleon devices. I see you all merging pull request after pull request here. Can some of the core developers shed some light on how they debug when writing new tag implementations for the firmware?

The DESFire support is notably and necessarily more complicated, with many more branches to take through the code, than any of the other tag emulation support I have seen so far. It's getting to the point where I'm spending 2/3 of my time on the project just re-flashing to see a couple of debug messages printed slowly over live logging. It would be very nice to simulate and set breakpoints, etc, and restart the firmware quickly when something inevitably crashes out.

Thanks ahead of time.

[ceres-c]

During the years I've been debugging the Chameleon in different ways, depending on the situation

When applicable, the easiest thing to do is to compile the code as a x86 application and debug that. It might be useful when trying to validate a single function with different inputs (we did it for ISO15693PrepareFrame) This is applicable only when you have a single entry point with different data and you want to evaluate the output of your function.

Debugging asynchronous functions such as ISRs is way more complex due to the precise timing required and breakpoints/logging would interfere with the other operations, breaking the software while trying to debug it. Imagine breaking on a function which is called every 20 us to sample data on the carrier: on the second hit you'd be unsynchronised to the external clock. In that scenario you can only use the PORTE pin on the back of the board (marked PE0 on the PCB) and an oscilloscope. The port is configured placing this line in a function which is called once (codec/application init, for example) PORTE.DIR = PIN0_bm; and placing, where needed, this line PORTE.OUTTGL = PIN0_bm; You'll see something similar to this screenshot and it'll be possible to (kinda) debug time critical functions. This debugging setup is still present in SniffISO14443-2A.c.

Ultimately, if you really need functions/data breakpoints you can use an ICE such as Atmel ICE which is supported by avarice, thus by avr-gdb. I own the bare PCB version of the debugger and wrote a blog post on getting it to work. Beware, the PCB version does not come with cables, so a you'll need a IDC cable (10 pins, 1,27mm pitch) to breakout your own connection board.

Also, you can debug with Atmel Studio which is, admittedly, much more powerful and stable than avarice. You don't need to build the codebase with Atmel Studio, but you can import separately the source and the elf built with make. There is an option somewhere in the file menu to import external elfs which will be disassembled by Atmel Studo and it should interleave the C source with disassembled code. Don't forget to configure debug settings in Atmel Studio if you want to use it: disable memory erasure and program download, it's better to build & upload with the makefile, just to be sure you're debugging the same file which is running in the Chameleon. You can also use data breakpoints with Atmel Studio, which are not supported by avarice (as far as I know). There were some limitations on breakpointing data and functions at the same time and they are documented in the CPU manual if I'm not mistaken.

[maxieds]

@ceres-c Thank you for all of the suggestions. I wonder if your solution is also a way to fix some broken RevG boards of mine that I have managed to snap off the MicroUSB connections to over the years? Unfortunately, as a non native EE type myself, my soldering skills are NULL, so I would need a friend to help get this option working.

I have previously tried using simavr with no luck due to apparently broken support for the XMega chips. It could also be that I changed too many settings with my DESFire code (to get some extra stack space) and it doesn't know how to handle that setup.

Just to follow up, my active setup has been to use LOGMODE=LIVE with my Android app, which is admittedly much less of a powerful way to debug all of the complications I have seen in my code. Nonetheless, this setup allowed me to get it more or less working reliably enough to release a working set of firmware binaries in #278. To get this working with my app while I was printing live debugging messages from within firmware setup and command routines, I had to make this modification, which essentially allows printing, or flushing serial USB output, only at certain "tick" times within the firmware execution. Otherwise, the logging identifiers can come out jumbled.

[ceres-c]

You could use the Atmel ICE to flash the MCU and debug it, but you wouldn't be able to communicate with it unless you heavily edit the current codebase since, if I'm not mistaken, the telnet interface is exposed only through USB. You would still need a working USB connector

By the way me and @MrMoDDoM hit that logging behavior as well while debugging other stuff

[maxieds]

@ceres-c The live logging behavior is unofficially fixed so that minicom and LOGMODE=LIVE functionality seem to work in the latest release of my DESFire firmware (after some prodding by a tester to fix this). Eventually this should get merged assuming the emsec maintainers are actually willing to merge the massive PR that would have to be a DESFire. If you want to take the code (and here and some more here to flush the terminal buffer with the LUFA library) for it and make a PR just for that before I get around to things, go for it!

It's been an issue for a while that I stumbled over with my Android app. It comes up there since the toolbar auto-refreshes the config and UID information every few seconds, which means that there is a handler going off regularly to call a terminal command. On the other hand, calling asynchronously scheduled commands as normal with the app that have logging commands in them also would have (in previous code) had flushes and writes to serial USB in the middle of some of the scheduled handler code returning data over USB. Kaboom! It collides and things get reported in jumbled order. I had to figure out a solution pretty early on this summer to use my testing setup with the firmware. Basically, the solution is to add anything printed to logs to a small linked list, and then only print this at a new LiveLogTick in the main loop.

Also, for whatever this is worth, it makes me wonder why we are using so many structures in the default LUFA configuration. One of the big things I had to change to make space for crypto storage compendia and stuff is to reduce the LUFA config by one massive (approx 400 bytes!) Endpoint structure. Has anyone looked into whether all of these structures from the default LUFA configuration are needed that way? Nothing is running in parallel as far as I can see. Those structures could be used to free up even more space if things get too tight with the DESFire mods.

[david-oswald]

Hi @maxieds and @ceres-c, yes a separate PT for the logmode functionality would of course be great.

Independent of that, we'd definitely consider also merging the DESFire code at some point - maybe as an optional feature (e.g. through Makefile defines) and not in the default build to address possible license issues and code size problems. We'll have to discuss that a bit internally.

[maxieds]

@david-oswald After I reduced the number of LUFA Endpoint structures by one, there should be plenty of space with the remaining code for the other configurations support. I get that as more functionality gets add for other tag support later things can get tight, and quickly. That being said, to my knowledge the DESFire support is probably going to be one of the more complex and memory intensive NFC tag types people seem to care about getting support for with the Chameleon devices, so most new tags added later will not have such necessarily bloated requirements on space.

When you decide how this will get handled, please also advise me how best to submit a PR this large, and if there are any other procedures that would help with easing this transition. For reference, I have had to change the RevG Makefile source considerably for my mods of the firmware. I would have to check, but it's possible that some things were also changed in the LUFA/Build/lufa_build.mk source as well. This should be more minor than the primary Makefile changes though. There is also some limited support for new terminal commands that are wrapped in defines so that they only get compiled in when the DESFire support is enabled in the Makefile.

I should be able to file an initial pull request to get the LIVE logging functionality changes in soon (maybe later today).

I also still think it's worth looking at whether enabling all of those LUFA Endpoint structures is actually necessary. It might be possible to get away with only 1-2 of them, say one for serial USB transfers, and possibly another for the RF transfers if that is how that functionality is working. This is by far based on my reads of avr-objdump output the quickest and easiest way to free up stack / .data segment space for buffers. There have been some other tweaks to my code for things like reducing the terminal buffer size. Once things get tight, 50-100 bytes is a massive improvement that is the difference between "bricked" firmware and things going smoothly! Like I said, each LUFA endpoint structure is somewhere around 380-400 bytes a pop, so that is definitely something developers might started considering.

[maxieds]

I also wonder about things like where best to place the libnfc testing code sources so users can (if they so choose) verify continued support for the functionality. Maybe this could get placed in the lower level Docs directory for reference?

[maxieds]

A hint as to whether or not this many LUFA Endpoint structures are needed is on this line. It appears that we only need one of these. That would free up something like 4 * 380 bytes immediately!

[skuep]

Plus 2 endpoints from this line: https://github.com/emsec/ChameleonMini/blob/603121bc21d0caa71e2faa67cfef81da3e8ee075/Firmware/Chameleon-Mini/LUFADescriptors.c#L157

However it may be possible to reduce the endpoint buffer size, if I am not mistaken.

[maxieds]

@skuep Right. I just noticed in testing with my pull request #279 that it doesn't work with one or two. Still looks like we can reclaim some real estate on the AVR chip by reducing the number of endpoints. :)

[maxieds]

@david-oswald I filed the pull request to add the enhancements to LIVE logging modes (#279). Do you have any feedback yet about procedures or how you all want to handle the pull request for DESFire emulation? I'm only getting some preliminary feedback from testers about install related issues. I think that with the testing code I have, the PR might be reasonable at this point. So it might work, if it is OK with you all, to merge the core of that emulation into main sources, and expect that it being there will get more folks to test it.