Alternative flash mode?

[felias-fogg]

Hi,

I like your debugger quite a lot and it works like a charm under platformIO. When reading the code while adapting the debugger to the ATmega1284 (see pull request), I was wondering whether another flash debugging mode would be possible. Currently you insert a "dead loop" at a breakpoint and discover that you have reached it by looking periodically using the WDT. Couldn't you also insert a "clear INTx output pin" instruction that raises a software interrupt using INTx? Since you use these software interrupts for single stepping anyway, this would not block another resource, and it would free the watchdog timer.

Best regards & thanks for your good work, Bernhard

[jdolinay]

Hi, Thank you! It is a good point about the external INT. I tried this option back in 2017 and the problem was that the program did not stop at the location of the breakpoint but one or 2 instructions later. To tell the truth I am not sure if I tried clearing the output pin, but I did try enabling the interrupt and having the pin low all the time. And there seemed to be the problem with stopping too late. I tried tricks like moving the PC one step back but it probably didn't work well, so I abandoned this method. You can find the code in the history around May 20, 2017. There is this comment in the code from that time:

/* For trapping we use RJMP on itself, i.e. endless loop,
   1100 kkkk kkkk kkkk, where 'k' is a -1 in words */
//#define TRAP_OPCODE 0xcfff
/* Trapcode for enabling INTx
  opcode SBI is 1001 1010 AAAA Abbb  (A is address, b is bit number)
  for EIMSK = 0x1d bit 0 AAAAA = 11101, bbb = 000 > 1001 1010 1110 1000 = 0x9ae8
  EIMSK |= _BV(INT0);   enable INTx interrupt
  143e: e8 9a           sbi 0x1d, 0 ; 29
  PORTD &= ~_BV(PD2);
  1440: 5a 98           cbi 0x0b, 2 ; 11
 */
#define TRAP_OPCODE 0x9ae8
// opcode with int and then infinite loop
//#define TRAP_OPCODE 0x9ae8cfff

From the comment it seems maybe I tried also the version with clearing the output pin. But if you feel like experimenting you could try :) I think the problem with this method is that there is latency between clearing the pin/enabling interrupt and actually breaking the program which means that the program stops "behind" the breakpoint, not at it. If I could find my notes from that time maybe I could give more details... :(

Thanks for your great commit with the ATmega1284 support! Jan

[felias-fogg]

Hi Jan,

well I feared something like that. Well, good to know that one should not go down this way ;-)

I now have an alternative idea. Instead of using the watchdog, one could use the TIMER0_COMPA interrupt, which is unused by Arduino and hardly ever used by somebody else. One could configure it so that it is called between two TIMER0_OVF interrupts (which count the millis). This "trick" is described here: https://learn.adafruit.com/multi-tasking-the-arduino-part-2/timers

That means that if Timer0 is untouched by the program and the compare interrupt is unused, one could alternatively use this method and does not need to use the watchdog. One would need to make sure that the watchdog does not cause an interrupt/reset while executing the gdb-stub, though. This means calling wdt_reset as often as possible.

I'll give it a try and tell you what came out of it.

BTW: In the 1284 commit, in line 134 I forgot to add: "|| defined(AVR_ATmega1284) || defined(AVR_ATmega1284P)"

Best, Bernhard

[jdolinay]

Hi Bernhard,

OK, that sounds interesting. I will take a look at the link and I am looking forward to hearing about the results of your experiments.

I added the defines for ATmega1284 now, I didn't notice that either when going over the changes. Good that it's just an error message for wrong configuration.

Best, Jan

[felias-fogg]

BTW, I finally figured out one important difference between the plain and the P-version. The latter has timer 3, while the plain 1284 doesn't have that. So, line 796 should only be conditionally included if TIMSK3 is defined (resp. if the __AVR_ATmega1284__ is undefined).

[jdolinay]

I'm trying to verify that but in the datasheet (page 115) they say "Timer/Counter3 is only available in ATmega1284/1284P" so I'd read it that both P an "non P" have the timer3. Also if I build a project for 1284 and 1284P there is no problem, TIMSK3 is defined for both. The definition points to iom1284.h or iom1284p.h respectively, so the selection of the MCU is properly applied. I tried this in Eclipse. Do you get a build error in PIO if you use TIMSK3 for the 1284?

I think this is not too important anyway, as the affected code (gdb_no_bootloder_prep) is just to provide better report if the user enables flash breakpoints without the proper bootloader... but it's interesting.

[jdolinay]

Back to the TIMER0_COMPA interrupt for checking for breakpoints. To tell the truth, I don't think this will be worth the effort. My thoughts:

1. Using interrupt from the timer that is already used in Arduino seems good, "it costs nothing", but I think the debugger is also used by people who do not use the Arduino framework, so they may need the timer for other purposes. I am also not sure if some Arduino libraries do not use the compare function of the timer. I could imagine servo or analogWrite-like functions that need to generate PWM on as many outputs as possible to use it.
   In general timer is a more precious resource. I'd guess that not many Arduino users use watchdog. And even if one needs watchdog, he doesn't probably need it while debugging the app; it can be enabled later for the release version of the code (using conditional compilation to turn off watchdog-related app code in debug version).

2. If you use this approach you rely on the timer being initialized by Arduino framework or you need to somehow check that it is initialized and do it yourself if it is not - if somebody uses the debugger without Arduino framework.

3. The interrupt will be generated every millisecond which is not necessary and wastes CPU time. In the current version the watchdog interrupt is triggered every 500 ms. Let's say the check takes 0.1 ms to execute (it's probably less but just to show the point). If you check every millisecond, the CPU spends 10% of its time on the check for breakpoints, leaving 90% to the debugged app. If you check every 500 ms, you only waste 0.02% of the CPU time. Well, 90% may seem not too bad, but consider that this is "high-priority" time in interrupt service routine. If you break the code running in the main loop (and any other interrupts with lower priority) every millisecond, it may affect timing of operations like software based serial communication and the like...

BTW, the last point is something the author of the article from your link probably did not consider. He says that calling millis more then once a millisecond is kind of waste, but his solution is actually waste in my opinion. It is okay to "waste" time in the loop; this has no effect on the other tasks the program may need to do in interrupt service routines (like processing received bytes from serial line). But putting such a lot of code into service routine that is executed every millisecond is not a good idea. It is good practice to do only the minimum required in the ISR and leave any processing to the main loop. :)

[felias-fogg]

I'm trying to verify that but in the datasheet (page 115) they say "Timer/Counter3 is only available in ATmega1284/1284P" so I'd read it that both P an "non P" have the timer3. Also if I build a project for 1284 and 1284P there is no problem, TIMSK3 is defined for both. The definition points to iom1284.h or iom1284p.h respectively, so the selection of the MCU is properly applied. I tried this in Eclipse. Do you get a build error in PIO if you use TIMSK3 for the 1284?

You are right! It would have been strange anyway because usually the P-version is just the pico-power version.

[felias-fogg]

Hi Jan,

well, I implemented it already and you might want to have a look at it anyway ;-).

I agree that this might not be useful for everybody, but only for those who use the Arduino framework. And then, of course, only, if they do not touch the timer. BTW: timer 0 and the output compare registers of that timer are not used for anything else (tone or servo or PWM) in the Arduino core.

The big advantage is that you do not any longer need an external pin for the software interrupt, since it can also be generated by using the same compare interrupt. Well, unfortunately this is only true for the flash breakpoints. With RAM breakpoints you start to advance the timer counter and then you spend all your time in the delay function :-(. So, this is not an option.

I agree that the statement by the author of the web page I mentioned is a bit silly. I just wanted to show it you so that you can see how easily one can have something like a parsitic interrupt. BTW, the time spent in the ISR is 39 µs (that is what my logic analyzer says), i.e. 4% of the CPU time. I would consider that as neclectable. You always have to pay some price for debugging. However, you are, of course, correct that this could break bit banging software. But this is also possible if you do it only every 500ms. And then it might be even harder to detect that this is an artifact of the debugger. So, I guess, the real problem is here that the ISR takes much too long.

If I am not mistaken, then one could do two quick checks, namely, is the current instruction a TRAP_OPCODE or are we single-stepping. Only if one of those conditions is true, then one starts seriously executing the ISR. Well, I tried it out (without touching the save and restore register functions) and saved 8 µs. So the key in shortening the execution time of the ISR would be to save only a few registers and then test the above conditions using inline assembler. Pew! This is not one of my favorites. In particular because the access to flash memory is different for the different type of MCUs.

Let me just tell you, how the extension works. If you define AVR8_USE_TIMER0=1 and you use flash breakpoints (and only then!), then instead of the WDT interrupt, the output compare interrupt A of timer 0 is used. In addition, instead of raising a software interrupt using external interrupts, I execute OCR0A = TCNT0 twice, which guarantees that the compare flag is set and an interrupt is raised when the interrupts are enabled globally again. This means that both cases end up in the same ISR, but that is not a problem. One just has to check whether it is a breakpoint or whether we are single stepping.

To make a long story short. I guess it might be another possible alternative with its own trade offs and people might find it useful. On the other hand there is, of course, always the possibility that people get confused if they have too many options.

Best, Bernhard

[felias-fogg]

Hi Jan,

following up on yesterday's discussion ...

one could do two quick checks, namely, is the current instruction a TRAP_OPCODE or are we single-stepping.

... I couldn't take my fingers off the keyboard.

For the ATmega328, I now have an inline assembler routine that checks whether we are possibly at a breakpoint or do single stepping. So the runtime for the timer interrupt is now down to 2.5µs. This is the same order of magnitude as Arduino's millis interrupt. For the other MCUs, the code is probably a bit more complicated and may take slightly more runtime (and, of course, more time to develop and test).

Best, Bernhard

[jdolinay]

Hi Bernhard,

OK, what you say makes sense. It is good to have another option. I will try it.

One more thought to the discussion - the priority of the interrupts. I wonder if using COMPA may cause any trouble with the single stepping or when trying to debug an ISR. The external INTx has the highest priority, so in the single stepping mode it is always executed first after each instruction of the program (there is this feature of the AVR that after return from ISR one instruction of the program is executed before next ISR is serviced). Now if there is the COMPA interrupt waiting for execution but also another interrupt with higher priority, the higher priority interrupt will be serviced instead of COMPA... I just wonder if it is good thing or bad or if it makes no difference. Also what effect will it have (if any) on debugging ISR code.

Best, Jan

[felias-fogg]

Hi Jan,

I haven't put much thought into interrupt prios and actually was not aware of them since usually you just want to make sure that all interrupts are served.

But for debugging it makes, of course, a difference. So you probably cannot single step into an ISR. However, that was already the case with using the INTx interrupts (if you use, e.g., INT6 and want to debug the INT1 ISR). If you want to debug an ISR, you better put a breakpoint into it, I guess. BTW, debugging ISRs is always a very delicate matter. If you make them non-blocking, you risk that they are called again, which is usually something, they are not programmed for. It is particularly bad, if you do this with a level-interrupt. If the level is still there (e.g. the INT0-pin is still low), then the ISR is called as often as there is free space on the stack. If you want to debug such an ISR, you better disable the particular interrupt, clear the flag, and only then enable interrupts globally. And when you leave the ISR, you have do it in the reverse order. In other words, the manual should probably contain some more words of caution when it comes to debugging ISRs.

On a related matter, I noticed that when you enable an interrupt with the flag already set, it can take up to four instructions before the MCU stops and starts to service the interrupt. This is something you mentioned as well, right? How does that go together with single-stepping? I guess, gdb takes care of that.

Best, Bernhard

[felias-fogg]

Hi Jan,

the "quick check" code now works for all supported MCUs. In the end it is less complicated than I thought. But, of course, it always takes a lot of time to program and test it. And debugging a debugger is definitely not the easiest thing in the world. I tested in particular whether it works in the "upper" regions of the bigger processors. And it does.

For the ATmega328, the ISR takes 2.6µs, for the ATmega2560 it is 3.9µs, which sounds quite good compared to the 38µs we had before. If it now breaks bitbanging code, then the ordinary millis interrupt will probably do the same.

It is highly modular in that it does not interact with rest of the code. So one can easily deactivate the code.

I have cancelled the former pull request and will come up with a new one. And then I have to take a rest from hacking the debugger!

What I really want to see is a cheap and portable debugWire debugger since with that you also can debug the ATtinys (not the big ATmegas, though). The hardware could probably be a cheap DigiSpark. And the software could be based on your avr8-stub and on other people's work on reverse engineering the debugWire protocol. It sounds like a lot of work, though. So, I probably won't start to work on it anytime soon.

Cheers, Bernhard

P.S.: And I am not so sure about the latency of the interrupts anymore. Maybe I misinterpreted something.

[jdolinay]

Hi Bernhard,

The quick "check code" times sound great. I didn't look at the last pull request yet, but I did some tests to confirm how it is with the interrupt priorities and I am afraid there is a problem with the single stepping using the COMPA interrupt. It affects stepping in the program in general, not just debugging ISRs. Let me explain.:

When single stepping, the INTx interrupt is always enabled and the pin is held low, so the interrupt request is always pending. So when the ISR finishes, it will be immediately started again. But thanks to the feature of the AVR architecture that after each return from ISR one instruction of the interrupted program is executed before any pending ISRs, the debugged program is advanced by 1 instruction before the ISR is executed next time. This way the ISR can examine the PC after every instruction of the debugged program...

If there is more than one interrupt waiting to be serviced, the one with highest priority (lowest vector number) is executed. After it finishes, 1 instruction of the interrupted program is executed and again the pending ISR with highest priority is executed.

Because INTx has the highest priority, it is always selected so it can examine the debugged program after every instruction. As a side effect, no other interrupts are serviced. This is already documented in the doc in the "Debugging code with millis and micros" section.

The problem is that the COMPA of timer0 has relatively low priority. If there is interrupt with higher priority used in the program, it will be serviced first, then one instruction of the debugged program is executed and then COMP0A ISR is executed. So the debugger will miss one instruction of the program on every step command. If there are 2 higher priority interrupts, it will miss 2 steps, etc.

You may not notice this if you debug code like the Blink example because stepping over a function is actually done by the gbd by inserting breakpoint after the function call, so you are actually running to breakpoint and not stepping. But you can see it if the code is doing simple operations like setting bit in a register, i++; etc.

I created test program for this and used it with your code with the AVR8_USE_TIMER0=1 and it confirms the problem.
In the program there is Timer2 overflow interrupt. This is what happens when debugging such program:

• the user issues step command.
• let's say there is no timer2 interrupt pending, so COMPA ISR is executed and program stops at the next instruction as expected.
• now before the user issues next step command the timer2 interrupt is requested. Almost any interrupt used in the program is likely to be requested because it will take few seconds for the user to click the Step button again (after examining variables etc.).
• user issues step command.
• the COMPA ISR ends, and 1 instruction of the user program is executed.
• timer2 ISR is executed because it has higher priority. After it ends, 1 more instruction of the user program is executed.
• COMPA ISR is executed and it communicates with the gdb. The user sees the program stopped after 2 lines of code which is not good. The above is true if one line of the C program equals one instruction. I haven't tested it but it may be even worse if the line in C equals several instructions. When user clicks Step, gdb actually sends step command to the stub repeatedly and tests if the address where the program stopped is the address where the assembly code generated for the C-line ends. If not, it sends another step command etc. But if higher priority ISR causes that the final instruction is missed by the stub, then the program will be stopped at unexpected location. I actually saw such situation while testing the program several times.

I guess there is not much to do about it, but it puts the advantage of saving the INTx pin to a different perspective. I think if this option should be available, for those who need the pin really badly, it should be separate option, not "bundled"with the AVR8_USE_TIMER0 option and well documented that it should be used only in programs that use no higher priority interrupts. This way one could use the watchdog by enabling the AVR8_USE_TIMER0 option without loosing the ability to debug the program... But thinking about it, the watchdog interrupt is also higher priority than timer0, so perhaps this pin-saving option does not make sense at all...because if you need watchdog you cannot use the COMPA stepping option and if you don't need watchdog, you don't need the COMPA option.

About the debugWire

debugWiWhat I really want to see is a cheap and portable debugWire debugger since with that you also can debug the ATtinys (not the big ATmegas, though). The hardware could probably be a cheap DigiSpark. And the software could be based on your avr8-stub and on other people's work on reverse engineering the debugWire protocol. It sounds like a lot of work, though. So, I probably won't start to work on it anytime soon.

I recently came across this project: https://github.com/dcwbrown/dwire-debug Haven't looked at it yet but it looks this could be it.

Here is the code of the test program:

const uint8_t LED = 11;
void timer2_init();

void setup(void)
{
    debug_init();
    pinMode(13, OUTPUT);
    pinMode(LED, OUTPUT);
    digitalWrite(LED, LOW); // turn off
    pinMode(12, OUTPUT);    // PB4
    digitalWrite(12, LOW);  // turn off

    // configure timer2 for overflow interrupt
    timer2_init();
}

void loop(void)
{
    //digitalWrite(13, HIGH);
    PORTB |= (1 << PORTB5); // pin 13 high
    PORTB |= (1 << PORTB4); // pin 12 high
    delay(100);
    //digitalWrite(13, LOW);
    PORTB &= ~(1 << PORTB5);    // pin 13 low
    PORTB &= ~(1 << PORTB4);    // pin 12 low
    delay(400);
}

void timer2_init()
{
  cli();  // disable interrupts
  TCCR2A = 0;
  TCCR2B = 0;
  TCNT2  = 0;
  TCCR2B |= (1 << CS22) | (1 << CS21) | (1 << CS20);
  // Enable timer overflow interrupt
  TIMSK2 |= _BV(TOIE2);
  sei();  // enable interrupts
}

// ISR for timer2 overflow
ISR(TIMER2_OVF_vect )
{
  static uint8_t toggle1 = 0; 
  static uint8_t counter = 0;
  counter++;
  if ( counter >= 30 ) {
    counter = 0;
    if (toggle1){
      digitalWrite(LED, HIGH);
      toggle1 = 0;
    }
    else{
      digitalWrite(LED, LOW);
      toggle1 = 1;
    }
  }
}

Try it with the timer2_init(); line commented out -stepping through loop should work fine. Then try with the line enabled and set breakpoint at the 1st PORTB line in loop. When you step over, the program should stop at delay() instead of the 2nd PORTB line.

BTW: if you try to step into the delay() function you may be able to see the program skipping over the ms--; line in the delay code:

while (ms > 0) {
        yield();
        while ( ms > 0 && (micros() - start) >= 1000) {
            ms--;
            start += 1000;
    }

Best, Jan

[felias-fogg]

Hi Jan,

you are right. The interrupt prios can mess up the stepping capability to a certain extend. As I mentioned before, this was already the case when you allowed to use the INT6 interrupt, but it definitely becomes more prominent with using the COMP0A interrupt as the software interrupt. So, I agree one should separate the usage of the COMP0A interrupt as a software interrupt from using it as a substitute for the WDT interrupt. I was actually thinking about it after I had implemented it.

When it comes to using another interrupt as the software interrupt, one could actually use also the PCINTx and TIMERx_COMPy interrupts. I am not sure whether it is worthwhile to consider all the cases because you either have a conflict with another external pin or a conflict with Arduino's tune, PWM, or servo functions. That was the reason, I loved the COMP0A interrupt so much.

In summary, I believe one could just add the "COMPnx" interrupts as possible software interrupts and you then have to specify the appropriate string instead of the number of the external interrupt, e.g.,

define AVR8_SWINT_SOURCE "COMP0A"

One should then point out the possible problems with the stepping function (and the compiler will raise an error if an interrupt vector is used more than once). I could add that in the code and in the manual if you want. Then there is also the question of merging the two respective interrupt functions. I did that already for the USE_TIMER0 case and think there is no real reason not to do it in general. Except you have one.

Concerning the usefulness of the TIMER0 interrupt when debugging WDT, I disagree. The WDT is either used as a timer in low power applications, i.e., when the MCU is powered down, or as a guard against infinite loops (sometimes for both purposes). So, exact timing doesn't play a role and in particular, when stopped during debugging, the WDT timer should not advance. For this reason, I put a lot of wdt_reset() calls into the debuggers code. It just does not make any sense to raise a WDT interrupt or reset while in debugging mode! In other words, when single-stepping or continuing, the WDT interrupt will never be raised and so the higher priority of the WDT interrupt over COMP0A is irrelevant. Along the same lines, one might think about clearing the respective timer interrupt flags when single stepping. This means, when one uses timer 1, all possible interrupt flags of timer 2 are cleared. What do you think?

I have also stumbled over the dwire-debug package, which looks quite promising. There are, however two problems. The gdb-server runs on the host system. This means you have to maintain the code for all relevant platforms (Linux, Windows, Mac). The second problem is that it allows only for one breakpoint. There is another implementation for the Arduino Uno https://github.com/wholder/DebugWireDebuggerProgrammer/blob/master/DebugWireDebuggerProgrammer/DebugWireDebuggerProgrammer.ino , however this debugger does not interface to gdb. So, the ideal system is not there yet. I dream of a gdb-server on an existing Arduino board (e.g. Nano/Uno), which implements the gdb-server protocol and that is able to switch an AVR MCU into debugWire mode using the ISP protocol. Afterwards it communicates with the MCU in debugWire mode, and finally the Ardunio debugger switches the target back to ISP mode and reprograms the DWEN fuse.

Cheers, Bernhard

[felias-fogg]

I now have now a version where you could set AVR8_SWINT_SOURCE to -1 (meaning to use the COMP0A interrupt) and AVR8_USE_TIMER0_INSTEAD_OF_WDT to 1 (or 0, default). I'll post a new pull request soon. I am still wondering whether one could merge the ISR for WDT/TIMER0_COMPA and the AVR8_SWINT interrupts. I believe they just differ in how BREAKPOINT_MODE == 1 is handled. But things got a bit messy there.

[jdolinay]

Hi Bernhard,

Sorry, I am a bit behind in responding to your messages, not to mention reviewing the code. Thanks for the link to the doc in the last message, you are right, I'd like to go over it and probably rewrite it in my words so that I can understand it :-)

Yes, the stepping is really "sensitive" to interrupt priorities. I never actually thought about it in such details as now. I think for the "end-user" this could be summed up like "For the debugging to work properly the interrupt used as the software interrupt for stepping should have the highest priority in the system."

It is true that various interrupts can be used for this, but I also think this is not worthwhile to implement. I'd say it's nice-to-have feature with low priority. It would mean lot of #ifdefs in the code, making it hard to maintain (I know it's a mess already and could use good clean up:)). Plus then you should test all the combinations with every release... So weighting the benefit (increased flexibility which probably few people need) with the risks (harder to understand code, higher risk of errors), I'd just stay with the INTx options.

You are right about the usefulness of TIMER0 with WDT, I took a "shortcut" in my statement, not really thinking it over. I actually only used WDT as a guard few times and never debugged code with it, so I am slow in getting the idea. Could you explain your scenario in which you need to debug a program and use WDT at the same time?

About the timer flags I am not sure if I understand the purpose - is it to "stop" the timer ISRs from running while debugging? If yes, then I would say this is also a feature that could be useful for some people (I think I saw the option to "stop timers while debugging" in some "real" debugger), but I'd prefer to keep the stub code simple. Those who need it can modify their program or the stub themselves. I think there is a point in making things too configurable when just figuring out the proper configuration is harder than implementing the feature yourself.

I have 2 questions about the note you added to the doc about breakpoints in ISRs (in the Quickcheck version):

..it might not be possible to single-step into an ISR

How does one single-step into ISR? I can only think of setting a breakpoint in ISR and continuing the program to get myself to the ISR for debugging.

2. About the ISRs being called again: Probably we see it the same way, but to clarify... If I set BP in ISR and the program stops on it, it is in the context of the ISR that checks for the BP (WDT). This ISR is normally blocking, so no other ISRs are executed as long as the program is stopped or stepped in the ISR. Once I click continue, the pending ISR will be executed inside the current invocation. It will stop on the BP again which the user will expect - if the ISR is set to happen e.g. every 100 ms and he spent 10s debugging it. What the user will not be aware of is that the ISR is actually nested and the previous invocation is not finished... If I remove the BP and continue the program the nested invocations will quickly finish and stack will be freed.

So, if I get it right, the main problem the user should be aware of is not that the stack is used up by this nesting (because there will be only as many levels of nesting as many times I click the continue button with BP in the ISR), but that if I continue from somewhere in the middle of an ISR, the code after that point will not be executed before I get back to the BP at the beginning of the ISR. Do you agree with this?

Thanks for the notes on the dwire-debug package. I am fan of open-source, so I am interested in such projects but I also have the AVR Dragon board, so it's more like a hobby to follow them. Looking at it from the perspective of someone who is not that much of a fan of AVR, why not switch to a different architecture which has proper debugger for reasonable price, like STM32? I haven't tried STM, so there may be some problems, but certainly the ARM is the future and 8-bit AVR is somehow obsolete. It seems 8-bit MCUs are going to be very rare in few years; the AVR may survive little longer thanks to Arduino, but even Arduino users are moving to more powerful 32-bit MCUs.

Best, Jan

[felias-fogg]

Hi Jan,

Just a short answer to your last question/comment.

Well, I do all of that as hobby projects - being a computer scientist (as you are) nonetheless ;-). And I use the AVRs mainly for small gadgets that should be as power efficient as possible, running on one battery for 10 years or so. And I guess in this league, the AVRs will prevail. ARMs are much faster and larger, but they are also much more power hungry, as far as I have seen. AVRs can run on 100nA in powerdown mode and can run on 2-3mA if the clock is reduced to 1MHz.

It is true that also hobbyists are moving to 32-bit ARM MCUs, and the German Make magazine declared that Micropython is the new Arduino. And I am all for making programming easier and more efficient, even if the programs are slower and much bigger. The human time is so much more valuable than computer time! However, as I said, when you really want to be low power, then I believe, the 8-bit AVRs are the way to go. And if you think about it, most of the things you want to control with an MCU easily fit into 32 MB.

I bought once an Atmel ICE, but then noticed that because of the brain damaged way Apple restricted access to kernel drivers and Atmel designed the USB interface, I couldn't use it on my Mac. I tried Atmel Studio using a Windows VM, but was completely put off by the user interface. So, I guess, I really would love a low cost debugWire/GDB piece of hardware, which should not be to difficult to build. But you are probably right that not too many people will be excited by it because most are moving to ARM processors.

Cheers, Bernhard

[jdolinay]

Hi Bernhard,

OK, thanks for explaining. I wasn't aware of this use of AVRs though I've heard that ARM is not so power efficient. It's good to learn something new. Just to let you know, I pulled in your changes to my local branch (another new thing I learned :) ) and started "processing" the changes. It will take me some time but in the end I think they will be there.

Cheers, Jan

[felias-fogg]

Hi Jan,

let me answer the remaining questions.

Yes, the stepping is really "sensitive" to interrupt priorities. I never actually thought about it in such details as now. I think for the "end-user" this could be summed up like "For the debugging to work properly the interrupt used as the software interrupt for stepping should have the highest priority in the system."

I agree. Nevertheless, even in other cases you probably get reasonable results. You should simply be aware.

It is true that various interrupts can be used for this, but I also think this is not worthwhile to implement. I'd say it's nice-to-have feature with low priority. It would mean lot of #ifdefs in the code, making it hard to maintain (I know it's a mess already and could use good clean up:)). Plus then you should test all the combinations with every release... So weighting the benefit (increased flexibility which probably few people need) with the risks (harder to understand code, higher risk of errors), I'd just stay with the INTx options.

Yes, so the only other options I favor now is the TIMER0_COMPA one (which you get when setting AVR8_SWINT_SOURCE to -1).

You are right about the usefulness of TIMER0 with WDT, I took a "shortcut" in my statement, not really thinking it over. I actually only used WDT as a guard few times and never debugged code with it, so I am slow in getting the idea. Could you explain your scenario in which you need to debug a program and use WDT at the same time?

In a recent project, I used the WDT counter as a time keeper and a watch dog to catch I2C stuck situations. The WDT ISR looked as follows:

ISR(WDT_vect) { wakemillis = millis(); // remember millis if (episode == SETUP_EPISODE) { // wdt during setup! wdt_enable(WDTO_15MS); } else { quartersecs++; if (wdtcnt++ > 3) { // stuck during executing main! wdt_enable(WDTO_15MS); } else WDTCSR |= (1<<WDIE); // re-enable watchdog interrupt } }

About the timer flags I am not sure if I understand the purpose - is it to "stop" the timer ISRs from running while debugging? If yes, then I would say this is also a feature that could be useful for some people (I think I saw the option to "stop timers while debugging" in some "real" debugger), but I'd prefer to keep the stub code simple. Those who need it can modify their program or the stub themselves. I think there is a point in making things too configurable when just figuring out the proper configuration is harder than implementing the feature yourself.

I agree. Don't put too much stuff into the debugger. For a hardware debugger there is much more control possible. But here we should as minimal invasive as possible.

I have 2 questions about the note you added to the doc about breakpoints in ISRs (in the Quickcheck version): 1.

..it might not be possible to single-step into an ISR

How does one single-step into ISR? I can only think of setting a breakpoint in ISR and continuing the program to get myself to the ISR for debugging.

This was definitely written without thinking too much. If you have a higher priority interrupt as the software debugging interrupt, you never single-step into an ISR. Only in the case, that the debugger interrupt has a lower priority, this can happen. But I agree, usually you will set a breakpoint in the ISR in order to debug it.

1. About the ISRs being called again: Probably we see it the same way, but to clarify...
   If I set BP in ISR and the program stops on it, it is in the context of the ISR that checks for the BP (WDT). This ISR is normally blocking, so no other ISRs are executed as long as the program is stopped or stepped in the ISR.
   Once I click continue, the pending ISR will be executed inside the current invocation. It will stop on the BP again which the user will expect - if the ISR is set to happen e.g. every 100 ms and he spent 10s debugging it.
   What the user will not be aware of is that the ISR is actually nested and the previous invocation is not finished... If I remove the BP and continue the program the nested invocations will quickly finish and stack will be freed.

So, if I get it right, the main problem the user should be aware of is not that the stack is used up by this nesting (because there will be only as many levels of nesting as many times I click the continue button with BP in the ISR), but that if I continue from somewhere in the middle of an ISR, the code after that point will not be executed before I get back to the BP at the beginning of the ISR. Do you agree with this?

The short way of saying, why debugging ISRs is delicate is that ISRs are usually not written to be interruptable (global interrupts are usually disabled). And they are certainly not written to be reentrant. So, if you allow interrupts globally, you should definitely make sure that the same interrupt cannot happen while executing the ISR. And even other interrupts can mess up things when they operate on the same (volatile) variables, because the interrupt might happen during the assignment of word variable.

Cheers, Bernhard

[jdolinay]

Hi Bernhard,

Thank you for the answers and also thank you for your contribution. I admit I was not 100 percent convinced about the changes, but after going through the code I am convinced they will be useful; not to mention the bugs/problems you fixed. So thanks again!

Cheers, Jan

[felias-fogg]

Just a quick note on the "ugly kludge": https://www.avrfreaks.net/forum/address-stack-has-always-bit

So, the ATmega16 had this problem. However, this seems to be the only one.

[jdolinay]

Hi Bernhard, Thanks for the link. Great to know the source of this! I looked at the Atmega16 datasheet and found the text about masking out unused bits in PC on stack they mention in the discussion. There is no such text for other AVRs I checked - Atmega168, 328 and 2560.

Now I wonder if Atmel really fixed it in the other variants, because in principle this could affect any AVR with PC shorter than 16 bits, including the Atmega328. It seems so, as it works in the stub without the masking. I may try to look at the PC pushed on stack via debug wire debugger just to be sure...

[felias-fogg]

Hi Jan,

So, I guess, I really would love a low cost debugWire/GDB piece of hardware, which should not be to difficult to build.

just to follow up on my wish: Now I made it happen. Perhaps you would like to try it out? You find it under https://github.com/felias-fogg/debugWIRE-probe

Cheers, Bernhard

[ccrause]

Even more simple (and possible a bit faster) is to use a USB-serial converter (and a diode or resistor) and a PC based gdbserver: https://github.com/dcwbrown/dwire-debug. There is also a Pascal version (warning: self promotion): https://github.com/ccrause/debugwire-gdb-bridge

[jdolinay]

Hi Bernhard, That's impressive work! I tried it with Atmega168 but so far I did not get past this error:

PlatformIO Unified Debugger -> http://bit.ly/pio-debug
PlatformIO: debug_tool = custom
PlatformIO: Initializing remote target...
0x00000000 in __vectors ()
.pioinit:15: Error in sourced command file:
Protocol error with Rcmd

There may be something wrong with my wiring but the "Protocol error with Rcmd seems strange"... I get the same error when trying from command line.

@ccrause thanks for the tips. I think Bernhard tried the dwire-debug so he could say more. I just looked at it and it's interesting to have it working with just USB-serial converter but if I understand it well it uses its own protocol so it cannot be used with gdb (?). I guess this is what your Pascal version solves, but to tell the truth I would not know how to use it without some step-by-step tutorial.

Cheers, Jan

[felias-fogg]

Hi ccrause,

Even more simple (and possible a bit faster) is to use a USB-serial converter (and a diode or resistor) and a PC based gdbserver: https://github.com/dcwbrown/dwire-debug. There is also a Pascal version (warning: self promotion): https://github.com/ccrause/debugwire-gdb-bridge

Thanks for the pointer. I had not noticed the Pascal version before. My main intention was to get something something that is platform independent. As I have written in the manual, dwire-debug does not work under macOS.

[felias-fogg]

Hi Jan,

PlatformIO Unified Debugger -> http://bit.ly/pio-debug
PlatformIO: debug_tool = custom
PlatformIO: Initializing remote target...
0x00000000 in __vectors ()
.pioinit:15: Error in sourced command file:
Protocol error with Rcmd

This is the reply when the debugger cannot connect to the target. I added now a message saying that explicitly. Reasons are: MCU does not support debugWIRE or wrong wiring. Since the 168 chip is supported, and I double checked the signature in the program, I suspect wrong wiring. But then there is always another possible cause such as a missing comma in the program text ;-).

Cheers, Bernhard

[jdolinay]

Hi Bernhard, Thanks. Now I was able to connect. The problem in my case seems to be in powering the target from pin 9, as shown in your manual. In this case I get the connection error. If I power it directly from Arduino 5V, then connection works.

However, it seems the program is not loaded. There is no error, everything runs well, same as in your example, but I suspect the program is not loaded to memory. Here is why:

• First, I tried according to manual and all went without error, but the program was in __vectors all the time after continue and break. It was not really running, LED not blinking.
• Then I loaded the program using AVR Dragon over debugWIRE (debugging it). It worked fine, LED blinking.
• Next I connected with your debugger, using the same .elf file for gdb and it worked.
• Next I changed the program (LED blinking faster) and loaded it using your debugger. I was able to step somehow but to wrong locations - most likely the program really in memory is different than the gdb thinks. Also the LED is blinking still at the same slow speed.

So this makes me think the program is not really loaded. First, there was no program and after loading it via AVR Dragon there is still the same program.

Could it be related to the fact that I am not using Arduino framework and bootloader in the target? It is just Atmega168 in breadboard.

Best, Jan

[felias-fogg]

Hi Jan,

thanks for the feedback. Indeed, automatic power-cycling is not implemented yet (something I missed). And flash programming of ATmegas is also untested. It is a little bit different from flash programming the ATtinys. So, it is quite possible that there is bug. I will have a look.

Cheers, Bernhard

[felias-fogg]

Hi Jan,

I added a lot of stuff (loading code into ATmegas, power-cycling, 16 MHz targets) and it works most of the time. However even under sort of very heavily tested conditions, sometimes very strange things happen, for which I do not find a reason. Timing appears to be extremely important. So I decided, to give up - which is not easy having spent quite a few days in programming and testing. So, I deleted the rep.

Best, Bernhard

[jdolinay]

Hi Bernhard,

I am sorry to hear that. I was hoping it would work; I could imagine a small box with Arduino nano inside acting as a debugWire probe...

When looking at the code when I tried it earlier, I had a feeling you put a lot of effort into implementing many features (like preventing flash wear) which are nice-to-have but not absolutely necessary - and can introduce bugs (I mean the more lines of code, the more likely there will be a bug). I'd try stripping the code down to bare minimum, but that's easier to say than done...

Maybe after a break from the project for you would find the problem, or maybe someone else. I hope the repo can be recovered in case you change your mind.

Best, Jan

[felias-fogg]

Hi Jan,

I had another go at it, this time trying to build up things from the bottom. I created a new serial communication class SingleWireSerial that is extremely accurate. I also designed a number of unit tests for the basic debugWIRE functions that led to the 'discovery' that all the small ATTinys support CALL and JMP despite what the Atmel documents claim: https://www.avrfreaks.net/forum/strange-observation-jmp-and-call-are-legal-instructions-attinys .

In any case, I am making some headway. BTW: The error you mentioned in trying to get the debugger to communicate with your mega168 (https://github.com/jdolinay/avr_debug/issues/27#issuecomment-856986665) is something I also had to fight. I decided to buy (almost) all of the AVR MCUs that support debugWIRE and got some surprises when testing the debugger on them. It turns out the ATmegaX8 AVRs with 16 KiB flash or less seem to require a particular way of setting the fuses, something I have not seen at other debugWIRE compatible chips. Avrdude is successful in changing fuses on them, so I just copied the way avrdude does it. Very strange. And I was not able to find anything documented or mentioned about that on the web.

I am currently working on building a test suite that can be applied semi-automatically and on designing the adapter board with level shifters and automatic power-cycling. I hope, I can count on you as the alpha tester in the not too distant future ;-)

Cheers, Bernhard

[jdolinay]

Hi Bernhard,

I am glad to hear that! It sounds promising. Sure you can count on me as the alpha tester :)

Cheers, Jan

[felias-fogg]

Here we go: Before the old year ends, I have it "released": https://github.com/felias-fogg/dw-link

Interestingly, some of the things we discussed came up also in this hardware-debugger: single-stepping into ISRs, dirty bits in program counters, you name it.

In any case, would love to hear your feedback Bernhard

[jdolinay]

Great news! I will try it and let you know. Jan

[jdolinay]

So I tried it with Atmega168 and it works just great! I used it with PIO and it's really comfortable to just click the debug button and in few seconds you are debugging. :) I will play with it more and I'd also like to build something like the "real" hardware debugger you describe in the doc later.

About the doc, first I have to say it is great, very detailed. I have 3 notes:

• in the 4.2 section there is a pin D11 listed both for MISO and SCK. The SCK is actually D12.
• It would be nice to have a table with the pins there to quickly find the pin numbers.
• the manual pdf is really huge - about 30 MB which seems to be because of the pictures, mainly the dwprobe1.0-in-action.png. I think the images could be resized and perhaps saved as .jpg to make the whole archive much smaller (and faster to download).

Thank you for making this excellent piece of work available! Jan

[felias-fogg]

Hi Jan,

thanks for the feedback! I guess, the docs are half the battle. Without it, nobody will try it out. I took your project as an example for it ;-).

And thanks for pointing out the problems in the docu. I fixed them. The next thing will be to port dw-link to ATmega32U4 boards, which is a bit more complicated than I thought initially. There are these nasty USB interrupts all the time.

Another thing: I tried to make dw-link/avr-gdb work with gdbgui, but was not successful. Did you try this system at some point? The nice thing is that it is a platform and OS independent GUI for GDB.

Cheers, Bernhard

P.S.: If you are interested, I can send you a fully assembled dw-probe V1.0 board.

[jdolinay]

Hi Bernhard,

you are welcome :) Yes, I also think that it's important to make it easy for the users by writing good docs. Now to let people know about your debugger you could write an article on CodeProject about it.

I never heard of gdbgui but I bookmarked it now and will give it a try. It looks interesting.

As for sending the probe, it's nice of you but I don't want to take advantage of you. To tell the truth when I read the doc yesterday I didn't notice that you also provide the files in the pcb section so one can build the probe without much effort. You put really lot of work into this!

Cheers, Jan

[felias-fogg]

Hi Jan,

ah! I noticed that you are a regular CodeProject author ;-). Sounds like a good idea! I'll give it a try.

Concerning the probe: I already have 2 lying on my bench (the version 1 types) and tomorrow the version 2 PCBs will arrive. In other words, it would not be exactly taking advantage of me. If you want to pay for the postage, you can send me the money via Paypal. So, if change your mind, drop me a note.

I also noticed from your replies to others on Github, that you are now more leaning towards PlatformIO. Did you abandon Eclipse?

Cheers, Bernhard

Am 06.01.2022 um 21:37 schrieb Jan Dolinay @.***>:

Hi Bernhard,

you are welcome :) Yes, I also think that it's important to make it easy for the users by writing good docs. Now to let people know about your debugger you could write an article on CodeProject about it.

I never heard of gdbgui but I bookmarked it now and will give it a try. It looks interesting.

As for sending the probe, it's nice of you but I don't want to take advantage of you. To tell the truth when I read the doc yesterday I didn't notice that you also provide the files in the pcb section so one can build the probe without much effort. You put really lot of work into this!

Cheers, Jan

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.

[jdolinay]

Hi Bernhard,

I see you already put the article on CodeProject. That's great!

It's a good question about Eclipse. When I think about it, I haven't used it for a long time. But it's also true that I didn't do anything I'd call embedded project for a long time. For small experiments I now really prefer PlatformIO as it is very easy to setup. I cannot say if I would choose Eclipse or PlatformIO for some larger project. Probably if it was Arduino based, I'd try PlatformIO first. But there is so much other work and distraction that it's hard to even imagine finding time for such a project :) That's also the reason I think the probe would be just lying on my desk for now. But I will definitely drop you a note if in the future I have more time for the embedded things. I suspect by that time there will be at least version 3 ;-)

Cheers, Jan

[felias-fogg]

Hi Jan,

I took the opportunity to submit my recent blog posts to them. Overall, I might have done it with one post to try what they do, since in general the articles look much better on my website.

I can fully understand your situation -- being myself an (almost -- three months to go) retired university professor. Even now, there are more things to pursue than I can handle. Let alone the things I really want to do. But I am getting closer ... ;-)

So, one of the things I plan is to write a CodeProject article on "Debugging your Arduino project on AVR MCUs using GDB". The idea is that it should be a tutorial with the idea of introducing avr_gdb for larger ATmegas and dw-link for ATtinys and ATMexgaX8s. The general setup would be the same. Using the different tools is then different. GDB usage again is the same. CodeProject by itself does not seem to support co-authored articles, do they? In any case, I would like to have you as a co-author.

Cheers, Bernhard

Am 07.01.2022 um 22:36 schrieb Jan Dolinay @.***>:

Hi Bernhard,

I see you already put the article on CodeProject. That's great!

It's a good question about Eclipse. When I think about it, I haven't used it for a long time. But it's also true that I didn't do anything I'd call embedded project for a long time. For small experiments I now really prefer PlatformIO as it is very easy to setup. I cannot say if I would choose Eclipse or PlatformIO for some larger project. Probably if it was Arduino based, I'd try PlatformIO first. But there is so much other work and distraction that it's hard to even imagine finding time for such a project :) That's also the reason I think the probe would be just lying on my desk for now. But I will definitely drop you a note if in the future I have more time for the embedded things. I suspect by that time there will be at least version 3 ;-)

Cheers, Jan

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.

[jdolinay]

Hi Bernhard,

I am not sure if CodeProject supports co-authored articles. I think I can say I write there just for fun, when there is something I want to share. For my university such articles are worthless :( They are only interested in articles in those journals indexed in the right databases...

Talking about debugging Arduino projects I am actually starting to think that the problem is not that there is no built-in debugger, but that most Arduino users don't even miss it. I meet many students who are satisfied with printing test messages or just modifying and re-running the program to see if it works. So I wonder how long it will take before debugger becomes the tool for few experts instead of the standard tools of a software developer - something like the assembly language is now :)

Cheers, Jan

[felias-fogg]

Hi Jan,

Talking about debugging Arduino projects I am actually starting to think that the problem is not that there is no built-in debugger, but that most Arduino users don't even miss it. I meet many students who are satisfied with printing test messages or just modifying and re-running the program to see if it works. So I wonder how long it will take before debugger becomes the tool for few experts instead of the standard tools of a software developer - something like the assembly language is now :)

Agreed! Most Arduino hobbyists don't miss debuggers. Most of the time, they do quite simple things and printf-debugging is all what they need and what they care about. Actually, when I develop a small Python script, that's what I do most of the time. When things can be solved with simple tools, why learn about a complicated debugger and how to set it up.

When it comes to CS students, I'ld say that they should be aware of the tools of the trade and know how to use them. Printf-debugging has severe restrictions and can mislead you in a number of ways. So, it may hide timing dependent bugs. It may suggest that you did not reach a program location because printing is performed in an interrupt-driven way and interrupts are off or the program dies before even the first letter is printed.

I believe in reasonably complex embedded systems with reasonably nasty bugs, only an embedded source code debugger and a logic analyzer will help you to locate bugs in reasonable time. With nasty bugs, I mean, e.g., timing dependent bugs, buffer overflows, stack overflows, dangling pointers. The latter kinds of bugs can be found with watchpoints (watching a particular memory cell) quite easily, with single-stepping (using the display command) also reasonably easy, while printf-debugging is at a loss here, except you surround each statement with a print-statement.

If you take, for example, one of my test programs in the dw-link repository: fibonacci.ino. It computes the Fibonacci number 7, first by using just the recursive definition getting the right result 13; second by using a memoization technique, i.e., remembering already computed values of the function. It looks all correct, but the result of the memoizing version is 1 instead of 13. If you set a watchpoint at memo[7], you will catch the bad guy very easily. Problem with the AVR MCUs is that they do not support watchpoints by hardware, so avr-gdb does a software version of it, resulting in internally single-stepping, which is not very efficient.

In any case, I do not believe that source code debuggers will become as specialized as assembly programming is right now. People, who seriously develop software (embedded or not), will always need advanced debugging and testing tools, I believe.

BTW: Some good reads on debugging are "Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems" "The Developer’s Guide to Debugging" "Why Programs Fail: A Guide to Systematic Debugging"

Cheers, Bernhard

P.S.: Just completed the test set of the classic ATtinys and implemented communication with super-slow MCUs (16 kHz CPU clock). So you now can set the clock source to the internal 128 kHz clock and program the CKDIV8 fuse. I am not sure if this is any good at all ;-).

[jdolinay]

Hi Bernhard,

Thank you! As usual you give me a lot of things to think about and lot of inspiration ;-) I will look at the links. I also believe that debugging is something CS students should learn about, so I am glad that your dw-link debugger exists. It opens the way to teach debugging in courses based on the cheapest Arduino boards with minimal investment. I am not sure when, but I plan to try it "on students" :-)

Cheers, Jan

[felias-fogg]

Hi Jan,

if you are using the hardware debugger in a classroom, then maybe my latest blogpost could help: https://hinterm-ziel.de/index.php/2022/01/13/a-debugwire-hardware-debugger-for-less-than-10-e/

Best, Bernhard

[jdolinay]

Hi Bernhard, yes, that's great. Thank you!

Cheers, Jan