Board programmer Upload 16u2 DFU (+ USBSerial) doesnt work properly

[NicoHood]

Hi, I modified the Atmega_Board_Programmer.ino to add a custom 16u2 Bootloader. Thats working fine. Now i want to switch back to the official 16u2 DFU. There are two options I tried and both arent working:

1) Uploading the combined hex file from address 0x0000 to 0x4000. The DFU once was present (it changed somehow) but never loaded the USB-Serial. But with manual FLIP sketch execution i could run the sketch. Now (after some attempts) the DFU isnt loading at all, only the sketch.

2) Uplading the official atmel DFU for the at90usb162 (had no 16u2 version handy, should be the same). You can use the DFU, program but after a replug the DFU always loads.

it is NOT a fuse problem. I used the correct fuses from the Arduino Documentation and i also tried with my 16u2 as ISP (see https://github.com/NicoHood/Hoodloader#16u2-as-isp-usage ) and with this one the DFU is working as intended.

You can get the source here: (tell me if the license thing is okay too) https://github.com/NicoHood/HoodLoader2/tree/master/tools/Atmega_Board_Programmer

[nickgammon]

Can you please report your current fuse setting, preferably using my Board Detector sketch?

On my unmodified Uno I get:

Atmega chip detector.
Written by Nick Gammon.
Version 1.10
Compiled on Nov  3 2014 at 07:02:29
Entered programming mode OK.
Signature = 1E 94 89 
Processor = ATmega16U2
Flash memory size = 16384 bytes.
LFuse = EF 
HFuse = D9 
EFuse = F4 
Lock byte = CF 
Clock calibration = 96 
Bootloader in use: No
EEPROM preserved through erase: No
Watchdog timer always on: No
Bootloader is 4096 bytes starting at 3000

Bootloader:

3000: 4B C0 00 00 64 C0 00 00 62 C0 00 00 60 C0 00 00 
3010: 5E C0 00 00 5C C0 00 00 5A C0 00 00 58 C0 00 00 
3020: 56 C0 00 00 54 C0 00 00 52 C0 00 00 EE C4 00 00 
3030: 4E C0 00 00 4C C0 00 00 4A C0 00 00 48 C0 00 00 
3040: 46 C0 00 00 44 C0 00 00 42 C0 00 00 40 C0 00 00 
...
3FD0: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
3FE0: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
3FF0: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 

MD5 sum of bootloader = D8 8C 70 6D FE 1F DC 38 82 1E CE AE 23 B2 E6 E7 
Bootloader MD5 sum not known.

First 256 bytes of program memory:

0: 90 C0 00 00 A9 C0 00 00 A7 C0 00 00 A5 C0 00 00 
10: A3 C0 00 00 A1 C0 00 00 9F C0 00 00 9D C0 00 00 
20: 9B C0 00 00 99 C0 00 00 97 C0 00 00 48 C4 00 00 
30: 0C C4 00 00 91 C0 00 00 8F C0 00 00 8D C0 00 00 
40: 8B C0 00 00 89 C0 00 00 87 C0 00 00 85 C0 00 00 
50: 83 C0 00 00 81 C0 00 00 7F C0 00 00 02 C1 00 00 
60: 7B C0 00 00 79 C0 00 00 77 C0 00 00 75 C0 00 00 
70: 73 C0 00 00 71 C0 00 00 6F C0 00 00 6D C0 00 00 
80: 6B C0 00 00 69 C0 00 00 67 C0 00 00 65 C0 00 00 
90: 63 C0 00 00 61 C0 00 00 12 01 10 01 02 00 00 08 
A0: 41 23 43 00 01 00 01 02 DC 01 09 02 3E 00 02 01 
B0: 00 C0 32 09 04 00 00 01 02 02 01 00 05 24 00 01 
C0: 10 04 24 02 06 05 24 06 00 01 07 05 82 03 08 00 
D0: FF 09 04 01 00 02 0A 00 00 00 07 05 04 02 40 00 
E0: 01 07 05 83 02 40 00 01 04 03 09 04 32 03 41 00 
F0: 72 00 64 00 75 00 69 00 6E 00 6F 00 20 00 28 00 

[nickgammon]

MD5 sum of bootloader = D8 8C 70 6D FE 1F DC 38 82 1E CE AE 23 B2 E6 E7 

This MD5 sum agrees with the file "Arduino-COMBINED-dfu-usbserial-atmega16u2-Uno-Rev3.hex" supplied with the 1.0.6 version of the IDE. Bootloader part only, of course, not the sketch part.

[nickgammon]

According to me the current fuses are:

Atmega fuse calculator.
Written by Nick Gammon.
Entered programming mode OK.
Signature = 0x1E 0x94 0x89 
Processor = ATmega16U2
Flash memory size = 16384
LFuse = 0xEF 
HFuse = 0xD9 
EFuse = 0xF4 
Lock byte = 0xCF 
Clock calibration = 0x96 
Hardare Boot Enable..................... [X]
Debug Wire Enable....................... [ ]
External Reset Disable.................. [ ]
Enable Serial (ICSP) Programming........ [X]
Watchdog Timer Always On................ [ ]
Preserve EEPROM through chip erase...... [ ]
Boot into bootloader.................... [ ]
Divide clock by 8....................... [ ]
Clock output............................ [ ]
Bootloader size: 4096 bytes.
Start-up time: SUT0: [X]  SUT1: [ ] (see datasheet)
Clock source: low-power crystal.
Brownout detection at: 3.0V.

The firmware on my Atmega16U2s is in this file:

http://gammon.com.au/Arduino/UNO16U2_firmware.HEX

If you upload that, and set the fuses as above, I don't see why it should not behave as normal.

[NicoHood]

I used this upload methode with this hex: Arduino_COMBINED_dfu_usbserial_atmega16u2_Uno_Rev3_hex

  { { 0x1E, 0x94, 0x89 }, "ATmega16U2",  16 * kb,   16*kb,
        UNO_dfu_and_usbserial_combined_hex,   // loader image
        0x0000,                   // start address (0x0000-0x4000)
        sizeof UNO_dfu_and_usbserial_combined_hex, 

        128,           // page size in bytes (for committing)
        0xFF,         // fuse low byte:
        0xD9,         // fuse high byte: 
        0xF4,         // fuse extended byte:
        0x2F },       // lock bits:

Atmega chip detector.
Written by Nick Gammon.
Version 1.10
Compiled on Nov  2 2014 at 22:03:13
Entered programming mode OK.
Signature = 1E 94 89 
Processor = ATmega16U2
Flash memory size = 16384 bytes.
LFuse = FF 
HFuse = D9 
EFuse = F4 
Lock byte = EF 
Clock calibration = 4F 
Bootloader in use: No
EEPROM preserved through erase: No
Watchdog timer always on: No
Bootloader is 4096 bytes starting at 3000

Bootloader:

3000: 4B C0 00 00 64 C0 00 00 62 C0 00 00 60 C0 00 00 
3010: 5E C0 00 00 5C C0 00 00 5A C0 00 00 58 C0 00 00 
3020: 56 C0 00 00 54 C0 00 00 52 C0 00 00 EE C4 00 00 
3030: 4E C0 00 00 4C C0 00 00 4A C0 00 00 48 C0 00 00 
3040: 46 C0 00 00 44 C0 00 00 42 C0 00 00 40 C0 00 00 
3050: 3E C0 00 00 3C C0 00 00 3A C0 00 00 38 C0 00 00 
3060: 36 C0 00 00 34 C0 00 00 32 C0 00 00 30 C0 00 00 
3070: 2E C0 00 00 2C C0 00 00 2A C0 00 00 28 C0 00 00 
3080: 26 C0 00 00 24 C0 00 00 22 C0 00 00 20 C0 00 00 
3090: 1E C0 00 00 1C C0 00 00 11 24 1F BE CF EF D2 E0 
30A0: DE BF CD BF 11 E0 A0 E0 B1 E0 EA EF FC E3 02 C0 
30B0: 05 90 0D 92 AA 33 B1 07 D9 F7 11 E0 AA E3 B1 E0 
30C0: 01 C0 1D 92 AB 35 B1 07 E1 F7 72 D3 14 C6 98 CF 
30D0: 98 2F 15 C0 80 91 F2 00 88 23 71 F4 80 91 E8 00 
30E0: 8B 7F 80 93 E8 00 03 C0 8E B3 88 23 51 F0 80 91 
30F0: E8 00 82 FF F9 CF 02 C0 80 91 F1 00 91 50 99 23 
3100: 49 F7 08 95 20 91 4A 01 30 91 49 01 80 91 48 01 
3110: 90 91 47 01 80 93 3F 01 90 93 40 01 C9 01 80 93 
3120: 41 01 90 93 42 01 08 95 DF 93 CF 93 00 D0 00 D0 
...
3FC0: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
3FD0: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
3FE0: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
3FF0: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 

MD5 sum of bootloader = D8 8C 70 6D FE 1F DC 38 82 1E CE AE 23 B2 E6 E7 
Bootloader MD5 sum not known.

First 256 bytes of program memory:

0: 90 C0 00 00 A9 C0 00 00 A7 C0 00 00 A5 C0 00 00 
10: A3 C0 00 00 A1 C0 00 00 9F C0 00 00 9D C0 00 00 
20: 9B C0 00 00 99 C0 00 00 97 C0 00 00 48 C4 00 00 
30: 0C C4 00 00 91 C0 00 00 8F C0 00 00 8D C0 00 00 
40: 8B C0 00 00 89 C0 00 00 87 C0 00 00 85 C0 00 00 
50: 83 C0 00 00 81 C0 00 00 7F C0 00 00 02 C1 00 00 
60: 7B C0 00 00 79 C0 00 00 77 C0 00 00 75 C0 00 00 
70: 73 C0 00 00 71 C0 00 00 6F C0 00 00 6D C0 00 00 
80: 6B C0 00 00 69 C0 00 00 67 C0 00 00 65 C0 00 00 
90: 63 C0 00 00 61 C0 00 00 12 01 10 01 02 00 00 08 
A0: 41 23 43 00 01 00 01 02 DC 01 09 02 3E 00 02 01 
B0: 00 C0 32 09 04 00 00 01 02 02 01 00 05 24 00 01 
C0: 10 04 24 02 06 05 24 06 00 01 07 05 82 03 08 00 
D0: FF 09 04 01 00 02 0A 00 00 00 07 05 04 02 40 00 
E0: 01 07 05 83 02 40 00 01 04 03 09 04 32 03 41 00 
F0: 72 00 64 00 75 00 69 00 6E 00 6F 00 20 00 28 00

Seems that LFuse, lock byte and the clock calibration is different

[NicoHood]

I tied again and changed the fuses to what you've got. The only thing that differs is the Clock calibration and i dont know how to change this.

Atmega fuse calculator.
Written by Nick Gammon.
Entered programming mode OK.
Signature = 0x1E 0x94 0x89 
Processor = ATmega16U2
Flash memory size = 16384
LFuse = 0xEF 
HFuse = 0xD9 
EFuse = 0xF4 
Lock byte = 0xCF 
Clock calibration = 0x4F 
Hardare Boot Enable..................... [X]
Debug Wire Enable....................... [ ]
External Reset Disable.................. [ ]
Enable Serial (ICSP) Programming........ [X]
Watchdog Timer Always On................ [ ]
Preserve EEPROM through chip erase...... [ ]
Boot into bootloader.................... [ ]
Divide clock by 8....................... [ ]
Clock output............................ [ ]
Bootloader size: 4096 bytes.
Start-up time: SUT0: [X]  SUT1: [ ] (see datasheet)
Clock source: low-power crystal.
Brownout detection at: 3.0V.

[NicoHood]

The weird thing is: If i burn this thing with my 16u2 as ISP i get this output, working! Replugging the Arduino and it goes into usb-serial mode. With your sketch it always enters dfu mode!? :S

`
Atmega fuse calculator.
Written by Nick Gammon.
Entered programming mode OK.
Signature = 0x1E 0x94 0x89 
Processor = ATmega16U2
Flash memory size = 16384
LFuse = 0xFF 
HFuse = 0xD9 
EFuse = 0xF4 
Lock byte = 0xEF 
Clock calibration = 0x4F 
Hardare Boot Enable..................... [X]
Debug Wire Enable....................... [ ]
External Reset Disable.................. [ ]
Enable Serial (ICSP) Programming........ [X]
Watchdog Timer Always On................ [ ]
Preserve EEPROM through chip erase...... [ ]
Boot into bootloader.................... [ ]
Divide clock by 8....................... [ ]
Clock output............................ [ ]
Bootloader size: 4096 bytes.
Start-up time: SUT0: [ ]  SUT1: [ ] (see datasheet)
Clock source: low-power crystal.
Brownout detection at: 3.0V.

[NicoHood]

Seems that my 2nd Arduino caused problems because ISP lines were still connected and Ground was not connected. With grounds and holding the 2nd Arduinos reset it works fine now. (will test a bit more, then close)

I am only wondering about 4 things: You added the 3000-4000, what about the 0-4000 for the combined hex file? Why do you have different fuse settings than the official documentation? And what setting makes more sense? How can i change the clock calibration, is that important? Is my published modification okay for you that i upload it this way?

Thx for the quick response!

[nickgammon]

The clock calibration is a factory figure, you won't be able to change that. I think it is for the internal oscillator anyway so it won't apply here.

I'm not quite clear on your workflow here. One thing I should make clear, in case it isn't obvious, is that the bootloader uploader will only upload the bootloader, not the sketch. Judging by the MD5 sum you have the bootloader, which it is not booting into (Boot into bootloader is false).

However since the chip is erased, it will be filled with 0xFF, so it will probably skim through the (empty) sketch and end up in the DFU (bootloader) code.

So without extra steps to upload the code from 0x0000 to 0x0FD0 into the device, it won't act as a USB serial device. (My Hex Uploader sketch would do that, as it reads from the SD card).

Would you like to confirm this is a probable explanation? If so we can work out ways around it. One idea that springs to mind is to modify the bootloader-uploader to upload a second batch of code in this particular case (the main sketch).

Failing that, the bootloader uploader has only installed the DFU code, and then you use Flip to upload the main code. Not that I've ever used Flip, as I use my own tools. :)

Why do you have different fuse settings than the official documentation?

In my original sketch there were no fuse settings for that chip. You put them there. So which settings are you referring to?

How can i change the clock calibration, is that important?

You don't, that is done by the manufacturer. It probably done as part of the production process to get you 8 MHz if running on the internal oscillator, tweaked to allow for manufacturing differences per chip.

Is my published modification okay for you that i upload it this way?

Sure. But you may need to take into account my comments above about the main sketch.

[NicoHood]

I refer to this fuse setting in the readme: https://github.com/arduino/Arduino/tree/master/hardware/arduino/firmwares/atmegaxxu2 So what fuses should i provide with my Hoodloader2? :S

I am not sure what you are talking about in the first part of your comment, i just write what i was thinking: I tested this thing and it works. It writes Bootloader + the Firmware. Its not intended to do that, but its working fine from what i can tell.

You might have missed this question You added the 3000-4000, what about the 0-4000 for the combined hex file? Maybe you wanted to say that you wont add this, since its not intended to write a whole sketch. If you do so, i will just keep it in my documentation though.

Anyways: Good work man, thanks!

[nickgammon]

You might have missed this question You added the 3000-4000, what about the 0-4000 for the combined hex file?

I did miss that, but from what I can see from your changes, it uploads to 0x3000:

        0x3000,                   // start address (0x3000-0x4000)
        sizeof Hoodloader2_Uno_Beta_hex, 

So what fuses should i provide with my Hoodloader2? :S

Your fuses seem to agree with that link, so that should be fine. The only difference I can see is the start-up time for the clock.

I tested this thing and it works. It writes Bootloader + the Firmware. Its not intended to do that, but its working fine from what i can tell.

So, problem solved?

[nickgammon]

I've added a post to my thread about the bootloader programmer sketch:

http://www.gammon.com.au/forum/?id=11635&reply=2#reply2

That explains that the bootloader programmer now can install the default DFU bootloader, and once you have done that you can upload the main code which handles the serial interface.

[NicoHood]

You have a little typo there: 16u2 is the usb chip, not 32u2 (even though i have two uno's with the 32u2 g ) In your Website post and the .h file it is written wrong.

Also this line is wrong, its a 4kb Bootloader: https://github.com/nickgammon/arduino_sketches/blob/master/Atmega_Board_Programmer/Atmega_Board_Programmer.ino#L213

I did miss that, but from what I can see from your changes, it uploads to 0x3000:

Yeah for the Hoodloader2 which is a bootloader only. But not for the DFU + USBSerial which is bootloader + sketch (full 16kb hex file). So you'd need to start the lua script and the bootloader from 0x0000 to upload both at once. The lua script might conflict if you want to add combined firmwares later on with different size. I didnt look too deep into the lua script. But cool that youve added it there.

And i'd also add the possibility to reflash the usb chip with the unos main 328 itself. Thats what i will put in my readme today when i finish this thing.

The difference between your sketch and my 16u2 as ISP might be that the 16u2 as ISP (equal to the Arduino as ISP sketch) releases the ISP lines. Maybe they draw current and then the HWBE isnt shorted to gnd or something like that. Maybe you could check that. https://github.com/NicoHood/Hoodloader/blob/master/avr/firmwares/Hoodloader/Lib/ISP.c#L217-L240

One thing that you could add to your github repository is the link to your websites posts so people can find it better. Maybe in the .ino too, that'd be nice :)

So, problem solved?

I'll report it later after all tests, but is looking good so far.

Edit: This line is also wrong (16u2!) https://github.com/nickgammon/arduino_sketches/commit/3144376f6737d9ae5f19887c279a94fc7e59e8c8#diff-99cb629c6b662247bdae91660ac81f5eR29

Edit2: The fuses for your Arduino: Is it an official R3? The official readme might be outdated. I dont want to change anything important.

[NicoHood]

I found a new bug in the sketch that might be important for verifying the lilypad: https://github.com/nickgammon/arduino_sketches/blob/master/Atmega_Board_Programmer/Atmega_Board_Programmer.ino#L507

replace with:

  for (i = 0; i < len; i++)

[nickgammon]

Also this line is wrong, its a 4kb Bootloader: https://github.com/nickgammon/arduino_sketches/blob/master/Atmega_Board_Programmer/Atmega_Board_Programmer.ino#L213

That line is correct. The figure of 512 is the "base boot size" (not used in that sketch anyway) which is used in other sketches to work out the total bootloader size based on the fuses. It multiplies that figure by 1, 2, 4 or 8 depending on the fuse.

16u2 is the usb chip, not 32u2 (even though i have two uno's with the 32u2 g )

Fixed the typos.

I found a new bug in the sketch that might be important for verifying the lilypad:

Fixed, well spotted!

Maybe in the .ino too, that'd be nice

Done.

The fuses for your Arduino: Is it an official R3? The official readme might be outdated. I dont want to change anything important.

I'm not sure now, I've probably reflashed them all a few times. However the entry from boards.txt agrees with what the sketch has:

uno.bootloader.low_fuses=0xff
uno.bootloader.high_fuses=0xde
uno.bootloader.extended_fuses=0x05

Which official readme are you alluding to?

[nickgammon]

One thing that you could add to your github repository is the link to your websites posts so people can find it better.

Good idea, I've updated the README.md file to better explain what it is all about. Thanks for the suggestion.

[NicoHood]

Which official readme are you alluding to? https://github.com/arduino/Arduino/tree/master/hardware/arduino/firmwares/atmegaxxu2

This doesnt work for me. You need to set the start at 0x0000 (also in the lua file) https://github.com/nickgammon/arduino_sketches/blob/master/Atmega_Board_Programmer/Atmega_Board_Programmer.ino#L218 Then you get the USB-Serial too. I dont get why you leave this out, maybe for space? If you ask me its easier for people who dont know about DFU to just upload the whole file, or optionally provide a #define for this too.

What about releasing the SPI lines (set to input again) after programming? I think that caused the error why my 16u2 as ISP worked first and this sketch didnt.

Also keep in mind that there is a hex file for Arduino Mega. You still can use the uno file, but maybe you want to add this one too with a #define. (look at my new version).

It is also possible to flash the 16u2 itself, but you'd need a capacitor for this and some modifications. see this: https://github.com/NicoHood/HoodLoader2/tree/master/tools/Atmega_Board_Programmer

[nickgammon]

https://github.com/arduino/Arduino/tree/master/hardware/arduino/firmwares/atmegaxxu2

It's just that all three Unos I tested had the fuses how I had them (EF D9 F4) so I don't see a huge problem with sticking to that.

I dont get why you leave this out, maybe for space?

Space is getting tight, indeed. And because the code blocks do not contain addresses I would have to either have one large block (0 to 0x4000) which is 16kB, or modify the code to upload two separate blocks.

What about releasing the SPI lines (set to input again) after programming? I think that caused the error why my 16u2 as ISP worked first and this sketch didnt.

See commit https://github.com/nickgammon/arduino_sketches/commit/88c7ed2e5c40ec3

[NicoHood]

So I think the actual bug is fixed. Nice work and thx for the support!

[NicoHood]

FYI: This is what my original unmodified Arduino Uno says:

Atmega chip detector.
Written by Nick Gammon.
Version 1.11
Compiled on Dec 21 2014 at 11:35:18
Entered programming mode OK.
Signature = 1E 94 89 
Processor = ATmega16U2
Flash memory size = 16384 bytes.
LFuse = EF 
HFuse = D9 
EFuse = F4 
Lock byte = CF 
Clock calibration = 9A 
Bootloader in use: No
EEPROM preserved through erase: No
Watchdog timer always on: No
Bootloader is 4096 bytes starting at 3000

Bootloader:

3000: 4B C0 00 00 64 C0 00 00 62 C0 00 00 60 C0 00 00 
3010: 5E C0 00 00 5C C0 00 00 5A C0 00 00 58 C0 00 00 
3020: 56 C0 00 00 54 C0 00 00 52 C0 00 00 EE C4 00 00 
3030: 4E C0 00 00 4C C0 00 00 4A C0 00 00 48 C0 00 00 
3040: 46 C0 00 00 44 C0 00 00 42 C0 00 00 40 C0 00 00 
3050: 3E C0 00 00 3C C0 00 00 3A C0 00 00 38 C0 00 00 

...

3FD0: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
3FE0: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
3FF0: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 

MD5 sum of bootloader = D8 8C 70 6D FE 1F DC 38 82 1E CE AE 23 B2 E6 E7 
Bootloader name: Arduino-dfu-usbserial-atmega16u2-Uno-Rev3

First 256 bytes of program memory:

0: 90 C0 00 00 A9 C0 00 00 A7 C0 00 00 A5 C0 00 00 
10: A3 C0 00 00 A1 C0 00 00 9F C0 00 00 9D C0 00 00 
20: 9B C0 00 00 99 C0 00 00 97 C0 00 00 48 C4 00 00 
30: 0C C4 00 00 91 C0 00 00 8F C0 00 00 8D C0 00 00 
40: 8B C0 00 00 89 C0 00 00 87 C0 00 00 85 C0 00 00 
50: 83 C0 00 00 81 C0 00 00 7F C0 00 00 02 C1 00 00 
60: 7B C0 00 00 79 C0 00 00 77 C0 00 00 75 C0 00 00 
70: 73 C0 00 00 71 C0 00 00 6F C0 00 00 6D C0 00 00 
80: 6B C0 00 00 69 C0 00 00 67 C0 00 00 65 C0 00 00 
90: 63 C0 00 00 61 C0 00 00 12 01 10 01 02 00 00 08 
A0: 41 23 43 00 01 00 01 02 DC 01 09 02 3E 00 02 01 
B0: 00 C0 32 09 04 00 00 01 02 02 01 00 05 24 00 01 
C0: 10 04 24 02 06 05 24 06 00 01 07 05 82 03 08 00 
D0: FF 09 04 01 00 02 0A 00 00 00 07 05 04 02 40 00 
E0: 01 07 05 83 02 40 00 01 04 03 09 04 32 03 41 00 
F0: 72 00 64 00 75 00 69 00 6E 00 6F 00 20 00 28 00