Flash Pro micro with mEDBG firmware using its USB connector and Arduino IDE

[MCUdude]

I'm sure this can be done very elegantly by providing a bare minimum Arduino core with a boards manager URL. Would you like to contribute to this @per1234

[per1234]

The default auto reset function (1200 baud hack) takes quite a bit of time. How do we deal with this when this can last longer than the bootloader timeout? A separate menu option in the tools menu for first time program and firmware upgrade?

I'm not sure I understand. The Arduino IDE's ATmega32U4 "1200bps_touch" feature waits some time for a new port to appear, but as soon as the new port appears (which happens as soon as you manually reset the board), it immediately runs the avrdude command.

---

I'm having an avrdude verification failure of the second write operation. Interestingly, the verification of the flash write fails if that's the second operation in the command, the verification of the .eep fails if that's the second operation in the command. The verification of the first operation always succeeds. From a quick test of only a few EEPROM values, it looks like the correct values are getting written. Do you have any ideas about that? Are you having any problems like that?

Another thing of interest: I noticed that if I try to run your avrdude command: -Ueeprom:w:mEDBG_UPDI_1.13.eep, it fails with:

invalid file format 'mEDBG_UPDI_1.13.eep' in update specifier

I am specifying the Intel hex file format in my avrdude command: -Ueeprom:w:mEDBG_UPDI_1.13.eep:i. Is that correct?

[per1234]

it fails with:

invalid file format 'mEDBG_UPDI_1.13.eep' in update specifier

I think this is a Windows-specific problem. I didn't provide my actual command:

tools.avrdude.upload.pattern="{cmd.path}" "-C{config.path}" {upload.verbose} {upload.verify} -p{build.mcu} -c{upload.protocol} "-P{serial.port}" -b{upload.speed} -D -Ueeprom:w:"{runtime.platform.path}/firmware/mEDBG_firmware/mEDBG_UPDI_1.13.eep" -Uflash:w:"{runtime.platform.path}/firmware/mEDBG_firmware/mEDBG_UPDI_1.13.hex":i

This specifies the full path to the .eep file, which contains "c:", so that provides the first colon avrdude expects in the command. I was hoping that quoting the path would fix that, but no luck. Well, It's no problem now that I understand the cause. I just need to specify the format, even if that is a for the default input file format.

[MCUdude]

The Arduino IDE's ATmega32U4 "1200bps_touch" feature waits some time for a new port to appear, but as soon as the new port appears (which happens as soon as you manually reset the board), it immediately runs the avrdude command.

You're right! I just assumed is were like this. By manually toggling reset the upload starts immediately.

EEPROM upload works fine here, given that EEPROM is flashed before the flash memory. A verification mismatch on address 0x7000 is to be expected because the mEDBG hex file also contains a similarly sized bootloader that can't be written.

avrdude: verifying ...
avrdude: verification error, first mismatch at byte 0x7000
         0x55 != 0x0c
avrdude: verification error; content mismatch

[per1234]

OK, I'm getting the verification error of the .hex file at 0x7000 and no verification error of the .eep file as long as I write the .eep first, then the .hex, so I'll not worry about that. This will currently cause the Arduino IDE to report the upload as a failure if people have verification enabled in preferences (which is the default). Should I configure the upload command to never verify, or is it possible to modify the firmware .hex file so that it doesn't try to write to the boot section?

[MCUdude]

It's a nice thing to know that the verification error occurred at 0x7000. That tells us that the rest of the firmware is fine.

It is possible to modify the mEDBG hex file and remove the bootloader part in order for the verification to succeed, but then I'll have to manually do this every time there's a new mEDBG firmware available.

I also want to keep the existing loader scripts with the default firmware. By using these the microUPDI programmer will software vise be a 100% "real" mEDBG with the original bootloader, and the firmware can be updated through Atmel Studio.

[per1234]

OK, I understand. To avoid the spurious upload failure, I should be able to make a wrapper script/batch file for avrdude that will suppress the exit status 1 only when it detects the line in the output:

avrdude: verification error, first mismatch at byte 0x7000

I'm sure I can do this fairly easily in a shell script, but doing anything complex in a batch file is usually more difficult than it has any reason to be.

[MCUdude]

If you're making a batch script it's also important to make sure it's supported on Mac an Linux as well.

[MCUdude]

How about the script outputs something like "DFU bootloader not written" or something similar when the 0x7000 error is detected? It is nice to know what worked and what didn't in an upload sequence.

[per1234]

If you're making a batch script it's also important to make sure it's supported on Mac an Linux as well.

Of course. The Arduino hardware system allows you to specify a different property for each OS. So I can do things like this:

# Used on Linux and macOS
tools.avrdude.cmd.path={path}/scripts/avrdudewrapper.sh
# Used on Windows
tools.avrdude.cmd.path.windows={path}/scripts/avrdudewrapper.bat

How about the script outputs something like "DFU bootloader not written" or something similar when the 0x7000 error is detected? It is nice to know what worked and what didn't in an upload sequence.

Sure. I'm going to provide a proof of concept that doesn't have the avrdude wrapper for you to test out first. That way, if you find a problem when you test that ends up making the Arduino IDE project impossible, I can avoid wasting a bunch of time figuring out how to write the wrappers.

[MCUdude]

Perfect, looking forward to test it!

[MCUdude]

I had a quick look at your microUPDIcore, and You'll probably have to build and upload for Arduino Leonardo, rather than Micro. I tried the Micro bootloader on my Pro Micr clone, and the RX and TX LEDs were constantly on. But please try to burn the Leonardo and the Micro bootloader to your hardware to see what works best in your case!

[per1234]

Here's the proof of concept: https://github.com/per1234/microUPDICore Manual installation only right now, but I'd be happy to add Boards Manager installation if it ends up being something useful.

To use it, select Tools > Board > microUPDI Firmware Uploader and then do an Upload. It makes no difference which sketch you have open. I already had a half-finished system I developed for another project of mine that stubs out all the compilation commands. I got that fully working and added it to my platform.txt. In addition to making the upload a little faster, I think it will eliminate confusion that might be caused by people seeing the sketch they happen to have open in the Arduino IDE being compiled, even though that sketch is not going to be uploaded.

[MCUdude]

Cool, works perfectly! How about the script that mutes the 0x7000 verification error? Would love to try that out as well

[per1234]

I tried the Micro bootloader on my Pro Micr clone, and the RX and TX LEDs were constantly on. But please try to burn the Leonardo and the Micro bootloader to your hardware to see what works best in your case!

I didn't specify a bootloader in my boards.txt simply because it's one more file to add to the core (you can't reference bootloaders from Arduino AVR Boards). I didn't think it very likely that anyone would try doing a Burn Bootloader with the "microUPDI Firmware Uploader" board selected. If you think that's something people would want then I can certainly add the bootloader file and update boards.txt accordingly.

Cool, works perfectly!

Great! I get a kick out of finding ways to make the Arduino IDE do things it was never intended to do.

How about the script that mutes the 0x7000 verification error? Would love to try that out as well

I'll get working on that now and let you know as soon as I have it ready.

[MCUdude]

I didn't specify a bootloader in my boards.txt simply because it's one more file to add to the core (you can't reference bootloaders from Arduino AVR Boards). I didn't think it very likely that anyone would try doing a Burn Bootloader with the "microUPDI Firmware Uploader" board selected. If you think that's something people would want then I can certainly add the bootloader file and update boards.txt accordingly.

That's completely fine. There's no need to burn the bootloader when the microUPDI Firmware Uploader board is selected. Fewer files to keep track of too. If users are having bootloader issues it can simply be re-flashed by selecting Arduino Leonardo or Micro as their board.

How about the script that mutes the 0x7000 verification error? Would love to try that out as well

I'll get working on that now and let you know as soon as I have it ready.

Perfect! I feel like the mega0, tiny0 and tiny1 series are gaining attraction now that there is a small community around them + some basic tools for interfacing with UPDI. I'm hoping that we can see real Optiboot support for them one day. That would make the mEDBG chip on the UNO Wifi Rev2 (and SAMD11 on Nano every) excess and could be replaced by a simple USB to serial chip. From a development board point of view, it would greatly reduce competitivity and reduce cost!

(TBH I'm surprised that a big player like Microchip haven't put a developer or two on the task in order to acheive this. I'm sure they'll get their money back in the form of interest in the new AVR family if they do!)

[per1234]

I have now finished the avrdude wrapper scripts and pushed them to my repository. Let me know how it works for you.

[MCUdude]

Works great! The outputted text is still red, but at least it says Done uploading. But could up add a few newlines to the NOTE you've added? Now the note is just a looong line.

NOTE: The verification error shown in the output above is normal and expected. It only indicates that the DFU bootloader included in the mEDBG firmware could not be written. The standard Pro Micro bootloader was retained.

NOTE: The verification error shown in the output above is normal and expected. 
It only indicates that the DFU bootloader included in the mEDBG firmware could not be written. 
The standard Pro Micro bootloader was retained.

[per1234]

The outputted text is still red

Upload output is always red in the Arduino IDE, error or no error, no matter which core you're using. In the past, the excuse I heard for this is that the Arduino IDE colors all stdout white and all stderr red. It is true that for some reason avrdude prints all its output to stderr. However, the echo statements I used to print my note should not be printed to stderr so it seems there is something janky with the Arduino IDE. It certainly has been the cause of a good deal of confusion for Arduino users used to red == bad.

But could up add a few newlines to the NOTE you've added?

Good idea! It's done now.

So if you think this is now a good solution, how do you want to proceed? I could submit a pull request to add the files to this repository, but I don't really see any reason for the hardware and firmware stuff to share the same repository. It would make some sense to do that if a change to the software could result in a change to the hardware, or vice versa, but I don't see any chance of that happening. So my suggestion would be for you to create a dedicated repository for the dummy core.

After that, I can add the Boards Manager installation support following our usual system (gh-pages branch).

[MCUdude]

I could submit a pull request to add the files to this repository, but I don't really see any reason for the hardware and firmware stuff to share the same repository.

It would be nice to have everything in one place. This isn't a project where continuous maintenance and support is needed so keeping everything in one place would be a bonus IMO.

Is it possible to achieve this while still having all the Arduino and firmware related files inside the firmware folder?

Also, a boards manager URL, when everything is ready to be released, would be awesome!

[per1234]

I can structure the Boards Manager archive file any way I like on each release so there's no limitation there. The only problem with putting the dummy core under the firmware folder is it makes manual installation a little more difficult. With your other cores, the user only needs to unzip or clone the repository into the hardware folder of their sketchbook to do a manual installation. With the dummy core under the firmware folder, they will need to move the core folder out of the firmware folder into the hardware folder.

Another option would be to put the dummy core in the root of the repository (like your other cores) and then put the hardware files under a subfolder. Something like

avr
|_extras
   |_hardware

Anyway, if you want me to submit a pull request to add the files to the repository, just let me know where you want it. You're also welcome to just copy the stuff over if that's more convenient for you.

---

One thing I noticed that is quite annoying is that Arduino claimed the mEDBG VID/PID as "Arduino Uno WiFi Rev2" in the megaAVR Boards boards.txt. This means that the port of any board running the mEDBG firmware is identified as "Arduino Uno WiFi Rev2" in the Arduino IDE's Tools > Port menu. This actually caused me a moment of confusion: "what, I didn't know I had my Uno WiFi Rev2 plugged in", and I'm sure it could cause even more confusion for others who don't understand the workings of the Arduino hardware system.

I had the idea to add a hidden board entry to the dummy core's boards.txt to override Arduino's label of the port, causing it to instead be labeled something more accurate like "mEDBG Virtual COM Port". It turns out the override precedence is pretty convoluted and I've already determined it's not possible to do this in a manual installation, but I think it may be possible to do it in a Boards Manager installation.

Is this something you would be interested in? If so, I can do an investigation and implement it when the time comes for the release.

[MCUdude]

it's totally OK to add the avr to the root. Will the Hardware folder cause any problems when located on the root as well? We can also move the flash scripts inside the avr folder. I still want them there because that's the only way of flashing the DFU bootloader the mEDBG comes with. It doesn't matter if they're tucked away in there. A little note in the readme will do.

About the Uno Wifi Rev2 identification. It would be great it is was possible to make it appear as microUPDI programmer instead. As far as I know, all communication with the mEDBG chip happens through a virtual serial port interface. Another pretty annoying thing too is that it gets a different serial port number (both on mac and windows) when in bootloader mode vs mEDBG mode. If you want to re-flash it you'll have to reset it, hurry up and select the serial port that presents itself an Arduino Leonardo, hit upload and possibly reset it again because the bootloader timed out. It's a quirk, but it's still doable as long as it's properly documented.

[per1234]

Will the Hardware folder cause any problems when located on the root as well?

Yes. It causes the Arduino IDE to regularly display this warning:

Could not find boards.txt in E:\arduino\hardware\microUPDI\hardware. Is it pre-1.5?

To avoid that, it needs to be somewhere under the avr folder.

It would be great it is was possible to make it appear as microUPDI programmer instead.

The problem with that is it will cause every mEDBG device to be labeled that way in the Arduino IDE, including the Uno WiFi Rev2. I think the only fair thing to do is to give it a generic label, or maybe give it no label at all (that would probably look something like Tools > Port > COM42 (-)).

If you want to re-flash it you'll have to reset it, hurry up and select the serial port that presents itself an Arduino Leonardo, hit upload and possibly reset it again because the bootloader timed out.

Using the Arduino IDE (or arduino-cli), that shouldn't be necessary. The IDE waits for a new port to appear and automatically starts the upload to it. So the only tricky part is to do a reset within the given time window. That definitely should be documented though.

[MCUdude]

To avoid that, it needs to be somewhere under the avr folder.

Dang. It feels kinda odd that the hardware files are located inside the avr folder. This is mostly a hardware project after all.

Here's what I think is best to do. I'll create a new repo where I host the mEDBG firmware. You'll find it at https://github.com/MCUdude/microUPDIcore/

We host the firmware files and the general Arduino support files, and link in the boards manager URL as well as the repo URL in the microUPDI readme.

When it comes to the Serial port name, how about calling it *Uno Wifi Rev2 or microUPDI?**

I'll create the firmware repo right away. Would you like to do a PR where you add the necessary files?

[per1234]

Sounds good.

[MCUdude]

I'll receive some microUPDI PCBs at the beginning of next week. I'll do some testing with it before we I do a microUPDIcore release (for boards manager).

[MCUdude]

I've received some microUPDI boards, and they work great! Maybe we can do a microUPDIcore release so that boards manager install is possible?

[per1234]

Sounds good. I'm ready to do the Boards Manager thing whenever you are.

Are you going to have microUPDI boards for sale? I definitely want to buy one when available.

[MCUdude]

Sounds good. I'm ready to do the Boards Manager thing whenever you are.

Great! Is there anything I'll have to do before you can submit a PR? make a gh-pages branch maybe?

Are you going to have microUPDI boards for sale? I definitely want to buy one when available.

I am. I only have four boards left, but I'll see if I can add them to Tindie soon. To keep shipping cost down I'm planning to only solder the SMD parts, and leave the UPDI and target voltage header for the user to solder. What do you think is a fair price for a board with all the SMD components on, 1x3 pin target voltage header, jumper, and a 2x3 pin UPDI header?

[per1234]

make a gh-pages branch maybe?

That's correct. I figure we can do it just like the other cores since that seems to work well.

To keep shipping cost down I'm planning to only solder the SMD parts, and leave the UPDI and target voltage header for the user to solder.

Sounds good!

What do you think is a fair price

I have no clue because I don't know all the parts, labor, and overhead costs.

For my glass work, I have a nice spreadsheet set up where I can input the materials cost, production time, and failure rate and it automatically calculates what I need to charge to cover materials and overhead cost as well as getting paid for my time. I just let the spreadsheet make the pricing decision rather than trying to figure out what people are willing to pay for the product. I put it on the market at that price and if it sells then it's a viable product.

It gets tricky trying to figure out how to distribute NRE costs though. If you had to charge enough on the first board to cover all the development and design time you spent, that would be a pricey board indeed! I have product development time and expense items in my overhead calculation that are really just a rough estimate of an average of how much I spend developing products. It's certainly not a perfect system but I think it averages out to be fairly accurate across many products.

The problem is that setting that whole system up in the first place and then gathering the data for each new product is a lot of work and somewhat boring. For this reason, a lot of my fellow glassblowers just sort of pull a price out of the air, but I think this is a dangerous thing to do. That practice is especially hazardous when it comes to production work, where if you set the price unnecessarily high the resellers will have trouble moving the product and not come back for a re-order, but if you set the price too low you end up making no profit on a large sale and having friction when you later try to raise the price on the next order from the buyer.

[MCUdude]

gh-pages branch created!

I have no clue because I don't know all the parts, labor, and overhead costs.

True. I spent some time trying to get the mEDBG firmware to work on custom hardware before I figured out I had to tie the AREF pin to VCC in order for it to actually work. I also spent some time designing the hardware, but this wasn't particularly difficult. For a start, I'm thinking somewhere between $10 and $15 + shipping (~$4). That will cover the cost, labor, and handling. Compared to a small board from China it's quite expensive, but I can never compete with their prices. I can, however, compete by providing good documentation and an easy-to-use solution.

[MCUdude]

@per1234 anything else you need to create a boards manager URL? I've also released v1.0.0.

[per1234]

When it comes to the Serial port name, how about calling it Uno Wifi Rev2 or microUPDI?

I thought I should document the findings of my investigation about overriding the board name associated with the mEDBG VID/PID by Arduino megaAVR Boards. Although I discovered that the VID/PID definition in the hardware core installed via Boards Manager which has a vendor name that comes first in lexicographical order will override any other definitions for the same VID/PID that come later, the packages of the arduino vendor always get priority, even if the 3rd party vendor name comes first (e.g., aaamicroUPDIcore). So I could not find any way to give the Tools > Port menu label for the microUPDI programmer a less confusing name than "Arduino Uno WiFi Rev2".

[MCUdude]

Good point! I'll add a note to the Usage section in the README.

I'll hope you'll find the microUPDI board useful! I sold all four boards in less than 24 hours, so there is some demand I guess. I'll try to make more board as quickly as I can. If this pinout is going to be the new "UPDI" standard It's important that as many people as possible can get one if they need one.