pyupdi-style uploads (it works, but discussion of some issues inside)

SpenceKonde commented 3 years ago

In 2.2.2 and 2.2.3, platform.txt is missing a critical change to enable uploads via just a serial adapter (pyupdi-style upload)

# board manager version
#tools.pymcuprog.cmd={runtime.tools.python3.path}/python3

# github version
tools.pymcuprog.cmd={runtime.platform.path}/tools/python3/python3

needs to change to

# board manager version
tools.pymcuprog.cmd={runtime.tools.python3.path}/python3

# github version
# tools.pymcuprog.cmd={runtime.platform.path}/tools/python3/python3

Anyone know how to have a bash script make this change automatically? (adding additional markings to platform.txt for the script to pick up on is fine) x_x

per1234 commented 3 years ago

I spent some time trying to figure out how to use the same tool path property value for both Boards Manager and manual installation and this is what I found: https://github.com/MCUdude/MightyCore/issues/32#issuecomment-238195865 Not very great.

I recommend taking a look at what the ESP8266 and ESP32 platforms did for their esptool and Python tools dependencies. It does look like they change platform.txt when making the release because I see this in the repo: https://github.com/esp8266/Arduino/blob/2.7.4/platform.txt#L141

tools.esptool.cmd={runtime.platform.path}/tools/python3/python3

and this in the Boards Manager installation:

tools.esptool.cmd={runtime.tools.python3.path}/python3

They have a lot of automation, so I'd guess this change is handled automatically during the release creation, but I didn't look to see if that's so.

SpenceKonde commented 3 years ago

Yeah, I think that they do it automatically. Which is my plan. I'm already using Hans's script for generating the board manager releases... but I need to tweak it to modify the platform.txt file to make this adjustment (I currently don't have a way to reach into the middle of the process of generating the release to manually modify the file)

adriantrains commented 3 years ago

Hello @SpenceKonde ! Thanks for your enormous work. I have tested version 2.2.3 and I get the following problem. With version 2.2.1 I have no problems.

SpenceKonde commented 3 years ago

Yes, that is the known issue described here. It is an issue caused by my new board manager package creation workflow, which doesn't leave room for the manual step that was used for 2.2.0 and 2.2.1.

I need to adapt this script https://github.com/SpenceKonde/ReleaseScripts/blob/master/build_megaTinyCore.sh to modify the platform.txt file before it creates the archive to achieve the change described at the top of the issue. I wasn;t able to work on it wednesday because I had an urgent job from a client, as well as very busy day job work.

SpenceKonde commented 3 years ago

Should be sorted with 2.2.4

goodmanj commented 3 years ago

Still having this problem (Cannot run program python3) with MegaTinyCore 2.2.4 on PC and Mac, haven't tried Linux. Version 2.2.3 works on PC, but does not work on Mac. Version 2.2.3 can be made to work on Mac by symlinking to a working python3:

cd ~/Library/Arduino15/packages/megaTinyCore/hardware/megaavr/2.2.3/tools
mkdir python3
cd python3
ln -s `which python3` python3

but this does not work with 2.2.4. Mac board manager installation takes about about a minute, or more, hangs on "Installing tools 1/1".

Error message is slightly different between the two versions: 2.2.4 says "Cannot run program "{runtime.tools.python3.path}/python3", 2.2.3 says "Cannot run program "[user home directory]/Library/Arduino15/packages/megaTinyCore/hardware/megaavr/2.2.3/tools/python3/python3"

SpenceKonde commented 3 years ago

Argh... what the heck...

SpenceKonde commented 3 years ago

Oh... Oh my... well, THAT would do it....

SpenceKonde commented 3 years ago

2.2.5 is now available - wanna go try it again? x_X

SpenceKonde commented 3 years ago

Can anyone confirm that this is fixed now?

microPaul commented 3 years ago

Yes, with 2.2.5 on a W10 machine using the Serial with 4.7K it now works for me. But for uploading 2229 program bytes to an ATtiny412 it takes 26 seconds from the first upload "ping" message until I get the "Verify successful. Data in flash matches data in specified hex-file" message, writing 35 blocks or pages (according to the messages at the bottom of the IDE window). so yes, it works, but it's kinda slow. I wonder how long it would take to upload a 32K program.

SpenceKonde commented 3 years ago

Woah....

That's awful. It's.... ah.... much faster than that for me, but one of the people working to make this a success noted that certain not all serial adapters are equal.... Mine are certainly faster than that, though I haven't done any sort of performance testing - but like... it ain't nearly that bad!

Once I have DxCore 1.3.0 and 2.2.6 a few critical projects out of the way, I'll look a little closer I'd wager that some quality time with an oscilloscope, a UPDI AVR, and a serial adapter, and common parts would be rewarding on that fdront - and even if not, i'm sure it would be a good experience in terms of building skills. Though my hope would be to get people talking and then sneak off while some enthusiastic community members cataloged available adapters, components and programming techniquest...

microPaul commented 3 years ago

My 26 second upload was with a USB to Serial that was 99% likely to be an FTDI clone from 2015 or earlier. Shakey flakey.

but I just now tried it with two different USB to Serials (two different modules of the same design) using my go-to always-works modules based on the Silcon Labs CP2102 and with that it doesn't upload at all. With those I get the following error message:

`Traceback (most recent call last): File "C:\Users\p\AppData\Local\Arduino15\packages\megaTinyCore\hardware\megaavr\2.2.5/tools/prog.py", line 210, in main() File "C:\Users\p\AppData\Local\Arduino15\packages\megaTinyCore\hardware\megaavr\2.2.5/tools/prog.py", line 98, in main return_code = pymcuprog_basic(args, fuses_dict) File "C:\Users\p\AppData\Local\Arduino15\packages\megaTinyCore\hardware\megaavr\2.2.5/tools/prog.py", line 144, in pymcuprog_basic packpath=False) File "C:\Users\p\AppData\Local\Arduino15\packages\megaTinyCore\hardware\megaavr\2.2.5/tools/prog.py", line 107, in run_pymcu_action status = func(backend, *args, args_pymcu) File "C:\Users\p\AppData\Local\Arduino15\packages\megaTinyCore\hardware\megaavr\2.2.5\tools\libs\pymcuprog\pymcuprog_main.py", line 538, in _start_session backend.start_session(sessionconfig) File "C:\Users\p\AppData\Local\Arduino15\packages\megaTinyCore\hardware\megaavr\2.2.5\tools\libs\pymcuprog\backend.py", line 362, in start_session sessionconfig.interface_speed) File "C:\Users\p\AppData\Local\Arduino15\packages\megaTinyCore\hardware\megaavr\2.2.5\tools\libs\pymcuprog\programmer.py", line 83, in setup_device options=self.options) File "C:\Users\p\AppData\Local\Arduino15\packages\megaTinyCore\hardware\megaavr\2.2.5\tools\libs\pymcuprog\nvm.py", line 42, in get_nvm_access_provider accessprovider = NvmAccessProviderSerial(transport, device_info, baud=frequency) File "C:\Users\p\AppData\Local\Arduino15\packages\megaTinyCore\hardware\megaavr\2.2.5\tools\libs\pymcuprog\nvmserialupdi.py", line 53, in init self.avr = UpdiApplication(port, baud, self.dut) File "C:\Users\p\AppData\Local\Arduino15\packages\megaTinyCore\hardware\megaavr\2.2.5\tools\libs\pymcuprog\serialupdi\application.py", line 75, in init datalink.init_datalink() File "C:\Users\p\AppData\Local\Arduino15\packages\megaTinyCore\hardware\megaavr\2.2.5\tools\libs\pymcuprog\serialupdi\link.py", line 45, in init_datalink raise PymcuprogError("UPDI initialisation failed") pymcuprog.pymcuprog_errors.PymcuprogError: UPDI initialisation failed pymcuprog.pymcuprog_errors.PymcuprogError: UPDI initialisation failed

`

I have no idea what this error message means. I'm pretty much just an AVR noob. If it's not something, it's something else.

SpenceKonde commented 3 years ago

It's pyupdi's STK500 sync error "it's a serial port, but I tried to use it to program, and I didn't hear anything back that looked like it could be something I can program" It's the error you get if UPDI is fused as reset, or the pin has a big capacitor on it, or if you picked the serial port on the jtag2updi nano, or the board isn't plugged in, or if the board isn't plugged in because that noise you heard was it crunching under the wheel of your office chair.

I thought I made one of those CP2102's work.... but I think I had to dial back the strength of the resistor significantly. Take a volt meter to it - they're 5v tolerant, but they don't actually output 5v! (which is probably why you're having problems)

I know I had at least 5 of them, but I'm not sure where they went. I think there has been at least one intermediate purge of serial adapters since I purged those damned CP2102s from the accessible serial adapter drawer...

CH340G for life yo. I was so excited about the Holtek HT42-whatever.... then I got them in, and they behaved oddly in practice... they hung my serial ap a couple of times, for no reason! (okay okay, maybe there was a reason - but I stand firm that it wasn't a good one) - but the real problem with them is that the modem control inputs are backwards (they all read not-asserted - that means "high" on a normal adapter" (and they are high). But the first time one goes low, all of them suddenly switch to inverted logic until the device is reset. Terribad! I was so psyched about them too

they have a REAL VDDIO pin! I have serial adapters designed with a 3-position voltage switch, 5V, 3.3V, and "VTarget". If I ever have the time, I'm going to write a python script based on this one that uses the Holtek programming API (for changing the name it shows up as and stuff. but I'd have it wired up so like, it's Tx went to one modem control input, DTR and RTS each on another pin, with the final one wired up so I could pull it low with some combination of those three output pins.... Anyway, the address space has a ton of unused bits in it.... with it wired up so I could see if I did it successfully I'd just start fuzzing it, writing random stuff to bytes that are.... (would be good practice automating electronics stuff, which could be leveracged to automated testing of the cores too...

(as an aside... a CH340N would be great as a dedicated pyupdi board wouldn't it? 3 pins on one end, micro USB at the other, CH340N in the middle.... the board could be utterly tiny.. CH340E would work jyst as good, maybe even a little smaller overall - thougjh those a re a bit more fit-for-purpose compared to the N.... maybe 0.4~0,5" x 0.8~1"?

microPaul commented 3 years ago

I think i have some CH340 modules. I'll give one of those a try. maybe later tonight, but most likely tomorrow.

I just checked one of my CP2102 modules with a scope. on the +5V pin I get 5.2 volts. On the 3V3 pin I get 4.2 volts (not sure what's going on with that). I then used RealTerm to send an ASCII text file at both 300 bps and 115200 bps. In both cases the TXD moved cleanly between 0.2V to 4.2V (4.2 being not all the way to 5.0, but 4.2V should be a valid 5 Volt TTL high). Is 4.2V high enough for a UPDI high when the MCU is powered by 5.0V?

SpenceKonde commented 3 years ago

I dunno, I remember reading the chips spec sheet and it not saying their voltage was lower than 5v, I only recently got the programming up and running myself, and I've bee bust working on finishing DxCore 1.3.0 and the megaTinyCore, releases, not uploading much, mostly just writing code and (today) shuffling files around. I'm going to get back to that so I can actualyl get it posted tonight.

microPaul commented 3 years ago

i don't know either, other than USB-serial in general often seems to be a source of frustration for many users, including me. I wonder if part of my problem is that I may be unwittingly working with counterfeit parts. Although the CP2102 and CH340 are so cheap from legit distributors that it wouldn't seem worthwhile to counterfeit them. I can understand why the FTDI would be a target of counterfeiters.

The pyupdi 4.7K programmer is a great long term goal, but the jtag2updi programmer seems to work ok, so there's no loss if it takes a while before you have time to look further into the pyupdi programming issues. Improvements to the Cores are probably more important. Thanks for doing all this work. I'm grateful, as I'm sure others are as well.

microPaul commented 3 years ago

I've had good luck with using the $2.50 ebay CP2102 USB to serial modules for Arduino Pro-Mini and I also sometimes retrofit them into Arduino Unos. I do that because several of my projects last year involved having a PC dump a lot of data over the serial link to the Arduino and the W10 driver for the CP2102 respects XonXoff flow control (the W10 driver for the CH340 does not). Without XonXoff flow control the Arudio would run out of buffer space because the output device it was controlling was slow.

I took a look at the CP2102 data sheet as you suggested. I can't see how/why the modules I have show 4.2VDC on the 3.3V net. In the data sheet Table 6 Voltage Regulator Electrical Specifications, indicates that the internal regulator has a fixed output that can range from 3.0 to 3.6 VDC, and there are no provisions for "adjustments" for that regulator. Also, there doesn't appear to be any mis-wiring on the module when compared with the data sheet, so I can't figure out why the 3.3V net has 4.2VDC sitting on it.

From the data sheet, Table 2. Absolute Maximum Ratings, "Voltage on any I/O Pin, VBUS, or RST with respect to GND" is 5.8VDC, so it would appear to be OK to have the TXD output forced to 5V by some external pullup, but what I don't get is why the output of my CP2102 module rests at 4.2 VDC. There are three LEDs mounted on the board with associated current limiting resistors. There's an LED on TXD that connects through a 220 ohm resistor to the 3.3V net, and another LED on RXD that connects through a 220 ohm resistor to the 3.3V net. And a third LED that connects from SUSPEND* through a 1K resistor to ground. When I remove all three LEDs from the board the TXD output doesn’t jump down to 3.3V, instead it stays at 4.2 VDC, and the 3.3V net remains at 4.2V. I have no explanation for any of this.

microPaul commented 3 years ago

I did a little bit of scope work with my (likely knockoff) FTDI USB to serial that does work (but slowly) with the pyupdi 4.7K programmer method. It appears that the serial data speed of the pyupdi 4.7K programmer method is 115,200 bps with that USB to serial converter. The PC to target messages appear to be one to four bytes long and the response from the target to PC appears to be one byte long and that that response byte is sent about 1.2 ms after the PC to target message is sent. The PC to target messages appear to occur every 15.7ms. With PC to target messages occurring only every 15.7 ms, then if two data bytes were transmitted with each message, it would take 17.5 seconds to program 2229 bytes into the device. If only one byte of data was transmitted with each message, it would take 35 seconds to program 2229 bytes into the device. I was seeing about 26 seconds of total elapsed time to actually program 2229 bytes into an ATtiny412, so the numbers sort of hang together. I think the primary question is "why does the programmer wait 15 ms between each PC to target message"? If that time could be reduced to something like 3 ms, then the programming would run about 5x faster, and the 26 seconds go down to about 5 seconds.

SpenceKonde commented 3 years ago

@qbolsee - Wasn't it the FT232 that you said had awful performance with pyupdi-style serial programming? Or am I misremembering...

microPaul commented 3 years ago

The FTDI type interface (likely knockoff FTDI) I tried works with pyupdi, but slowly. When I tried the CP2102 and CH340 type devices I got errors, the list of errors ending with "pymcuprog.pymcuprog_errors.PymcuprogError: UPDI initialisation failed"

SpenceKonde commented 3 years ago

CH340 serial adapters can definitely work, and they're fast enough that I found them entirely usable.. I mean, if they didn't, I couldn't have tested this feature, because that's pretty much all I have...

What model do you have? can you show pic or link?

Putting that on the scope would probably be more illuminating - my guess is that your serial adapter has extra resistance in series with the TX line, and the combination of the two is keeping it from being able to drive the pin low enough to be seen as a LOW.

microPaul commented 3 years ago

Ah, I didn't think to check for series resistors on the TX lead. Yes, I can see how that would mess things up. The CH340s that I have are a bit goofy in that they have the fat-flat USB connector that requires a USB extension cable to connect to the PC. They do physically look suspect from the get go. Anyway, I'll send you some photos of them, check for series R and retest them with the scope, but I probably won't be able to get to it until later tonight.

SpenceKonde commented 3 years ago

Yeah, those awful connectors are almost ubiquitous on CH340G adapters.... One of the reasons I designed my own!

microPaul commented 3 years ago

I did some snooping on my CH340G module and did find a 1500 ohm resistor in series with the TX lead. Why the developer included that, I have no idea. Anyway, I removed that resistor and replaced it with a short. Still get the error messages. But when I disconnect the CH340G from the ATtiny412 target and have it transmit at 115200 using RealTerm the TX signal has nice clean transitions from 0 to +5 and back again. Same thing when the pyupdi program is sending data to the target, although not much data gets sent. it looks like a byte or two then a wait, then another byte or two, then a 30+ms break, then another byte, and the error messages are pouring out.

here's a photo of the module I'm using, for what it's worth.

qbolsee commented 3 years ago

@qbolsee - Wasn't it the FT232 that you said had awful performance with pyupdi-style serial programming? Or am I misremembering...

Yes absolutely, and it's worth investigating. Now about the "UPDI initialisation failed" it could indeed be anything, even wrong settings on the port (parity even, two stop bits). I'm also curious whether the manual break to wake up the chip works with all those serial adapters or not. Normally yes, as it simply involves a change to a lower baud (300 I think) to simulate a break.

If I can get my hands on those faulty serial adapters I can try to debug the error and/or the speed issues. I'm guessing the behavior is identical with pyupdi and pymcuprog (here).

SpenceKonde commented 3 years ago

Why do I still see THREE 1.5k resistors... then? Oh for the three leds? Wait THREE leds?! CH340G doesn't have a separate line for the RX LED... ONE OF THOSE LEDS IS ON RX! Kill the RX led. RX leds (except on parts that bring out a separate control line for the RX led) are kyptonite here.

as for the TX-series-resistor: They put the resistor there for protection - so you can connect it to a 3.3v microcontroller and that resistance plus the protection diode will generally keep the target from being damaged...

How I remove SMD LEDs: rest soldering iron tip on them and munge off the plastic.. then angle the iron low and parallel long part of the LED with a blob of solder on it to make contact with both sides - takes em off easy.

microPaul commented 3 years ago

Bingo! I should have known that the LEDs would cause problems.

I removed the LEDs from this CH340G module and it now works great. The time gap between USB-to-target messages is around 1.5ms and the 2229 byte program uploads to the ATtiny412 in about 6 seconds. I also removed all the LEDs from the CP2102 module (the one with the onboard regulator the produces 4.2 volts on the 3.3v net). That too now works.

So definitely best to remove LEDs from these USB to serial modules unless the user KNOWS FOR SURE that they are implemented in a way to not interfere with the serial communications.

Thanks for your advice Spence.

SpenceKonde commented 3 years ago

AAHA! Thanks for the confirmation. I'll make sure that is covered in the docs, heh...

An LED on TX is fine (they get connected before the resistor). I recommend one, so you can see what your board and programmer are doing....

But on RX you end up with - in effect - a resistor ladder between (Vcc - Vf(led)) (Vf is 2v or so for red, 3v or so for blue, maybe a little less at such low current... and 4.7 vs a 1.5 means the TX pin is only going to be able to get the line down to 3/4ths of (Vcc-Vfled), so, in the range of 1.4-1.8v.. but it needs to go below Vcc * 0.3 (and do so with something close to nominal timing). The only way to get an RX resistor is to either have an adapter based on a chip that directly drives that LED (only one I know if that does that is the FTDI part; the Holtek HT42B534 has such a pin (it goes on for both TX and RX), though the response time on it is pretty slow - when I was playing around with that project, I I was going to use a bicolor LED, with resistors and LED colors chosen so that you'd get red and green (looks yellow) for TX, and just red for RX - as I think I mentioned, I gave up on that when I couldn't get the modem control inputs to work like a normal part on them..... I might revisit that though with a different objective...

I love the modem control lines, but nobody else seems to give a damn about them! (I did a big long thing where those lines were worth their weight in gold - I built a few boards with a bunch of comparators wired up, with two extra serial adapters on each line - each one had it's RX being driven by a '339 or similar to mirror a one of the two lines on the interface of interest (which was a serial connection between the thing I was writing f/w for, and some other serial device with a serial interface to a mesh network) - but in addition to that, I had their TX lines connected to separate comparator channels that could drive their lines low... that is, I had "spy" devices on both directions of the serial interface which could not only listen in, but inject messages to the serial data stream pretending to be either of the two sides.... anyway, in addition to that, a key thing in my toolbox was those modem control inputs... each one was wired to something on the system I was working with (which was awkwardly located and shaped, so they were way better than a physical LED I'd jhave to peer over at)

microPaul commented 3 years ago

I should have figured out the LED problem sooner, as I had suffered through a similar issue with Arduino Clones that use the CH340G. There's a 1 K series resistor from the CH340G TXD to the 328P RXD, and the designer of the clone board decided to put the RX led on the 328P side of the 1K resistor, so that the LED current passed through the series 1K on its way to ground, thus producing a 2V voltage drop across that series resistor. That caused the 328P RX input to only go from +2V to +5V. The 328P would still "see" the correct signal (although I don't know how based on the data sheet specs), but I only discovered problem when my cheap logic analyzer gave me no indication of what was going on with the 328P RX input. It was only when I put a scope on it that I saw the issue. So now I remove the RX LED on all the Arduino clones that I use.

SpenceKonde commented 3 years ago

These observations have been noted both here and on DxCore where the same functionality is available. Gladthat sorted it out

SpenceKonde / megaTinyCore

pyupdi-style uploads (it works, but discussion of some issues inside) #293