monome / crow

Crow speaks and listens and remembers bits of text. A scriptable USB-CV-II machine
GNU General Public License v3.0
162 stars 34 forks source link

Crow can fail to boot with Doepfer PSU-3 #489

Open trentgill opened 11 months ago

trentgill commented 11 months ago

When in a case using the Doepfer PSU-3, Crow sometimes fails to boot when powering on the system. Whether or not it works is dependent on how heavily the power supply is loaded (ie how many, and which, other modules).

The workaround right now is to turn on the case for a few seconds, then quickly turn the case off and back on again. The off & on flick should be less than 1 second. This will almost certainly make crow come to life.


Now some background on why I think this is happening, but why I haven't yet been able to fix it:

The PSU3 has undefined & highly variable rail bringup. Depending on the modules loaded into the case, the +12V and -12V rails can come online substantially far apart in time -- Well into the hundreds of milliseconds. In addition, there is a soft-start ramping of voltage that is not consistent. Both of these characteristics are totally different when booting up a cold system, vs a hot-reboot. As such, if the +12V rail normally comes on first in a cold-start, it will typically come on second with a hot-power-cycle.

I have personally experienced this behaviour causing PTC fuses to trip on the Mannequins modules. This happens in modules which are sensitive to one-rail being high while the other hasn't come up yet. Specifically those with SSI2164 / V2164 where they can draw substantial current if the -12V rail isn't yet active when +12V comes online. In these situations the PTC fuses will trip, and the module will be non-responsive until rebooting in a manner that allows the PTC to reset.

Back to crow. I have done some testing, and when these situation occurs, all of the power rails are measuring exactly as they should for a working system. The fuses remain in normal operating mode. When pressing the RESET button, Crow still fails to boot, which suggests the failure is being triggered on a hardware level. There is very things in the digital domain on crow -- The microcontroller, the ADC and the DAC.

My current assumption is that something (uC, ADC or DAC) must be entering a hardware-fault state that is not cleared until power is removed. I believe it is the uC due to what "fails to boot" means. Specifically, Crow never succeeds in initializing the debug-uart port which happens immediately after low-level system init. This suggests there is an issue in system_init() [perhaps we are undervolting the chip?]. Even if the ADC or DAC were failing on boot, they shouldn't stop the uC from booting through the initial stage, so I think we can rule them out.

Undervolting, or some other power-related uC thing seems like the likely culprit as the trigger situation is power related. I can imagine that if we're under powering the chip it could be stable in a well-behaved case, but due to the strange startup on PSU3 we're seeing the expression of an edge case that otherwise is never noticed.

d42kn355 commented 11 months ago

maybe a silly question, but would a soft start unit help with that? so it can boot up on its own. I think they only delay the power around 400-500ms.

trentgill commented 11 months ago

@d42kn355 probably? but i'm trying to find a zero-cost/waste solution because it doesn't seem like there's anything wrong (as in, "not recoverable") with the crow hardware when this occurs. this problem is particularly frustrating because i can't find a viewable cause for the behaviour -- it just doesn't work, seemingly inexplicably.


Just spent a little while playing with the Voltage Scaling mode (ie. voltage setting for driving the uC core), and it did make a difference to current draw of the module, but no difference to the startup behaviour (well the lowest voltage bricked the module & had to be reflashed manually).

eethann commented 8 months ago

I'm having a similar issue when powering on with the USB connected. I had a similar issue when I first hooked up a few devices via TXb, but that was resolved with some cable swaps. I now just need to be sure to turn on my case with the USB unplugged for the module to reliably start up.