4.12.0-rc4 Kernel: Reboot does not work

[sghazagh]

Kernel 4.12.0-RC4 is compiled successfully and boots the Tinker Board just fine. Thank you for adding WIFI to, so WIFI is working just fine as well.

However, REBOOT does not restart the board and after turning off the device, cannot get it up again! I have a same issue if I use "echo b > /proc/sysrq-trigger" as well.

[Miouyouyou]

Alright, I'll see if I can find the main reason of this issue and resolve it.

[Miouyouyou]

I catched that in rockchip's kernel miniarm DTS file :

        /*
         * NOTE: vcc_sd isn't hooked up on v1.0 boards where power comes from
         * vcc_io directly.  Those boards won't be able to power cycle SD cards
         * but it shouldn't hurt to toggle this pin there anyway.
         */
        vcc_sd: sdmmc-regulator {
                compatible = "regulator-fixed";
                gpio = <&gpio7 11 GPIO_ACTIVE_LOW>;
                pinctrl-names = "default";
                pinctrl-0 = <&sdmmc_pwr>;
                regulator-name = "vcc_sd";
                regulator-min-microvolt = <3300000>;
                regulator-max-microvolt = <3300000>;
                startup-delay-us = <100000>;
                vin-supply = <&vcc_io>;
        };

This might be related to the reboot bug.

[sghazagh]

I didn't get your point? can you explain please?

At the moment yous rk3288-miniarm.dtsi file has a same code. and, in official image, he reboot works fine and only sysrq does not work!

[Tonymac32]

The supplies are separated on later boards, although you may be correct in a way, power cycling the SD card on shutdown kills it, something that wasn't an issue on 1.0 boards. Also, the system does not bring the SD card back to 3.3V despite returning to the lower speed operating point. https://github.com/TinkerBoard/debian_kernel/commit/c6ad239fbdfc5cf3009b6f1cbf88cfc4df6d3eb9 https://github.com/TinkerBoard/debian_kernel/commit/5883709c4dd0dd495d006e1adee1ef9204ff2c93

[Tonymac32]

So, @sghazagh , the reason it works on the official image and the Armbian 4.4 is due to a straight up hack of the driver code. I attempted to do the same thing on 4.11 and it did nothing.

[sghazagh]

Why this Tinker Board has this much issue then???! If pros like you guys and community have found hard to resolve the issues, AUSU team definitely can't do much to improve it too for sure!

ASUS had to provide a better alternative to RPi3 if they really were looking to hit the market!!! Disappointing!

[Miouyouyou]

Hmm, @wzyy2 and @hizukiayaka might have a clue about how to handle power cycle issues. I'll try to ping them in IRC.

[sghazagh]

That would be good if that can be fixed first as it's very annoying when you're working remotely and you cannot reboot due to this issue!

[wzyy2]

00:22 That issue in tinker board will happend in warm rest. 00:22 and WIFI module are not reset. 00:22 ASUS have fix it in their kernel. 00:23 let me search which commit are used to fix it...... 00:29 https://github.com/TinkerBoard/debian_kernel/commit/33085685b4506e3b1a03c275a1eb9561249659c0 00:30 = =I think it will be this commit. 00:31 BTW, https://github.com/TinkerBoard/debian_kernel/commit/5883709c4dd0dd495d006e1adee1ef9204ff2c93. You might also need this commit when run mainline or rockchip 4.4 kernel in tinker board.

[Miouyouyou]

Alright, most of these patches look small enough to give them a try.

I'll try to add all these patches this evening, see if that resolve the reboot issue on 4.12 kernels.

[Miouyouyou]

New release integrates https://github.com/TinkerBoard/debian_kernel/commit/5883709c4dd0dd495d006e1adee1ef9204ff2c93 patch. The DTS patches were already added previously.

See if that resolves the issue. I don't think this will be enough, as @TonyMac32 already tried that.

[sghazagh]

No, the reboot issue hasn't resolved!!!

[Tonymac32]

I'm somewhat at a loss on this one, is this due to the tinker board hardware, or is it something driver related? I would have assumed the option was there to keep the vmmc active (device tree or otherwise), also shouldn't switching vqmmc back to 3.3 V from 1.8 be standard? I just want to make sure some funny joke isn't being played on everyone involved... ;-)

[wzyy2]

"I'm somewhat at a loss on this one, is this due to the tinker board hardware, or is it something driver related? "

Rockchip bootrom code only allow normal sd mode, so if the vqmmc is 1.8v, it will failed to load firmware from sd-card. It's necessary to set it back to 3.3v during warm reset.

[Tonymac32]

Thanks for the reply, and that I completely understand the mechanism of failure. I'm just surprised to see it show up as a bug that is only addressed via a "hack" or two. The power supply is also apparently switched off before the warm reset. Is there a bootrom fix, or an expected bugfix to the driver?

As for the patch, I think the patch to restart the vmmc is also very important, however it has a delay built into it which appear to break compilation on 4.12. I don't have time tonight to dig into that one.

[Tonymac32]

OK, put the hack inplace, can verify it is being executed, but it is not having the desired effect. Not having appropriate schematics or documentation for this board I'm having a near-impossible time checking physical voltages.

[wzyy2]

Did it still fail to boot? Could you try to disable uhs mode by remove "uhs" prop in sdmmc dts node?

[Miouyouyou]

I'll try to remove it today for the next release, see if that changes anything.

[Tonymac32]

I was hoping to not have to disable one of the more attractive features of the board, Obviously that's the trigger to drop to 1.8V. Perhaps now it needs a delay after telling it to set, the delay position in the code was somewhat of a mystery to me from the beginning, it's the first thing done in the subroutine rather than the last. I would have thought you'd need to restart vmmc, set vqmmc back to 3.3V, then wait for it to stabilize before triggering the SoC to reset. I compiled a test of that last night, but had to get to sleep before it was finished compiling for test. I'll check it later today.

[Tonymac32]

No Dice, removed the uhs props from it to no effect. Also added delay after setting supplies, also no effect.

[wzyy2]

So it might be another reason make it failed to boot. Could you connect otg port with host PC and try reboot?

If "lsusb" in host PC show maskrom device, then it means bootrom failed to load firmware from SD-Card. If there is no maskrom device, then it means kernel have hang in reboot.

[Miouyouyou]

Alright, so, the MMC hack introduced reboot issues with the MiQi so I removed these hacks for now. I've commented 'uhs' references in the sdmmc driver.

Meanwhile, another way to test if the machine reboot or not is to ping the device. If the device still answers a few seconds after closing the SSH session, it means that it could not reboot and is still running...

[Miouyouyou]

Meanwhile if you can:

• Get a serial console output using the USB<->Serial communication system
• Reboot the machine and look at the logs and see if there's anything useful, like crashes, OOPS messages or things like that

If that's the case, paste these messages there.

[Tonymac32]

I'm afraid I almost work exclusively though the serial output on the board, there are no messages, it successfully shuts down, and simply does not restart. This is definitely related to the SoC bootloader (not U-boot). I will plug in a USB cable to see if I can see the maskrom device or not. I will have to check with the team to make sure the Armbian MiQi build is still rebooting properly.

Is it possible supply assignments in the kernel changes between 4.4 and 4.12?

[Miouyouyou]

It's entirely possible. Meanwhile, it might be interesting to open a thread in mainling list consulted by Rockchip enthusiasts. I'll try find the URL on this mailing list again.

[sghazagh]

The "sysrq" boot issue that I had reported on Tinker Board GITHUB issue page received a reply from Tinker Board Support Team and just said: "will be fixed in the next release"!!!!!!!!!!!!!!!!!

https://github.com/TinkerBoard/debian_kernel/issues/2

That's for 4.4.16 anyway!

[Miouyouyou]

Well, I guess we'll see how they fixed in the next release (unless they just import @Tonymac32 fix in their code... )

[Tonymac32]

Lol well, I don't think sysrq boot works on Armbian either, remember I used their code to get reboot going on 4.4. ;-)

Using MicroUSB I am definitely stuck in maskrom mode, I'm staring at a "RockUSB" device.

[wzyy2]

"The "sysrq" boot issue that I had reported on Tinker Board GITHUB issue page received a reply from Tinker Board Support Team and just said: "will be fixed in the next release"!!!!!!!!!!!!!!!!!"

That issue is "sysrq" won't call "dw_mci_rockchip_platfm_shutdown" to switch vccio_sd to 3.3v.

[Miouyouyou]

Edit : I guess I misread that. It seems that the patch forces SysRQ to call the shutdown routine, which it didn't in the first place, alright :smiley:

[sghazagh]

So, is it fixed now? Or can that get fixed here?

[Tonymac32]

I would make the educated guess it will be fixed on the 4.4 kernel. As for 4.12, I haven't figured it out to fix it as yet.

[Miouyouyou]

So, to have a clear view on the situation :

• The only thing needed is to keep the SDCard supplied with 3.3V during the reboot process ?

[Tonymac32]

vmmc and vqmmc both need to be 3.3V on reboot. If you have a UHS card, the driver kicks the I/O down to 1.8V, so it needs reset to 3.3, and the kernel is apparently shutting down the vmmc supply automatically, I tried to set the supply in the device tree to always on but that didn't help (was a long shot). I have to confess I haven't looked at this in a few days, a lot of other things going on, and realistically I needed to walk away from the power/BT issues for a few days.

[Miouyouyou]

Alright, I'll poke some Rockchip guys in the IRC channel meanwhile :smiley_cat:

[Tonymac32]

Yes, in the Tinker Board team's eyes I believe this was closed with the earlier "HACK" labelled patches to 4.4, however those haven't proven effective with kernel 4.12, I haven't dug deep enough to find the actual differences (my newness to Linux kernel development is showing)

[Tonymac32]

...and the Android solution is different, although it appears they have their vmmc and vqmmc defined backwards... So maybe it isn't actually fixed, it just doesn't ever swap the I/O voltage... https://github.com/TinkerBoard/android_kernel/blob/m-mr1-rk3288/arch/arm/boot/dts/rk3288-miniarm.dts#L267

https://github.com/TinkerBoard/android_kernel/commit/a7df6278476767337f1644c8aa4adb214ed7dc68

[sghazagh]

Tony, Is it possible you spent sometime and see if you can fix this?! You just know what might cause the issue and have knowledge of changing the kernel codes. This will be very helpful when you work remotely as you cannot reboot the machine otherwise the access will be lost.

Thanks

[Tonymac32]

Considering I'm the only Armbian dev with this device, it seems unlikely that I wouldn't be looking at it. ;-) The biggest issue comes down to the job I'm paid to do and the family I get paid to support. :-P There are only so many hours in a day. I'm also now dealing with the possibility that any solution will break the MiQi board kernel, and I don't have one of those to verify with.

[sghazagh]

You're 100% right and of course this is only should happened if it does not affect your normal life. So as you wish...

By te way, I thought you only working on this repository here and not the Armbian one. If it was just here, we could check the kernels both for Tinker Board and MiQI devices!

[Tonymac32]

Learning how shutdown in Linux work to figure out if there's another way to tell it not to shut things off.

[Miouyouyou]

I'll try to see if I can apply this patch while adding a special kernel option to activate it. Bear with it until this week-end though.

[Tonymac32]

Well, it doesn't appear to function on 4.12, so that's moot at the moment. I think something changed in the regulator driver itself perhaps.

[Miouyouyou]

Well, this will at least serve as a framework for future similar patches.

Here's the (non-working) reboot fix again, with a special boot parameter protection included ! As long as you don't pass rockchip_tinkerrebootfix=on , the fix won't be applied and everything will work fine on other boards (tested on MiQi devices). When you enable it, you should see the following error in your logs :

( Myy ) Enabling Tinkerboard's reboot fix !

That means the fix is enabled. Also tested on the MiQi (which didn't reboot properly with the option on).

The patch in question : https://github.com/Miouyouyou/MyyQi/blob/e22a6a277082c76cf51c8b55adf0f0dc127b523a/patches/kernel/v4.12/0023-The-tinkerboard-boot-fix-AGAIN.patch

[Tonymac32]

This board is giving me migraine headaches.

[sghazagh]

See it as practice and fun... Spend time whenever you are happy to challenge with Kernel codding otherwise you find it as you said... I am sure when you achieve something that you are working on, that will be so sweet to let you forget the bad feelings ;)

[Miouyouyou]

Electrically, using tools to measure the voltage, does the SDCard gets the juice when rebooting or has the system call absolutely 0 effect ?

I could try to disable this part in dw_mmc-rockchip.c :

static int dw_mci_rockchip_remove(struct platform_device *pdev)
{
        pm_runtime_get_sync(&pdev->dev);
        pm_runtime_disable(&pdev->dev);
        pm_runtime_put_noidle(&pdev->dev);

By keeping the function call but commenting the pm_runtime calls and see how it goes. I don't know what takes priority, shutdown or remove. If once the module removed, the part gets disabled AND the shutdown function has no effect, then if might be the reason.

[wzyy2]

.... try add some log in regulator driver?...

https://github.com/torvalds/linux/blob/master/drivers/regulator/rk808-regulator.c#L366 https://github.com/torvalds/linux/blob/master/drivers/regulator/rk808-regulator.c#L344

and pm_runtime is used to control power-domain/clk, it's about SOC, not regulator.

[Miouyouyou]

Alright, I'll try to add a big log patch for Friday. It might clear up some issues.

[Tonymac32]

I haven't probed the SD lines yet, I can do that tonight. Would this be an issue with any rk3288 system booting solely on SD? So far this is the only thing keeping this device from being viable with the 4.12 kernel.