cpuidle: On Armada XP, SATA transfer hangs with cpuidle

[gclement]

basically, any serious PCIe activity seems to be hanging the system. Most of the time it's a hard lockup (no sysrq, the HW debugger says the cores are around 0xffff0000), a few times sysrq was still working, but not showing anything useful.

A simple: for i in $(seq 0 64) ; do dd if=/dev/urandom of=file$i bs=1M count=16 ; done executed in an ext4 filesystem mounted on a single SATA drive connected with a Marvell SATA PCIe card, will hang the system. And it's reproducible. It can also reproduced by doing: md5sum -c MD5SUMS

[stsp]

Any updates to this? Seems like I am having this too with the vanilla kernel but not with marvell's 3.2 kernel. I am using mvsas driver (my board has sas controller), and on hdd activity there is a lock-up.

[stsp]

Also this may be related: http://www.spinics.net/lists/arm-kernel/msg411431.html

[stsp]

By reverting the aforementioned patch and also disabling pmsu, one of our engineers seem to have got a stable system, no PCIe hangs. Can this help?

[tpetazzoni]

I will be taking care of this issue next week.

[stsp]

It seems the hang happens only when the pmsu is in DEEP_IDLE state. There is a patch: http://lkml.iu.edu/hypermail/linux/kernel/1503.1/05016.html With this patch we do not observe ever entering the DEEP_IDLE state, end neither the lock-up. Of course I suspect this is not a real fix at all. Maybe it will enter DEEP_IDLE state under some other conditions and will hang again.

[tpetazzoni]

Reverting http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/drivers/bus/mvebu-mbus.c?id=1737cac6936938a9bc52c03c4a3ff2032c702fa5 only should fix the problem. I don't really see why reverting another patch would be needed.

Are you working on Armada XP or Armada 38x ?

[stsp]

I am on armada-XP, but the difference is that I use mvsas controller instead of sata. Please note that the patch http://lkml.iu.edu/hypermail/linux/kernel/1503.1/05016.html may probably already be applied for you, so you may not notice the problem because of that.

[tpetazzoni]

I sent a patch which fixes DMA problems on Armada XP, see http://lists.infradead.org/pipermail/linux-arm-kernel/2015-May/346937.html.

I will test after reverting the patch http://lkml.iu.edu/hypermail/linux/kernel/1503.1/05016.html, to see if I can reproduce the issue reported by @stsp

[tpetazzoni]

I just did a test after reverting "cpuidle: mvebu: Update cpuidle thresholds for Armada XP SOCs", running bonnie++ with a 8GB size on an ext4 filesystem mounted over a PCIe SATA card, and it worked alright.

I'll close the bug for now because I believe things are working. If not, don't hesitate to reopen, but please provide a detailed test case for us to reproduce the problem.

[stsp]

Thomas, do you have a git tree on which I can test things to make sure we are testing the same thing? For now I am just collecting the patches from here and there to make things barely working, so our tests do not match.

[stsp]

Please note that I can't re-open this bug (not that I wanted, but still). You might have locked that possibility somehow.

[tpetazzoni]

@stsp Can you test with the branch at https://github.com/MISL-EBU-System-SW/mainline-public/tree/4.1/mvebu-mbus-fixes-new, using mvebu_v7_defconfig as the configuration?

[stsp]

No because your defconfig doesn't support mvsas (and many other things I need). I'll be testing your branch with my config now.

[stsp]

OK, I want to re-open this bug because this command: dd if=/dev/sda of=/dev/null bs=1M count=1000000 iflag=direct immediately hangs the system. Here's the git log so that you know what am I up to:

commit c0091b649bcc4462b94e8913a5c263dc4b3382fa Author: stsp Date: Thu May 28 13:35:29 2015 +0300

Revert "ARM: Blacklist GCC 4.8.0 to GCC 4.8.2 - PR58854"

This reverts commit 7fc150543c73de71859631c8a6b17e3067fe7617.

commit 8af2007c8ccb95a7a0c04d6b290830a6061fe8f7 Author: stsp Date: Thu May 28 13:32:23 2015 +0300

Revert "cpuidle: mvebu: Update cpuidle thresholds for Armada XP SOCs"

This reverts commit ce6031c89a35cffd5a5992b08377b77f49a004b9.

commit 00f8d638b33d3243c8720675024f7a670bb82efd Author: Thomas Petazzoni thomas.petazzoni@free-electrons.com Date: Wed May 27 10:42:00 2015 +0200

bus: mvebu-mbus: add mv_mbus_dram_info_nooverlap()

[stsp]

I will now mail you my config and dts as it seems github doesn't support file attachments.

[tpetazzoni]

Can you confirm on which kernel version / branch exactly you can reproduce this issue? Can you test with the internal SATA ?

[stsp]

Can you confirm on which kernel version / branch exactly you can reproduce this issue?

See the git log above: I am on commit 00f8d63, which is of the branch 4.1/mvebu-mbus-fixes-new.

Can you test with the internal SATA ?

I'll try, this will take a few days perhaps.

Can you try my binary kernel I've mailed you or my config? Maybe you'll reproduce...

[stsp]

Can you test with the internal SATA ?

Oh, if you mail me the u-boot image for armada-xp-gp board, then yes! :) So far I wasn't able to build anything that works.

[stsp]

Hello Thomas. I've finally booted my RD-78460-SERVER board and reproduced the lock-up there as well, with exactly same kernel. So mv-sata is affected as well. What can I do to help you reproduce the problem?

[tpetazzoni]

I'm currently away this week. But maybe @gclement can try to reproduce. If not, I'll try to reproduce next week.

[stsp]

Thanks! If Gregory also needs my config or binary kernel with appended DT - please tell me and I'll mail it.

In fact, I'd like to test with defconfig and default DT to reduce the problem of reproducing this. Could you please hint me what DT should be used for RD-78460-SERVER board?

[stsp]

OK, reproduced with defconfig and armada-xp-gp dts.

[gclement]

OK so I will try on the Armada XP GP board

[stsp]

The test command is: dd if=/dev/sda of=/dev/null bs=1M count=1000000 iflag=direct

My board is RD-78460-SERVER, not sure how it differs from XP GP.

[stsp]

Please don't forget about git revert ce6031c89

[gclement]

I didn't mange to reproduce it, I had I/O error before (But it is possible that my board was damaged). Could you sent the kernel image and your .config?

[stsp]

Will do tomorrow, thanks for trying. Although since I've used the default config and default DT, it will unlikely help... Just to make sure, you reverted ce6031c before testing, right? Do you have the RD-78460-SERVER reference board somewhere? Also can we somehow compare the CPU cut/stepping/whatever? Maybe your soc is somehow a bit newer than mine?

[stsp]

Also please add "pm_disable" to your bootargs. That's how I tested.

[gclement]

Having your binary may help to be sure that it was not a compiler related problem. From U-Boot I got: Board: RD-AXP-GP rev 1.0 SoC: MV78460 B0 running 4 CPUs Custom configuration CPU: Marvell PJ4B (584) v7 (Rev 2) LE. I didn't used pm_disable, I will try with this one

[stsp]

I didn't used pm_disable, I will try with this one

Please don't - I already tested and made sure it is irrelevant. I'll mail you everything very soon.

[stsp]

Gregory, I mailed you my kernel (with DT), config and u-boot. Please don't tell me its still not reproduceable. :))

Here's my /proc/cpuinfo:

processor : 0 model name : ARMv7 Processor rev 2 (v7l) BogoMIPS : 1594.16 Features : half thumb fastmult vfp edsp thumbee vfpv3 tls idiva idivt vfpd32 lpae CPU implementer : 0x56 CPU architecture: 7 CPU variant : 0x2 CPU part : 0x584 CPU revision : 2

(4 CPUs of that kind)

[stsp]

Board: RD-AXP-GP rev 1.0
SoC: MV78460 B0
running 4 CPUs
Custom configuration
CPU: Marvell PJ4B (584) v7 (Rev 2) LE

That's what I get from u-boot. So either our CPUs are similar, or u-boot just configured that way and doesn't read the actual info.

[gclement]

Finally I managed to reproduce the issue. Actually, I didn't use the mvebu_v7_defconfig in the first place. The other change I made was initially I used a initramfs, now I use an nfsroot. The last change is that I perform the dd on a small SSD disk on sda whereas initially I used a "real" hard drive on sdb.

[stsp]

Wow!!! Great news! Please keep me updated.

[stsp]

Any news? I would be especially interested to know with what exactly config you were not able to reproduce it initially? Is there some macigal config that just works?

[tpetazzoni]

I've tested with Linux 4.1, mvebu_v7_defconfig with no change except using an initramfs, on Armada XP GP, with a 8 GB SSD drive, and dd works perfectly fine:

# /usr/bin/dd if=/dev/sda of=/dev/null bs=1M count=1000000 iflag=direct
7627+1 records in
7627+1 records out
7998455808 bytes (8.0 GB) copied, 163.763 s, 48.8 MB/s
# 

I.e, no problem at all.

[stsp]

Gregory, could you help Thomas reproducing?

Thomas, my testing shows that reproducing with SSD is more difficult. Do you have HDD?

[tpetazzoni]

@stsp Test started on a 500 GB HDD.

[stsp]

If you don't reproduce it in 10 seconds, you probably can't, so don't spend too much time on testing. Hope Gregory can help... He said he can reproduce.

[stsp]

Just to make sure, you reverted ce6031c before testing, right?

[tpetazzoni]

No, I did not revert ce6031c before testing. So what you say is that with the current kernel it works thanks to the cpuidle threshold change, but if we revert back to the more aggressive thresholds, it no longer works?

And indeed, I have been running the dd test on a HDD since 20 minutes, and the board is still alive so far.

[stsp]

Yes, exactly. I mentioned many times in this thread, that commit ce6031c obscures the problem by never entering DEEP_IDLE state. That was the trick to hide the real bug perhaps (it comes from some marvell patch). You need to revert that commit, and, as soon as you in DEEP_IDLE state, you lock up.

[stsp]

And indeed, I have been running the dd test on a HDD since 20 minutes, and the board is still alive so far.

Don't spend much time on that. Better to do 5 1-minute tests changing SSD/HDD than 1 20-minute test, because the problem either reproduces immediately, or not at all. Though I hope after reverting the commit you'll have no problems reproducing.

[tpetazzoni]

I confirm I can reproduce the issue after reverting ce6031c.

[stsp]

Yeah! Great news, thank you. I wish such a patch is in fact reverted: it secretly disables DEEP_IDLE, which is not only obscuring the bug but is also deteriorating the power saving. Such things must be clearly changelogged, or never done. Gross.

[tpetazzoni]

@stsp Contrary to what you say, ce6031c does not make it impossible to enter the idle state. Indeed in the default mvebu_v7_defconfig, NO_HZ is not enabled, so we have a tick every 10 ms, so we never enter DEEP_IDLE. However, if you enable NO_HZ, the 10 ms tick is disabled when the system is idle. And we indeed enter DEEP_IDLE, both when the system is idle and when the dd program is running. But I can't reproduce the system freeze in this case.

[tpetazzoni]

@stsp Correction: the freeze just occured. It just took longer.

[stsp]

Thanks for an update. I was about to start testing with NO_HZ. Now this won't be needed.

[stsp]

Now this means you can reproduce the bug even without any reverts - very good!

[tpetazzoni]

Reverting the patch, I could reproduce the problem faster, and this time got a:

[ 218.943155] Unable to handle kernel NULL pointer dereference at virtual address 00000000

but nothing else.