Known issue: L2 cache flushing due to StarFive 7100 architecture

pdp7 commented 3 years ago

NOTE: I am reposting my forum post here for greater visibility.

The BeagleV beta boards have the StarFive 7100 SoC which was from a limited multi-project wafer run. We only have enough chips to produce 300 BeagleV beta boards. StarFive should be releasing a public datasheet next week for the 7100. The 7100 has 2x StarFive U74 cores.

StarFive will be putting in a full production order an updated design called the 7110. This will have 4x SiFive U74 cores as well as Imagination GPU and PCIe controller.

This brings me to design issues in the 7100:

L2 cache controller has a bug where it incorrectly raises the L2 DirFail interrupt. The only work around for this is to mask the L2 cache interrupt on the PLIC.
All the peripherals in the 7100 are on a non-coherent bus. StarFive refers to this as the sysbus port. This requires the drivers to flush L2 cache in many instances where it is not necessary on other SoC’s. This design choice reduces the performance of the 7100.

StarFive tells me that they have resolved the L2 cache controller DirFail interrupt bug in the 7110. And more importantly they have improved the SoC architecture. High performance peripherals like the Gigabit Ethernet and USB3 controller will be connected to a cache-coherent bus (StarFive refers to it as the front port).

My understanding so far is that it may be difficult to upstream the 7100 support given that the peripherals are not coherent. I would very much like to get input from you and others in the beta as to what the correct way to implement the drivers might be.

I also need to work with StarFive to create some diagrams of the SoC architecture that shows exactly how the different IP blocks are interconnected and whether or not those connections are coherent.

Thank you, Drew Fustini BeagleBoard.org Foundation

pdp7 commented 3 years ago

Please see this issue for discussion of L2 cache flushing specific to the u-boot code:

https://github.com/starfive-tech/beagle_uboot-opensbi/issues/1

pdp7 commented 3 years ago

refer to https://github.com/starfive-tech/sft-riscv-linux-5.10/issues/2

xypron commented 3 years ago

Would the following observation match the problem you reported?

# dd if=/dev/sda of=/dev/null bs=16M count=64

leading to


[ 1972.045204] L2CACHE: flush64 out of range: 2080200000(80000), skip flush
[ 1972.091354] L2CACHE: flush64 out of range: 2080200000(80000), skip flush
[ 1972.133556] L2CACHE: flush64 out of range: 2080200000(80000), skip flush
[ 1972.176994] L2CACHE: flush64 out of range: 2080200000(80000), skip flush
[ 1972.219621] L2CACHE: flush64 out of range: 2080200000(80000), skip flush
[ 1972.262798] L2CACHE: flush64 out of range: 2080200000(80000), skip flush

xypron commented 3 years ago

The problem seems to stop us from using an USB connected drive:

[   24.670434] usbcore: registered new interface driver uas
[   24.670760] L2CACHE: flush64 out of range: 2080200000(24), skip flush
[   24.671320] L2CACHE: flush64 out of range: 2080200000(60), skip flush
[   24.671868] scsi 0:0:0:0: Direct-Access     Samsung  SSD 860 EVO 1TB  4101 PQ: 0 ANSI: 6
[   24.672010] L2CACHE: flush64 out of range: 2080200000(ff), skip flush
[   24.672478] L2CACHE: flush64 out of range: 2080200000(ff), skip flush
[   24.672909] L2CACHE: flush64 out of range: 2080200000(ff), skip flush
[   24.673345] L2CACHE: flush64 out of range: 2080200000(ff), skip flush
[   24.673877] L2CACHE: flush64 out of range: 2080200000(1000), skip flush
[   24.676117] L2CACHE: flush64 out of range: 2080200000(20), skip flush
[   24.676452] sd 0:0:0:0: Attached scsi generic sg0 type 0
[   24.676961] L2CACHE: flush64 out of range: 2080200000(40), skip flush
[   24.677387] L2CACHE: flush64 out of range: 2080200000(40), skip flush
[   24.677768] L2CACHE: flush64 out of range: 2080200000(40), skip flush
[   24.678083] sd 0:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[   24.678101] sd 0:0:0:0: [sda] 4096-byte physical blocks
[   24.678200] L2CACHE: flush64 out of range: 2080200000(4), skip flush
[   24.688780] sd 0:0:0:0: [sda] Write Protect is off
[   24.688802] sd 0:0:0:0: [sda] Mode Sense: 53 00 00 08
[   24.688941] L2CACHE: flush64 out of range: 2080200000(4), skip flush
[   24.689404] L2CACHE: flush64 out of range: 2080200000(20), skip flush
[   24.689835] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   24.689953] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.690538] L2CACHE: flush64 out of range: 2080200000(40), skip flush
[   24.690897] sd 0:0:0:0: [sda] Optimal transfer size 33553920 bytes not a multiple of physical block size (4096 bytes)
[   24.760643] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.760941] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.761213] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.761484] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.761754] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.762066] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.762352] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.762690] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.763905] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.764259] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.764616] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.764972] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.765326] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.765689] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.766044] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.766419] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.767054] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.767402] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.767750] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.768100] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.768274] Unable to handle kernel paging request at virtual address 0000005f81a4e6b2
[   24.768291] Oops [#1]
[   24.768296] Modules linked in: uas usb_storage
[   24.768326] CPU: 0 PID: 176 Comm: systemd-udevd Tainted: G        W         5.10.6+ #26
[   24.768336] epc: ffffffdf81a4e6b2 ra : ffffffe00090bf54 sp : ffffffe084a6f980
[   24.768342]  gp : ffffffe0018416a8 tp : ffffffe080615040 t0 : 0000000000000040
[   24.768349]  t1 : ffffffe1fed7d000 t2 : 0000000000000000 s0 : ffffffe084a6f9b0
[   24.768360] L2CACHE: flush64 out of range: 2080200000(200), skip flush
[   24.768367]  s1 : ffffffe08603bc00 a0 : ffffffe08603bc00 a1 : ffffffe001843228
[   24.768374]  a2 : ffffffe001843228 a3 : 0000000000000000 a4 : ffffffe08019f800
[   24.768381]  a5 : ffffffdf81a4e6b2 a6 : 0000000000000000 a7 : 000000000000001f
[   24.768389]  s2 : ffffffe0807f6000 s3 : 0000000000000000 s4 : ffffffe084c3b680
[   24.768397]  s5 : ffffffe0018ab6d8 s6 : ffffffe00004b480 s7 : ffffffe0807f61ac
[   24.768405]  s8 : ffffffe00004b480 s9 : ffffffe00004b330 s10: ffffffe001844208
[   24.768410]  s11: ffffffe084a73318 t3 : 000000000000007f t4 : 0000000000000402
[   24.768417]  t5 : ffffffe0842f65a8 t6 : ffffffe0842f65b0
[   24.768423] status: 0000000200000120 badaddr: 0000005f81a4e6b2 cause: 000000000000000c
[   24.768551] ---[ end trace 6c3bac6aaa60c4fa ]---
[   24.768564] Kernel panic - not syncing: Fatal exception in interrupt
[   24.768574] SMP: stopping secondary CPUs
[   27.015675] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

pdp7 commented 3 years ago

@xypron I believe this is related to https://github.com/starfive-tech/beaglev_fedora/issues/3

@tekkamanninja @MichaelZhuxx would you be able to comment as to your current understanding of this issue?

MichaelZhuxx commented 3 years ago

yes,that's a know issue about L2 cache

pdp7 commented 3 years ago

FYI - note on JH7100 Cache Coherence has been published https://github.com/starfive-tech/beaglev_doc/blob/main/JH7100%20Cache%20Coherence%20V1.0.pdf

stffrdhrn commented 3 years ago

Should we close the fedora issue and link to this one? Also the excessive logging bug.

https://github.com/starfive-tech/Fedora_on_StarFive/issues/3
https://github.com/starfive-tech/Fedora_on_StarFive/issues/20 - excessive logging

pdp7 commented 3 years ago

@stffrdhrn yes, that is a good idea I think.

pdp7 commented 3 years ago

_[This is a comment from @stffrdhrn in duplicate issue]_

I understand this is only expected in the VIC7100. But, this seems pretty consistent happening with this address 2080200000.

On my boad plugging in a USB drive I also see something like the below.
[riscv@fedora-starfive ~]$ sudo vgs
L2CACHE: flush64 out of range: 2080200000(20000), skip flush
L2CACHE: flush64 out of range: 2080200000(20000), skip flush
  WARNING: PV /dev/sda1 in VG data2 is using an old PV header, modify the VG to update.
  VG    #PV #LV #SN Attr   VSize     VFree
  data2   1   1   0 wz--n- <1024.00g <774.00g
This seems related to this kernel patch: esmil/linux@1ab9c4c

I am guessing we are somewhere calling flush where we shouldn't. Do we know which call to flush is actually causing the noise above with usb? It seems like the above patch to flush dcache for every write is to workaround the issue of memory writes not working correctly on the VIC7100.

Some links:

https://forums.sifive.com/t/flush-invalidate-l1-l2-on-the-u54-mc/4483

1

So I suspect if we fix up the dcache flush workaround the USB may be more stable. Should this issue be moved to linux instead of fedora?

geertu commented 3 years ago

@pdp7 @stffrdhrn

L2CACHE: flush64 out of range: 2080200000(20000), skip flush L2CACHE: flush64 out of range: 2080200000(20000), skip flush

I stuck a WARN() in the cache flushing code, so it will print a backtrace when the issue happens again

https://github.com/esmil/linux/pull/9

stffrdhrn commented 3 years ago

@geertu Thank's, that's what I was thinking to do you beat me to it. For me these are all coming from uas_alloc_data_urb which is accessed via scsi / blk layer. This is consistent with people complaining about mass storage io having problems.

The below shows up when I plug in a 1TB HDD drive via a USB hub (the hub is needed for powering the disk).

During bootup

L2CACHE: flush64 out of range: 2080200000(40), skip flush
WARNING: CPU: 0 PID: 7 at drivers/soc/sifive/sifive_l2_cache.c:146 sifive_l2_flush64_range+0x8e/0xaa
CPU: 0 PID: 7 Comm: kworker/u4:0 Tainted: G        W         5.13.0-rc2-starlight-00343-g445714951383-dirty #477
Hardware name: BeagleV Starlight Beta (DT)
Workqueue: events_unbound async_run_entry_fn
epc : sifive_l2_flush64_range+0x8e/0xaa
 ra : sifive_l2_flush64_range+0x8e/0xaa
epc : ffffffff803a2c36 ra : ffffffff803a2c36 sp : ffffffe07fefb6b0
 gp : ffffffff812c9278 tp : ffffffe07fecbc00 t0 : ffffffd0045ec110
 t1 : 0000000000000000 t2 : ffffffff80c528a8 s0 : ffffffe07fefb6c0
 s1 : ffffffe080dc7600 a0 : 0000000000000039 a1 : ffffffff81282328
 a2 : 00000000ffffefff a3 : 0000000000000000 a4 : ffffffff81212268
 a5 : ffffffff81212268 a6 : ffffffe07ffaa850 a7 : ffffffffad55ad55
 s2 : ffffffe08136bd10 s3 : ffffffe081328790 s4 : 0000000000000040
 s5 : ffffffe080ae7800 s6 : ffffffe080ae7800 s7 : 0000000200000022
 s8 : 0000000200000020 s9 : ffffffff80435dd0 s10: ffffffe080cfbc00
 s11: ffffffe07fefb8d0 t3 : 0000000000000001 t4 : 0000000000000002
 t5 : 0000000000000010 t6 : ffffffe07fefb410
status: 0000000200000100 badaddr: ffffffe07fecbc20 cause: 0000000000000003
Call Trace:
[<ffffffff803a2c36>] sifive_l2_flush64_range+0x8e/0xaa
[<ffffffff804ea6f4>] uas_alloc_data_urb.constprop.0+0x72/0xb2
[<ffffffff804ea876>] uas_submit_urbs+0x142/0x3ce
[<ffffffff804eacb6>] uas_queuecommand+0xe2/0x1f6
[<ffffffff80437352>] scsi_queue_rq+0x2f2/0x7de
[<ffffffff802f8572>] blk_mq_dispatch_rq_list+0xd8/0x6fa
[<ffffffff802fd998>] __blk_mq_sched_dispatch_requests+0xd4/0x146
[<ffffffff802fdb2a>] blk_mq_sched_dispatch_requests+0x2c/0x56
[<ffffffff802f700e>] __blk_mq_run_hw_queue+0x4c/0x74
[<ffffffff802f71be>] __blk_mq_delay_run_hw_queue+0x188/0x18e
[<ffffffff802f845e>] blk_mq_run_hw_queue+0x6c/0xa8
[<ffffffff802fdd3e>] blk_mq_sched_insert_request+0x92/0xf6
[<ffffffff802f3a44>] blk_execute_rq_nowait+0x42/0x52
[<ffffffff802f3a88>] blk_execute_rq+0x34/0x74
[<ffffffff804367a0>] __scsi_execute+0xd0/0x166
[<ffffffff8043158a>] scsi_vpd_inquiry+0x56/0x94
[<ffffffff8043186e>] scsi_get_vpd_page+0x34/0x9e
[<ffffffff804428aa>] sd_revalidate_disk.isra.0+0x14e8/0x1b36
[<ffffffff8044327e>] sd_probe+0x240/0x386
[<ffffffff8040dd02>] really_probe+0x9c/0x3a4
[<ffffffff8040e056>] driver_probe_device+0x4c/0x94
[<ffffffff8040e1ce>] __device_attach_driver+0xa4/0x102
[<ffffffff8040c634>] bus_for_each_drv+0x4a/0x7c
[<ffffffff8040d76c>] __device_attach_async_helper+0x70/0x84
[<ffffffff80025e98>] async_run_entry_fn+0x18/0x9c
[<ffffffff8001cfc8>] process_one_work+0x186/0x2ba
[<ffffffff8001d228>] worker_thread+0x12c/0x402
[<ffffffff800228f0>] kthread+0xda/0xe8
[<ffffffff80003046>] ret_from_exception+0x0/0xc

Running vgs (since the disk is LVM)

[riscv@fedora-starfive ~]$ sudo vgs
[sudo] password for riscv: 
------------[ cut here ]------------
L2CACHE: flush64 out of range: 2080200000(20000), skip flush
WARNING: CPU: 1 PID: 411 at drivers/soc/sifive/sifive_l2_cache.c:146 sifive_l2_flush64_range+0x8e/0xaa
CPU: 1 PID: 411 Comm: vgs Tainted: G        W         5.13.0-rc2-starlight-00343-g445714951383-dirty #477
Hardware name: BeagleV Starlight Beta (DT)
epc : sifive_l2_flush64_range+0x8e/0xaa
 ra : sifive_l2_flush64_range+0x8e/0xaa
epc : ffffffff803a2c36 ra : ffffffff803a2c36 sp : ffffffe08d43f6d0
 gp : ffffffff812c9278 tp : ffffffe080b14600 t0 : ffffffd0043d0110
 t1 : 0000000000000000 t2 : ffffffff80c528a8 s0 : ffffffe08d43f6e0
 s1 : ffffffe080dc7b40 a0 : 000000000000003c a1 : ffffffff81273820
 a2 : 0000000000000001 a3 : 0000000000000008 a4 : ffffffe1f8d0d558
 a5 : ffffffffffffffd8 a6 : ffffffe07ffabbb0 a7 : ffffffffad55ad55
 s2 : ffffffe086ad1c10 s3 : ffffffe081328790 s4 : 0000000000020000
 s5 : ffffffe080ae7800 s6 : ffffffe080ae7800 s7 : 0000000200000022
 s8 : 0000000200000020 s9 : ffffffff80435dd0 s10: ffffffe080cfbc00
 s11: ffffffe08d43f8f0 t3 : 0000000000000001 t4 : 0000000000000002
 t5 : 0000000000000010 t6 : ffffffe08d43f430
status: 0000000200000100 badaddr: ffffffe080b14620 cause: 0000000000000003
Call Trace:
[<ffffffff803a2c36>] sifive_l2_flush64_range+0x8e/0xaa
[<ffffffff804ea6f4>] uas_alloc_data_urb.constprop.0+0x72/0xb2
[<ffffffff804ea876>] uas_submit_urbs+0x142/0x3ce
[<ffffffff804eacb6>] uas_queuecommand+0xe2/0x1f6
[<ffffffff80437352>] scsi_queue_rq+0x2f2/0x7de
[<ffffffff802f8572>] blk_mq_dispatch_rq_list+0xd8/0x6fa
[<ffffffff802fd752>] __blk_mq_do_dispatch_sched+0x198/0x1ea
[<ffffffff802fd9d4>] __blk_mq_sched_dispatch_requests+0x110/0x146
[<ffffffff802fdb2a>] blk_mq_sched_dispatch_requests+0x2c/0x56
[<ffffffff802f700e>] __blk_mq_run_hw_queue+0x4c/0x74
[<ffffffff802f71be>] __blk_mq_delay_run_hw_queue+0x188/0x18e
[<ffffffff802f845e>] blk_mq_run_hw_queue+0x6c/0xa8
[<ffffffff802fde26>] blk_mq_sched_insert_requests+0x84/0x10e
[<ffffffff802f97ee>] blk_mq_flush_plug_list+0xa8/0xe4
[<ffffffff802efdc4>] blk_flush_plug_list+0x98/0xac
[<ffffffff802efdf0>] blk_finish_plug+0x18/0x24
[<ffffffff801461cc>] blkdev_direct_IO+0x3bc/0x412
[<ffffffff800c3fba>] generic_file_read_iter+0x6a/0x114
[<ffffffff80145ade>] blkdev_read_iter+0x32/0x48
[<ffffffff8015680c>] aio_read.constprop.0+0x88/0xf8
[<ffffffff80157ffc>] io_submit_one.constprop.0+0x422/0x650
[<ffffffff80158d9a>] sys_io_submit+0x6e/0x14c
[<ffffffff80003038>] ret_from_syscall+0x0/0x2
---[ end trace d0a696e8a86ca45f ]---
------------[ cut here ]------------

stffrdhrn commented 3 years ago

I think this is happening because this code passes in a NULL for the transfer buffer. The dcache flush code translates null to 2080200000.

static struct urb *uas_alloc_data_urb(struct uas_dev_info *devinfo, gfp_t gfp,
                                      struct scsi_cmnd *cmnd,
                                      enum dma_data_direction dir)
{
        struct usb_device *udev = devinfo->udev;
        struct uas_cmd_info *cmdinfo = (void *)&cmnd->SCp;
        struct urb *urb = usb_alloc_urb(0, gfp);
        struct scsi_data_buffer *sdb = &cmnd->sdb;
        unsigned int pipe = (dir == DMA_FROM_DEVICE)
                ? devinfo->data_in_pipe : devinfo->data_out_pipe;

        if (!urb)
                goto out;
        usb_fill_bulk_urb(urb, udev, pipe, NULL, sdb->length,    <--- here
                          uas_data_cmplt, cmnd);

stffrdhrn commented 3 years ago

This patch fixes the issue for me.

diff --git a/include/linux/usb.h b/include/linux/usb.h
index 4cc4e39a3ac9..5eafbb8db2df 100644
--- a/include/linux/usb.h
+++ b/include/linux/usb.h
@@ -1612,14 +1612,18 @@ struct urb {
 #include <soc/starfive/vic7100.h>
 static inline void cdns_flush_dcache(unsigned long start, unsigned long len)
 {
+       if (start == 0UL)
+               return;
+
        starfive_flush_dcache(_ALIGN_DOWN(start, 64), len + (start & 63));
 }

 static inline void cdns_virt_flush_dcache(void *virt_start, unsigned long len)
 {
-       unsigned long start = dw_virt_to_phys(virt_start);
+       unsigned long start = (virt_start == NULL)
+                               ? 0UL : dw_virt_to_phys(virt_start);

-       starfive_flush_dcache(_ALIGN_DOWN(start, 64), len + (start & 63));
+       cdns_flush_dcache(start, len);
 }
 #endif

[riscv@fedora-starfive ~]$ sudo vgs
[sudo] password for riscv: 
  WARNING: PV /dev/sda1 in VG data2 is using an old PV header, modify the VG to update.
  VG    #PV #LV #SN Attr   VSize     VFree   
  data2   1   1   0 wz--n- <1024.00g <774.00g

stffrdhrn commented 3 years ago

I submitted a patch for the above : https://github.com/esmil/linux/pull/10

pdp7 commented 3 years ago

@stffrdhrn thank you very much! this is great progress. great to see it is now in @esmil starlight branch https://github.com/esmil/linux/commit/a22fc12294bf6621fdc4a5a346b201ab49a09bb7

cc @MichaelZhuxx @tekkamanninja

pdp7 commented 3 years ago

I think this is an interesting idea Nick Kossifidis (@mickflemm):

Maybe we can hack our way using a fake IOMMU driver, I haven’t thought this through much but this was my initial idea:

a) Create a fake IOMMU driver that its reg property on the device tree is a reserved DRAM region from the cached range (e.g. the top XMBytes, removed from the /memory node), and the same region from the uncached alias.

b) Attach the devices to that fake IOMMU driver (with the iommus property), this would allow us to hook dma_ops when dma_alloc_coherent is being called, so basicaly dma_alloc_coherent will end up calling our fake IOMMU driver.

c) When ops->alloc() is caled, the IOMMU driver will return physical addresses from the reserved 32bit addressable DRAM region to give to the device, and virtual addresses pointing to the same region on the uncached alias to give to the CPU. So the device would use DRAM with 32bit addressing, and the CPU would see the same region from the uncacheable alias, bypassing the cache.

I’ll need to read the DMA/IOMMU APIs more but if we can make it work we 'll only need to upstream that fake IOMMU driver (that may be useful to others as well) and change the device tree, we won’t need to modify the drivers or anything on /arch.

(copied from the BeagleV Beta developer forum)

@tekkamanninja @tpetazzoni @palmer-dabbelt @stffrdhrn @avpatel @geertu @teknoraver @kbingham @mripard Any feedback on this approach of creating a 'fake' IOMMU driver to avoid having L2 Cache flush hacks for GMAC, MMC and USB drivers?

avpatel commented 3 years ago

Hi Nick, your idea of fake/dummy IOMMU sounds good to me. We will anyway need something similar for 32bit devices on a RV64 system.

The ARM64 kernel did something similar for legacy 32bit devices so maybe that can be good reference. For RV64, we need to use IOMMU for uncached region and 32bit devices.

Having said above, this only solves half-problem for DMA mappings. We still need a way to explicitly sync cached memory instead of using bounce buffers in uncached regions.

The RV CMO extension will take time probably more than a year so I propose we define a SBI DMA sync call.

The Linux RISC-V kernel, will provide arch specific DMA sync implementation which will: 1) Prefer CMO whenever it is available (in future) 2) Use SBI DMA sync if that is available 3) Do nothing if there is no way to sync

Guo Ren had proposed SBI DMA sync call but there should a SBI spec definition. Someone please volunteer and prepare writeup SBI DMA sync patch for the SBI spec.

Regards, Anup