search

OP-TEE / optee_os

Trusted side of the TEE
Other
1.59k stars 1.07k forks source link

Memory limit error OP-TEE driver #6019

Closed Yuhala closed 1 year ago

Yuhala commented 1 year ago

I am building a test OP-TEE driver for an I2S device on an Nvidia Jetson AGX Xavier. I have begun by mapping the I2S base register of my device (paddr_t pa = 0x2901000) to OP-TEE's virtual address space using the core_mmu_add_mapping routine as such:

 vaddr_t va = (vaddr_t)core_mmu_add_mapping(MEM_AREA_IO_SEC, pa, 0x100);

The mapping above succeeds and does not produce any error.

I then have a driver function tegra_i2s_read_i2s_mem_range which tries to write and read a range of the mapped memory as shown below (MEMORY_CHUNK = 8). The value in i2s_device.i2s_base.va is the same as va above.

static TEE_Result tegra_i2s_read_i2s_mem_range(uint32_t iter, uint8_t *res)
{
    TEE_Result rc = TEE_SUCCESS;

    uint8_t content;
    uint8_t val = 77;
    vaddr_t base = i2s_device.i2s_base.va;    

    for (uint32_t i = 0; i < iter; i++)
    {
        // write and read data in registers

        io_write8(base + MEMORY_CHUNK, val);
        DMSG(">>>>>>>>>> TEGRA I2S DRIVER::after io_write8  >>>>>>");
        content = io_read8(base + MEMORY_CHUNK);
        *res = content;
        DMSG(">>>>>>>>>> TEGRA I2S DRIVER::after io_read8   >>>>>>");

    }

    DMSG(">>>>>>>>>> TEGRA I2S DRIVER::read_mem_range END >>>>>>");
    return rc;
}

This driver function is invoked from a PTA routine, which itself is invoked from a TA. The TA-PTA connection succeeds (tried with other test routines) but the driver function tegra_i2s_read_i2s_mem_range fails. Below is part of the log result when it is called with param iter = 1.

D/TA:  TA_CreateEntryPoint:40 has been called
D/TA:  __GP11_TA_OpenSessionEntryPoint:159 has been called
I/TA: Hello World!
F/TC:? 0 plat_prng_add_jitter_entropy:72 0x41
D/TA:  inc_value:196 has been called
I/TA: Got value: 42 from NW
I/TA: Increase value to: 43
D/TA:  connect_to_jetson_i2s_pta:69 >>>>>>>> Trying to connect to Jetson I2S PTA >>>>>>>
F/TC:? 0 trace_syscall:151 syscall #8 (syscall_check_access_rights)
F/TC:? 0 trace_syscall:151 syscall #5 (syscall_open_ta_session)
D/TC:? 0 tee_ta_init_pseudo_ta_session:296 Lookup pseudo TA 8d2db51d-de52-44ca-a410-445906386bf1
D/TC:? 0 tee_ta_init_pseudo_ta_session:309 Open jetson_i2s.pta
D/TC:? 0 tee_ta_init_pseudo_ta_session:326 jetson_i2s.pta : 8d2db51d-de52-44ca-a410-445906386bf1
D/TA:  connect_to_jetson_i2s_pta:92 >>>>>>>> After open TA session >>>>>>>
F/TC:? 0 trace_syscall:151 syscall #8 (syscall_check_access_rights)
F/TC:? 0 trace_syscall:151 syscall #8 (syscall_check_access_rights)
F/TC:? 0 trace_syscall:151 syscall #8 (syscall_check_access_rights)
F/TC:? 0 trace_syscall:151 syscall #5 (syscall_open_ta_session)
D/TC:? 0 tee_ta_init_pseudo_ta_session:296 Lookup pseudo TA 3a2f8978-5dc0-11e8-9c2d-fa7ae01bbebc
D/TC:? 0 tee_ta_init_pseudo_ta_session:309 Open system.pta
D/TC:? 0 tee_ta_init_pseudo_ta_session:326 system.pta : 3a2f8978-5dc0-11e8-9c2d-fa7ae01bbebc
F/TC:? 0 trace_syscall:151 syscall #8 (syscall_check_access_rights)
F/TC:? 0 trace_syscall:151 syscall #7 (syscall_invoke_ta_command)
F/TC:? 0 trace_syscall:151 syscall #7 (syscall_invoke_ta_command)
D/TC:? 0 tegra_i2s_execute_cmd:229 >>>>>>>>>> tegra_i2s_execute_cmd:: cmd_id = 22
��[ 5128.255618] CPU:0, Error: cbb-noc@2300000, irq=15
[ 5128.255844] **************************************
[ 5128.256036] CPU:0, Error:cbb-noc
[ 5128.256219]  Error Logger        : 0
[ 5128.256390]  ErrLog0         : 0x80000008
[ 5128.256470]    Transaction Type  : WR  - Write, Incrementing
[ 5128.256587]    Error Code        : SLV
[ 5128.256661]    Error Source      : Target
[ 5128.256733]    Error Description : Target error detected by CBB slave
[ 5128.256870]    AXI2APB_1 bridge error: SFIFONE - Status FIFO Not Empty interrupt
[ 5128.257021]    AXI2APB_1 bridge error: SLV - SLVERR interrupt
[ 5128.257133]    Packet header Lock    : 0
[ 5128.257225]    Packet header Len1    : 0
[ 5128.257306]    NOC protocol version  : version >= 2.7
[ 5128.257399]  ErrLog1         : 0x32000b
[ 5128.257463]  ErrLog2         : 0x0
[ 5128.257519]    RouteId       : 0x32000b
[ 5128.257594]    InitFlow      : ccroc_p2ps/I/ccroc_p2ps
[ 5128.257686]    Targflow      : axis_satellite_grout/T/axis_satellite_grout
[ 5128.257804]    TargSubRange      : 0
[ 5128.258011]    SeqId         : 0
[ 5128.258226]  ErrLog3         : 0x2901008
[ 5128.258866]  ErrLog4         : 0x0
[ 5128.261838]    Address accessed  : 0x2901008
[ 5128.266036]  ErrLog5         : 0x289f811
[ 5128.269106]    Non-Modify        : 0x1
[ 5128.272775]    AXI ID        : 0x5
[ 5128.275403]    Master ID     : CCPLEX
[ 5128.278641]    Security Group(GRPSEC): 0x7e
[ 5128.282842]    Cache         : 0x1 -- Bufferable 
[ 5128.287299]    Protection        : 0x0 -- Unprivileged, Secure, Data Access
[ 5128.293605]    FALCONSEC     : 0x0
[ 5128.296753]    Virtual Queuing Channel(VQC): 0x0
[ 5128.301660]  **************************************
[ 5128.306714] kernel BUG at drivers/soc/tegra/cbb/tegra194-cbb.c:2057!
[ 5128.313027] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[ 5128.318549] Modules linked in: lzo_rle(E) lzo_compress(E) zram(E) cfg80211(E) fuse(E) snd_soc_tegra186_dspk(E) snd_soc_tegra210_ope(E) snd_soc_tegra186_asrc(E) snd_soc_tegra186_arad(E) snd_soc_tegra210_iqc(E) snd_soc_tegra210_mvc(E) snd_soc_tegra210_afc(E) loop(E) snd_soc_tegra210_adx(E) snd_soc_tegra210_dmic(E) snd_soc_tegra210_amx(E) snd_soc_tegra210_admaif(E) snd_soc_tegra_pcm(E) snd_soc_tegra210_i2s(E) snd_soc_tegra210_mixer(E) snd_soc_tegra210_sfc(E) input_leds(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) ghash_ce(E) sha2_ce(E) snd_hda_codec_hdmi(E) sha256_arm64(E) sha1_ce(E) snd_soc_tegra210_adsp(E) ucsi_ccg(E) snd_soc_rt5659(E) snd_soc_tegra_machine_driver(E) snd_hda_tegra(E) typec_ucsi(E) snd_soc_tegra_utils(E) snd_soc_spdif_tx(E) snd_soc_simple_card_utils(E) binfmt_misc(E) snd_soc_rl6231(E) max77620_thermal(E) nct1008(E) typec(E) nvadsp(E) snd_soc_tegra210_ahub(E) bluedroid_pm(E) userspace_alert(E) tegra_bpmp_thermal(E) tegra210_adma(E) snd_hda_codec(E)
[ 5128.318779]  snd_hda_core(E) spi_tegra114(E) ahci(E) libahci(E) libata(E) ina3221(E) pwm_fan(E) nvgpu(E) nvmap(E) ip_tables(E) x_tables(E) [last unloaded: mtd]
[ 5128.418915] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           OE     5.10.104-tegra #3
[ 5128.427300] Hardware name: Unknown Jetson-AGX/Jetson-AGX, BIOS 2.1-32413640 01/24/2023
[ 5128.435186] pstate: 60400089 (nZCv daIf +PAN -UAO -TCO BTYPE=--)
[ 5128.441484] pc : tegra194_cbb_err_isr+0x19c/0x1b0
[ 5128.445943] lr : tegra194_cbb_err_isr+0x11c/0x1b0
[ 5128.450666] sp : ffff800010003df0
[ 5128.454081] x29: ffff800010003df0 x28: 0000000000000001 
[ 5128.459592] x27: 0000000000000080 x26: ffffb83b6ffb3f70 
[ 5128.465104] x25: ffffb83b70ceae10 x24: 0000000000000001 
[ 5128.470617] x23: ffffb83b70297000 x22: ffffb83b70a2ea00 
[ 5128.476298] x21: 000000000000000f x20: 0000000000000005 
[ 5128.481811] x19: ffffb83b70a2e9f0 x18: 0000000000000010 
[ 5128.486978] x17: 0000000000000000 x16: 0000000000000000 
[ 5128.492492] x15: ffffb83b70592bf0 x14: ffffffffffffffff 
[ 5128.498174] x13: ffff8000900039e7 x12: ffff8000100039ef 
[ 5128.503256] x11: 0000000000000038 x10: 0101010101010101 
[ 5128.509024] x9 : ffff800010003d00 x8 : 2a2a2a2a2a2a2a2a 
[ 5128.514536] x7 : 2a2a2a2a2a2a2a09 x6 : c0000000ffff7fff 
[ 5128.519793] x5 : ffff637c3fd1c958 x4 : ffffb83b705a7968 
[ 5128.525474] x3 : 0000000000000001 x2 : ffffb83b6ea4e170 
[ 5128.530811] x1 : ffffb83b70592680 x0 : 0000000100010001 
[ 5128.536150] Call trace:
[ 5128.538349]  tegra194_cbb_err_isr+0x19c/0x1b0
[ 5128.542892]  __handle_irq_event_percpu+0x68/0x2a0
[ 5128.547439]  handle_irq_event_percpu+0x40/0xa0
[ 5128.551643]  handle_irq_event+0x50/0xf0
[ 5128.555411]  handle_fasteoi_irq+0xc0/0x170
[ 5128.559716]  generic_handle_irq+0x40/0x60
[ 5128.563464]  __handle_domain_irq+0x70/0xd0
[ 5128.567669]  efi_header_end+0xb0/0xf0
[ 5128.571164]  el1_irq+0xd0/0x180
[ 5128.574145]  cpuidle_enter_state+0xb8/0x410
[ 5128.578601]  cpuidle_enter+0x40/0x60
[ 5128.581841]  call_cpuidle+0x44/0x80
[ 5128.585339]  do_idle+0x208/0x270
[ 5128.588768]  cpu_startup_entry+0x30/0x70
[ 5128.592511]  rest_init+0xdc/0xe8
[ 5128.595748]  arch_call_rest_init+0x18/0x20
[ 5128.600201]  start_kernel+0x514/0x54c
[ 5128.603531] Code: a9446bf9 a94573fb a8c67bfd d65f03c0 (d4210000) 
[ 5128.609749] ---[ end trace e205cb873c92c80e ]---
[ 5128.614466] Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt
[ 5128.621731] SMP: stopping secondary CPUs
��mem_range:341 >>>>>>>>>> TEGRA I2S DRIVER::after io_write8  >>>>>>
D/TC:? 0 tegra_i2s_read_i2s_mem_range:344 >>>>>>>>>> TEGRA I2S DRIVER::after io_read8   >>>>>>
D/TC:? 0 tegra_i2s_read_i2s_mem_range:371 >>>>>>>>>> TEGRA I2S DRIVER::read_mem_range END >>>>>>
F/TC:? 0 trace_syscall:151 syscall #8 (syscall_check_access_rights)
F/TC:? 0 trace_syscall:151 syscall #7 (syscall_invoke_ta_command)
D/TC:3 0 abort_handler:550 [abort] abort in User mode (TA will panic)
E/TC:? 0 
E/TC:? 0 User mode data-abort at address 0x16 (translation fault)
E/TC:? 0  esr 0x92000005  ttbr0 0x20000cb16c800   ttbr1 0x00000000   cidr 0x0
E/TC:? 0  cpu #3          cpsr 0x20000130
E/TC:? 0  x0  00000000400953c8 x1  00000000400953c9
E/TC:? 0  x2  0000000000000016 x3  00000000400953c8
E/TC:? 0  x4  000000004006f479 x5  0000000000000000
E/TC:? 0  x6  0000000000000000 x7  0000000040014dd8
E/TC:? 0  x8  0000000000000000 x9  0000000000000000
E/TC:? 0  x10 0000000000000000 x11 0000000000000000
E/TC:? 0  x12 0000000000000000 x13 0000000040014dd8
E/TC:? 0  x14 00000000400721c9 x15 0000000000000020
E/TC:? 0  x16 00000000cb059f88 x17 0000000000000000
E/TC:? 0  x18 0000000000000000 x19 0000000000000000
E/TC:? 0  x20 0000000000000000 x21 0000000000000000
E/TC:? 0  x22 0000000000000000 x23 0000000000000000
E/TC:? 0  x24 0000000000000000 x25 0000000000000000
E/TC:? 0  x26 0000000000000000 x27 0000000000000000
E/TC:? 0  x28 0000000000000000 x29 0000000000000000
E/TC:? 0  x30 0000000000000000 elr 0000000040088858
E/TC:? 0  sp_el0 0000000040014fb0
E/LD:  Status of TA 8aaaf200-2450-11e4-abe2-0002a5d5c51b
E/LD:   arch: arm
E/LD:  region  0: va 0x40004000 pa 0xcb240000 size 0x002000 flags rw-s (ldelf)
E/LD:  region  1: va 0x40006000 pa 0xcb242000 size 0x008000 flags r-xs (ldelf)
E/LD:  region  2: va 0x4000e000 pa 0xcb24a000 size 0x001000 flags rw-s (ldelf)
E/LD:  region  3: va 0x4000f000 pa 0xcb24b000 size 0x004000 flags rw-s (ldelf)
E/LD:  region  4: va 0x40013000 pa 0xcb24f000 size 0x001000 flags r--s
E/LD:  region  5: va 0x40014000 pa 0xcb27a000 size 0x001000 flags rw-s (stack)
E/LD:  region  6: va 0x4006f000 pa 0x00001000 size 0x01e000 flags r-xs [0]
E/LD:  region  7: va 0x4008d000 pa 0x0001f000 size 0x00c000 flags rw-s [0]
E/LD:   [0] 8aaaf200-2450-11e4-abe2-0002a5d5c51b @ 0x4006f000
E/LD:  Call stack:
E/LD:   0x40088858
E/LD:   0x400721c9
E/LD:   0x400721ed
E/LD:   0x4006f1ed
E/LD:   0x4006f4c3
E/LD:   0x40073c7f
E/LD:   0x4006f58f
E/LD:   0x40073941
E/LD:   0x400739cb
E/LD:   0x4006f50f
E/LD:   0x4006f5a8
D/TC:? 0 user_ta_enter:176 tee_user_ta_enter: TA panicked with code 0xdeadbeef
D/TC:? 0 destroy_ta_ctx_from_session:322 Remove references to context (0xcb1573a8)
D/TC:? 0 destroy_context:307 Destroy TA ctx (0xcb157390)
D/TC:? 0 tee_ta_close_session:510 csess 0xcb155c50 id 1
D/TC:? 0 tee_ta_close_session:529 Destroy session
D/TC:? 0 tee_ta_close_session:510 csess 0xcb155a60 id 2
D/TC:? 0 tee_ta_close_session:529 Destroy session
��[ 5129.632278] Kernel Offset: 0x383b5e890000 from 0xffff800010000000
[ 5129.633322] PHYS_OFFSET: 0xffff9c8c40000000
[ 5129.634023] CPU features: 0x8240002,03802a30
[ 5129.634728] Memory Limit: none
[ 5129.635490] ---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt ]---

The Jetson AGX Xavier "freezes" after that error and reboots. I would appreciate any tips on why this happens, and what should be corrected in my code.

I use kernel-5.10 on the Jetson AGX Xavier and the corresponding OP-TEE OS source can be viewed here: https://gitlab.com/Yuhala/jetson-xavier/-/tree/master/nvidia-jetson-optee-source/optee

jenswi-linaro commented 1 year ago

Some of the reads are perhaps causing external aborts. I wasn't expecting the kernel to panic from that though.

github-actions[bot] commented 1 year ago

This issue has been marked as a stale issue because it has been open (more than) 30 days with no activity. Remove the stale label or add a comment, otherwise this issue will automatically be closed in 5 days. Note, that you can always re-open a closed issue at any time.