search

polarfire-soc / platform

Other
6 stars 11 forks source link

DDR Training Fail on ICICLE Kit #13

Closed fcuzzocrea closed 2 years ago

fcuzzocrea commented 2 years ago

Hi!

Sorry to bother you again, but it seems that I cannot make another one of my ICICLE Kit pass the DDR training. I am using main branch of the BsP.

To enable LPDDR4 support, I just added the #define DDR_SUPPORT to my platform_config. My full file is here: https://pastebin.com/EN90euEF. I did not add any of the other #define seen in the HSS or the examples as they were for configurator < 2021.1 while I am using the 2021.3 ATM.

My configuration file from the MSS Configuration is here: https://pastebin.com/L7n8u5bg

The logs of the training instead are here: https://pastebin.com/wiVAapBX

Do you have any clue about what is happening? Am I configuring wrong the platform from the configurator?

Thanks!

fcuzzocrea commented 2 years ago

Actually sorry, was an issue on my side, closing this now.

fcuzzocrea commented 2 years ago

Actually reopening because I am noticing another strange behaviour, sometimes, the DDR training just stucks at "Loading test pattern", here the log from when I switch on my ICICLE Kit to when it sucks. Am I configuring something wrong?

 Start training. TIP_CFG_PARAMS:07CFE02A
 dpc_bits: 0004C422
 PCODE = 00000016
 NCODE = 0000001F
 addr_cmd_value: 00000000
 bclk_sclk_offset_value: 00000000
 dpc_vrgen_v_value: 00000000
 dpc_vrgen_h_value: 00000000
 dpc_vrgen_vs_value: 00000000
 tip_cfg_params: 07CFE02A
 ca_indly 00000000 vref 00000005 a5_dly_max:00000000 a5_dly_min:00000080 a5_dly_min_last:00000080 range_a5:FFFFFF80 deltat:00000080 in_window:00000000 vref_answer:00000080
 ca_indly 00000000 vref 00000006 a5_dly_max:00000000 a5_dly_min:00000080 a5_dly_min_last:00000080 range_a5:FFFFFF80 deltat:00000080 in_window:00000000 vref_answer:00000080
 ca_indly 00000000 vref 00000007 a5_dly_max:00000000 a5_dly_min:00000080 a5_dly_min_last:00000080 range_a5:FFFFFF80 deltat:00000080 in_window:00000000 vref_answer:00000080
 ca_indly 00000000 vref 00000008 a5_dly_max:00000070 a5_dly_min:0000006D a5_dly_min_last:00000080 range_a5:00000003 deltat:00000013 in_window:00000000 vref_answer:00000080
 ca_indly 00000000 vref 00000009 a5_dly_max:00000002 a5_dly_min:00000001 a5_dly_min_last:0000006D range_a5:00000001 deltat:0000006C in_window:00000000 vref_answer:00000080
 ca_indly 00000000 vref 0000000A a5_dly_max:00000004 a5_dly_min:00000004 a5_dly_min_last:00000001 range_a5:00000000 deltat:00000003 in_window:00000001 vref_answer:00000080
 ca_indly 00000000 vref 0000000B a5_dly_max:00000006 a5_dly_min:00000005 a5_dly_min_last:00000004 range_a5:00000001 deltat:00000001 in_window:00000003 vref_answer:00000080
 ca_indly 00000005 vref 00000005 a5_dly_max:00000000 a5_dly_min:00000080 a5_dly_min_last:00000004 range_a5:FFFFFF80 deltat:00000080 in_window:00000000 vref_answer:00000080
 ca_indly 00000005 vref 00000006 a5_dly_max:00000000 a5_dly_min:00000080 a5_dly_min_last:00000080 range_a5:FFFFFF80 deltat:00000080 in_window:00000000 vref_answer:00000080
 ca_indly 00000005 vref 00000007 a5_dly_max:00000000 a5_dly_min:00000080 a5_dly_min_last:00000080 range_a5:FFFFFF80 deltat:00000080 in_window:00000000 vref_answer:00000080
 ca_indly 00000005 vref 00000008 a5_dly_max:00000000 a5_dly_min:00000080 a5_dly_min_last:00000080 range_a5:FFFFFF80 deltat:00000080 in_window:00000000 vref_answer:00000080
 ca_indly 00000005 vref 00000009 a5_dly_max:00000036 a5_dly_min:00000035 a5_dly_min_last:00000080 range_a5:00000001 deltat:0000004B in_window:00000000 vref_answer:00000080
 ca_indly 00000005 vref 0000000A a5_dly_max:00000038 a5_dly_min:00000037 a5_dly_min_last:00000035 range_a5:00000001 deltat:00000002 in_window:00000001 vref_answer:00000080
 ca_indly 00000005 vref 0000000B a5_dly_max:00000001 a5_dly_min:00000001 a5_dly_min_last:00000037 range_a5:00000000 deltat:00000036 in_window:00000001 vref_answer:00000080
 ca_indly 00000005 vref 0000000C a5_dly_max:00000003 a5_dly_min:00000002 a5_dly_min_last:00000001 range_a5:00000001 deltat:00000001 in_window:00000003 vref_answer:00000080
 ca_indly 0000000A vref 00000005 a5_dly_max:00000000 a5_dly_min:00000080 a5_dly_min_last:00000001 range_a5:FFFFFF80 deltat:00000080 in_window:00000000 vref_answer:00000080
 ca_indly 0000000A vref 00000006 a5_dly_max:00000000 a5_dly_min:00000080 a5_dly_min_last:00000080 range_a5:FFFFFF80 deltat:00000080 in_window:00000000 vref_answer:00000080
 ca_indly 0000000A vref 00000007 a5_dly_max:00000000 a5_dly_min:00000080 a5_dly_min_last:00000080 range_a5:FFFFFF80 deltat:00000080 in_window:00000000 vref_answer:00000080
 ca_indly 0000000A vref 00000008 a5_dly_max:00000000 a5_dly_min:00000080 a5_dly_min_last:00000080 range_a5:FFFFFF80 deltat:00000080 in_window:00000000 vref_answer:00000080
 ca_indly 0000000A vref 00000009 a5_dly_max:00000031 a5_dly_min:00000030 a5_dly_min_last:00000080 range_a5:00000001 deltat:00000050 in_window:00000000 vref_answer:00000080
 ca_indly 0000000A vref 0000000A a5_dly_max:00000033 a5_dly_min:00000032 a5_dly_min_last:00000030 range_a5:00000001 deltat:00000002 in_window:00000001 vref_answer:00000080
 ca_indly 0000000A vref 0000000B a5_dly_max:00000034 a5_dly_min:00000033 a5_dly_min_last:00000032 range_a5:00000001 deltat:00000001 in_window:00000003 vref_answer:00000080
 vref_answer found0000000B
   rx_a5   00000003   rx_ck   00000005   rx_ck_last  00000005   transition_a5   00000036   transition_ck   0000001F   Iteration:  00000036   REFCLK_PHASE:   00000000
   rx_a5   00000003   rx_ck   0000000A   rx_ck_last  00000005   transition_a5   00000035   transition_ck   00000018   Iteration:  00000035   REFCLK_PHASE:   00000001
   rx_a5   00000003   rx_ck   0000000A   rx_ck_last  00000005   transition_a5   00000035   transition_ck   00000012   Iteration:  00000035   REFCLK_PHASE:   00000002
   rx_a5   00000003   rx_ck   0000000A   rx_ck_last  00000005   transition_a5   00000036   transition_ck   0000000A   Iteration:  00000036   REFCLK_PHASE:   00000003
   rx_a5   00000003   rx_ck   0000000A   rx_ck_last  00000005   transition_a5   00000035   transition_ck   00000003   Iteration:  00000035   REFCLK_PHASE:   00000004
   rx_a5   00000003   rx_ck   00000005   rx_ck_last  00000005   transition_a5   00000036   transition_ck   00000035   Iteration:  00000036   REFCLK_PHASE:   00000005
   rx_a5   00000003   rx_ck   00000005   rx_ck_last  00000005   transition_a5   00000036   transition_ck   0000002E   Iteration:  00000036   REFCLK_PHASE:   00000006
   rx_a5   00000003   rx_ck   00000005   rx_ck_last  00000005   transition_a5   00000035   transition_ck   00000027   Iteration:  00000035   REFCLK_PHASE:   00000007
   rx_a5   00000003   rx_ck   00000005   rx_ck_last  00000005   transition_a5   00000035   transition_ck   0000001F   Iteration:  00000035   REFCLK_PHASE:   00000008
   rx_a5   00000003   rx_ck   0000000A   rx_ck_last  00000005   transition_a5   00000035   transition_ck   00000018   Iteration:  00000035   REFCLK_PHASE:   00000009
   rx_a5   00000003   rx_ck   0000000A   rx_ck_last  00000005   transition_a5   00000035   transition_ck   00000012   Iteration:  00000035   REFCLK_PHASE:   0000000A
   rx_a5   00000003   rx_ck   0000000A   rx_ck_last  00000005   transition_a5   00000035   transition_ck   0000000A   Iteration:  00000035   REFCLK_PHASE:   0000000B
   rx_a5   00000003   rx_ck   0000000A   rx_ck_last  00000005   transition_a5   00000035   transition_ck   00000003   Iteration:  00000035   REFCLK_PHASE:   0000000C
   rx_a5   00000003   rx_ck   00000005   rx_ck_last  00000005   transition_a5   00000036   transition_ck   00000034   Iteration:  00000036   REFCLK_PHASE:   0000000D
   rx_a5   00000003   rx_ck   00000005   rx_ck_last  00000005   transition_a5   00000036   transition_ck   0000002E   Iteration:  00000036   REFCLK_PHASE:   0000000E
   rx_a5   00000003   rx_ck   00000005   rx_ck_last  00000005   transition_a5   00000035   transition_ck   00000027   Iteration:  00000035   REFCLK_PHASE:   0000000F
   difference  00000017   REFCLK_PHASE    00000000
   difference  0000001E   REFCLK_PHASE    00000001
   difference  00000024   REFCLK_PHASE    00000002
   difference  0000002C   REFCLK_PHASE    00000003
   difference  00000033   REFCLK_PHASE    00000004
   difference  00000001   REFCLK_PHASE    00000005
   difference  00000008   REFCLK_PHASE    00000006
   difference  0000000F   REFCLK_PHASE    00000007
  MANUAL ADDCMD TRAINING Results:
          PLL OFFSET:  00000005
          transition_a5_max:  00000036
          CA Output Delay:  00000001
 Returning FPGA CA VREF & CA drive to user setting.
 00000000

 ADDCMD_OFFSET 00000001

 pll_phadj_after_hw_training 01060104

 pll_phadj_after_hw_training 02000106

 DDR FINAL_MODE: 00014B24

 training status = 0000001D
 PCODE = 00000016
 NCODE = 0000001F
 WRCALIB_RESULT: 00000000
 sro_ref_slewr  = 0000000A
 sro_ref_slewf  = 00000010
 sro_slewr  = 0000000A
 sro_slewf  = 00000010

 lane_select     gt_err_comb     gt_txdly    gt_steps_180    gt_state    wl_delay_0      dqdqs_err_done      dqdqs_state     delta0      delta1
 00000000    00000000    070D070D    00000009    0000000B    0000001C    00000008        00000008    02030001    02040202
 00000001    00000000    070C060C    00000009    0000000B    0000001D    00000008        00000008    00010001    00010202
 00000002    00000000    070D060D    00000009    0000000B    00000023    00000008        00000008    00040404    00030303
 00000003    00000000    070D060C    00000009    0000000B    00000020    00000008        00000008    01010000    02030302

 lane_select     rdqdqs_status2  addcmd_status0  addcmd_status1  addcmd_answer1  dqdqs_status1

 00000000    00000016    00000000    00000000    00000000    00000008
 00000001    00000013    00000000    00000000    00000000    0000000A
 00000002    0000000F    00000000    00000000    00000000    00000008
 00000003    00000014    00000000    00000000    00000000    00000007
 ****************************************************00000000
Calibration offset used:00000000
Lane failed:00000000 All lanes status:00000000
Lane failed:00000001 All lanes status:00000000
Lane failed:00000002 All lanes status:00000000
Lane failed:00000003 All lanes status:00000000
Calibration offset used:00000001
Lane failed:00000000 All lanes status:00000000
Lane failed:00000001 All lanes status:00000000
Lane failed:00000002 All lanes status:00000000
Lane failed:00000003 All lanes status:00000000
Calibration offset used:00000002
Lane failed:00000000 All lanes status:00000000
Lane failed:00000001 All lanes status:00000000
Lane failed:00000002 All lanes status:00000000
Lane failed:00000003 All lanes status:00000000
Calibration offset used:00000003
Lane failed:00000000 All lanes status:00000000
Lane failed:00000001 All lanes status:00000000
Lane failed:00000002 All lanes status:00000000
Lane failed:00000003 All lanes status:00000000
Calibration offset used:00000004
Lane failed:00000000 All lanes status:00000000
Lane failed:00000001 All lanes status:00000000
Lane failed:00000002 All lanes status:00000000
Lane failed:00000003 All lanes status:00000000
Calibration offset used:00000005
Lane passed:00000000 All lanes status:00000001
Lane passed:00000001 All lanes status:00000003
Lane passed:00000002 All lanes status:00000007
Lane passed:00000003 All lanes status:0000000F

 wr calib result 00005555

 DDR SANITY_CHECKS: 00000000

 Passed MTC full check 00000000
     Pattern: 0x00000000
     Pattern: 0x00000001
     Pattern: 0x00000002
     Pattern: 0x00000003
     Pattern: 0x00000004
     Pattern: 0x00000005
Loading test pattern

So it seems to be stuck somewhere after this: https://github.com/polarfire-soc/platform/blob/13c8cc74e445008bf8615097d44ea28af4ff8d97/mpfs_hal/common/nwc/mss_ddr_debug.c#L759

This is random, e.g. if i turn on and off the board several times, then it works and passes that part.

hughbreslin commented 2 years ago

Hey @fcuzzocrea :) We're going to be mirroring an update to the DDR training soon with improvements which may resolve this issue, your ESE should be able to provide you with a pre-release copy to try out and see if it resolves your issue

fcuzzocrea commented 2 years ago

Thanks. But he provided just and hex to flash with libero. It is possible to have access to the sources too so I can try to just update that part of the bsp?

Also, would be possible for you to use this repo in the HSS as a submodule instead of copying it into hss repo? In this way could be easier for us users to track down which version was used where.

Thank you!

hughbreslin commented 2 years ago

You should be able to get the sources...

There is a version file included in the HAL which could make life easier, including a sub module / sub tree is something we could discuss :)

fcuzzocrea commented 2 years ago

Yes, thanks!

fcuzzocrea commented 2 years ago

So, some updates!

I was able to put my hands on the sources of the HSS, so I pulled from there the updated mpfs_hal, and plugged it into the bsp (the platform repo). After doing that I updated the mss_sw_config.h whit what I found on the HSS board folder, and now DDR works correctly when using the bsp on my custom application every time I plug on and off the board.

However, for this to work, I had to disable 64bit and 32bit access test (I had to disable it also before):

#define TEST_64BIT_ACCESS 0
#define TEST_32BIT_ACCESS 0

otherwise the board stucks. Does this happens on your side too?

For what concerns the submodule organization, I was thinking that Including the bsp as sumbodule into the hss / examples could help making sure that the bsp always comes from the same source and no extra modification is added on top by the particular application which consumes it. This might help end users to track down issues and versions in a way which is easier than comparing the hal version from the header :)

Thanks for your help!

hughbreslin commented 2 years ago

However, for this to work, I had to disable 64bit and 32bit access test (I had to disable it also before)

Hmm no you shouldn't have to do this. If you connect your debugger can you see where its getting stuck?

On the submodule set up, definitely this is something we can discuss :)

fcuzzocrea commented 2 years ago

Alright thanks, will try to do some tests on the weekend! Thanks for you help :)

hughbreslin commented 2 years ago

Hey @fcuzzocrea, how did you get on with your testing? Are you happy enough to mark this as closed if this is resolved for you? :)

fcuzzocrea commented 2 years ago

Sorry, wasn't able to perform the test, will try to test asap

fcuzzocrea commented 2 years ago

I just tried now to connect with the debugger and, when I enable 32 and 64 bit access test on my icicle kit the boards ends up in trap_from_machine_mode. This happens when using the new mpfs_hal imported into the platform repo on a baremetal project (not HSS).

hughbreslin commented 2 years ago

Hey, thats odd. You should be able to see mcasue (to tell you what the exception was) and mepc (tells you the program counter at the time the exception occurred) from the trap function, knowing where it trapped would be useful, you can use mepc to check from the map / lst file.

Also you mentioned you're using XML from 2021.3, it might be worth while using XML generated using the latest configurator as there were changes to some of the DDR training values between the releases.

nitindeshpande commented 2 years ago

There are further updates to the DDR training with 2022.08 release. Please reopen the issue if there is something more to discuss.