search

lowRISC / opentitan

OpenTitan: Open source silicon root of trust
https://www.opentitan.org
Apache License 2.0
2.51k stars 746 forks source link

Build & Run are failing on CI(private) #12575

Closed Jacob-Levy closed 2 years ago

Jacob-Levy commented 2 years ago

This is from the build.log:

Error-[SVA-COBMMI] Can only bind to modules or instances.
../src/lowrisc_dv_aes_sva_0.1/[aes_bind.sv](https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Faes_bind.sv%2F&data=05%7C01%7Cjacob.levy%40nuvoton.com%7C1a7f7c99749d4110d53a08da32a47637%7Ca3f24931d4034b4a94f17d83ac638e07%7C0%7C0%7C637877979210638123%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tWo14PXqQEEZ594rP2esdOgjZx2zNCeI3ofWNu3Dc%2BU%3D&reserved=0), 54
aes_bind
  Can only bind to modules or module instances.
  bind aes_prng_masking aes_masking_reseed_if u_aes_masking_reseed_if( .clk_i 
  (clk_i),  .rst_ni (rst_ni),  .entropy_i (entropy_i),  .prng_seed_en 
  (prng_seed_en),  .prng_seed (prng_seed),  .lfsr_q_0 
  (gen_lfsrs[0].u_lfsr_chunk.lfsr_q),  .lfsr_q_1 
  (gen_lfsrs[1].u_lfsr_chunk.lfsr_q),  .lfsr_q_2 
  (gen_lfsrs[2].u_lfsr_chunk.lfsr_q),  .lfsr_q_3 
  (gen_lfsrs[3].u_lfsr_chunk.lfsr_q),  .lfsr_q_4 
  (gen_lfsrs[4].u_lfsr_chunk.lfsr_q));

@tjaychen @msfschaffner @rswarbrick

msfschaffner commented 2 years ago

@Jacob-Levy, can you point us to the PR where this is currently failing?

Jacob-Levy commented 2 years ago

@msfschaffner It is not a PR yet... I run the CI(private) using dvsim on a PR candidate. I sent you an email with the full build.log.

msfschaffner commented 2 years ago

Ah sorry about that - I just saw the email.

@weicaiyang @rasmus-madsen Is this error maybe due to the VCS version? @Jacob-Levy is using R-2020.12-1 if I am not mistaken...

weicaiyang commented 2 years ago

We just switched to S-2021.09-SP2-1. Greg sent an announcement last night. https://github.com/lowRISC/opentitan-private-ci/commit/181fa4149f13c96739909182b943b4792b1ed08d

perhaps push it to a PR so that we can take a look.

Jacob-Levy commented 2 years ago

Will update VCS version and rerun CI(private)

Jacob-Levy commented 2 years ago

II just updated VCS&VERDI versions to S-2021.09-SP2 while waiting to our CAD to pull S-2021.09-SP2-1. I also pulled the opentitan latest master and run the following from opentitan (no AST PR files!):

% util/dvsim/dvsim.py hw/top_earlgrey/dv/top_earlgrey_sim_cfgs.hjson --purge -w -i smoke --max-parallel 3 --tool=vcs --fixed-seed=1

The result:

INFO: [FlowCfg] [scratch_path]: [chip] [/tanap1/proj_cd14/opentitan/jlevy/ot_PR/opentitan/scratch/master/chip_earlgrey_asic-sim-vcs]
## TOP_EARLGREY_BATCH_SIM Simulation Results (Summary)
### Wednesday May 11 2022 11:39:37 UTC
### GitHub Revision: [`6195eefb3`](https://github.com/lowrisc/opentitan/tree/6195eefb387fb0876c57c9db30333fb2fd9c1704)
### Branch: master

|       Name       |  Passing  |  Total  |  Pass Rate  |
|:----------------:|:---------:|:-------:|:-----------:|
|     TL_AGENT     |     1     |    1    |  100.00 %   |
|     ADC_CTRL     |     3     |    3    |  100.00 %   |
|       AES        |     0     |    3    |   0.00 %    |  <<<<<<<<
|       AES        |     3     |    3    |  100.00 %   |
|    AON_TIMER     |     3     |    3    |  100.00 %   |
|      CLKMGR      |     3     |    3    |  100.00 %   |
|      CSRNG       |     3     |    3    |  100.00 %   |
|       EDN        |     3     |    3    |  100.00 %   |
|   ENTROPY_SRC    |     3     |    3    |  100.00 %   |
|    FLASH_CTRL    |     3     |    3    |  100.00 %   |
|       GPIO       |     3     |    3    |  100.00 %   |
|       HMAC       |     4     |    4    |  100.00 %   |
|       I2C        |     2     |    3    |   66.67 %   |  <<<<<<<<
|      KEYMGR      |     3     |    3    |  100.00 %   |
|       KMAC       |     3     |    3    |  100.00 %   |
|       KMAC       |     3     |    3    |  100.00 %   |
|     LC_CTRL      |     4     |    4    |  100.00 %   |
|       OTBN       |     3     |    3    |  100.00 %   |
|     OTP_CTRL     |     3     |    3    |  100.00 %   |
|     PATTGEN      |     3     |    3    |  100.00 %   |
|    PRIM_ALERT    |     1     |    1    |  100.00 %   |
|     PRIM_ESC     |     1     |    1    |  100.00 %   |
|    PRIM_LFSR     |     1     |    1    |  100.00 %   |
|   PRIM_PRESENT   |     1     |    1    |  100.00 %   |
|   PRIM_PRINCE    |     1     |    1    |  100.00 %   |
|       PWM        |     3     |    3    |  100.00 %   |
|      PWRMGR      |     3     |    3    |  100.00 %   |
|     ROM_CTRL     |     3     |    3    |  100.00 %   |
| RSTMGR_CNSTY_CHK |     1     |    1    |  100.00 %   |
|      RSTMGR      |     3     |    3    |  100.00 %   |
|      RV_DM       |     7     |    7    |  100.00 %   |
|     RV_TIMER     |     3     |    3    |  100.00 %   |
|     SPI_HOST     |     3     |    3    |  100.00 %   |
|    SPI_DEVICE    |     3     |    3    |  100.00 %   |
|    SRAM_CTRL     |     3     |    3    |  100.00 %   |
|    SRAM_CTRL     |     3     |    3    |  100.00 %   |
|   SYSRST_CTRL    |     3     |    3    |  100.00 %   |
|       UART       |     3     |    3    |  100.00 %   |
|      USBDEV      |     2     |    2    |  100.00 %   |
|  ALERT_HANDLER   |     3     |    3    |  100.00 %   |
|       MAIN       |     1     |    1    |  100.00 %   |
|       PERI       |     1     |    1    |  100.00 %   |
|       CHIP       |     6     |    6    |  100.00 %   |

INFO: [FlowCfg] [results:email]: [/tanap1/proj_cd14/opentitan/jlevy/ot_PR/opentitan/scratch/email.html] 
ERROR: [dvsim] Errors were encountered in this run.

          [   legend    ]: [Q: queued, D: dispatched, P: passed, F: failed, K: killed, T: total]                                                                                        
00:18:51  [    build    ]: [Q: 000, D: 000, P: 043, F: 001, K: 000, T: 044] 100%                                                                                                        
00:46:00  [     run     ]: [Q: 000, D: 000, P: 115, F: 001, K: 003, T: 119] 100%
rswarbrick commented 2 years ago

Hi Jacob,

Looks like the build failed for that block. Probably worth having a look at the build log to see what happened.

Rupert

On Wed, 11 May 2022 at 07:34, Jacob @.***> wrote:

II just updated VCS&VERDI versions to S-2021.09-SP2 while waiting to our CAD to pull S-2021.09-SP2-1. I also pulled the opentitan latest master and run the following from opentitan (no AST PR files!):

% util/dvsim/dvsim.py hw/top_earlgrey/dv/top_earlgrey_sim_cfgs.hjson --purge -w -i smoke --max-parallel 3 --tool=vcs --fixed-seed=1

The result:

INFO: [FlowCfg] [scratch_path]: [chip]
[/tanap1/proj_cd14/opentitan/jlevy/ot_PR/opentitan/scratch/master/chip_earlgrey_asic-sim-vcs]
## TOP_EARLGREY_BATCH_SIM Simulation Results (Summary)
### Wednesday May 11 2022 11:39:37 UTC
### GitHub Revision:
[`6195eefb3`](https://github.com/lowrisc/opentitan/tree/6195eefb387fb0876c57c9db30333fb2fd9c1704)
### Branch: master

|       Name       |  Passing  |  Total  |  Pass Rate  |
|:----------------:|:---------:|:-------:|:-----------:|
|     TL_AGENT     |     1     |    1    |  100.00 %   |
|     ADC_CTRL     |     3     |    3    |  100.00 %   |
|       AES        |     0     |    3    |   0.00 %    |
|       AES        |     3     |    3    |  100.00 %   |
|    AON_TIMER     |     3     |    3    |  100.00 %   |
|      CLKMGR      |     3     |    3    |  100.00 %   |
|      CSRNG       |     3     |    3    |  100.00 %   |
|       EDN        |     3     |    3    |  100.00 %   |
|   ENTROPY_SRC    |     3     |    3    |  100.00 %   |
|    FLASH_CTRL    |     3     |    3    |  100.00 %   |
|       GPIO       |     3     |    3    |  100.00 %   |
|       HMAC       |     4     |    4    |  100.00 %   |
|       I2C        |     2     |    3    |   66.67 %   |
|      KEYMGR      |     3     |    3    |  100.00 %   |
|       KMAC       |     3     |    3    |  100.00 %   |
|       KMAC       |     3     |    3    |  100.00 %   |
|     LC_CTRL      |     4     |    4    |  100.00 %   |
|       OTBN       |     3     |    3    |  100.00 %   |
|     OTP_CTRL     |     3     |    3    |  100.00 %   |
|     PATTGEN      |     3     |    3    |  100.00 %   |
|    PRIM_ALERT    |     1     |    1    |  100.00 %   |
|     PRIM_ESC     |     1     |    1    |  100.00 %   |
|    PRIM_LFSR     |     1     |    1    |  100.00 %   |
|   PRIM_PRESENT   |     1     |    1    |  100.00 %   |
|   PRIM_PRINCE    |     1     |    1    |  100.00 %   |
|       PWM        |     3     |    3    |  100.00 %   |
|      PWRMGR      |     3     |    3    |  100.00 %   |
|     ROM_CTRL     |     3     |    3    |  100.00 %   |
| RSTMGR_CNSTY_CHK |     1     |    1    |  100.00 %   |
|      RSTMGR      |     3     |    3    |  100.00 %   |
|      RV_DM       |     7     |    7    |  100.00 %   |
|     RV_TIMER     |     3     |    3    |  100.00 %   |
|     SPI_HOST     |     3     |    3    |  100.00 %   |
|    SPI_DEVICE    |     3     |    3    |  100.00 %   |
|    SRAM_CTRL     |     3     |    3    |  100.00 %   |
|    SRAM_CTRL     |     3     |    3    |  100.00 %   |
|   SYSRST_CTRL    |     3     |    3    |  100.00 %   |
|       UART       |     3     |    3    |  100.00 %   |
|      USBDEV      |     2     |    2    |  100.00 %   |
|  ALERT_HANDLER   |     3     |    3    |  100.00 %   |
|       MAIN       |     1     |    1    |  100.00 %   |
|       PERI       |     1     |    1    |  100.00 %   |
|       CHIP       |     6     |    6    |  100.00 %   |

INFO: [FlowCfg] [results:email]: [/tanap1/proj_cd14/opentitan/jlevy/ot_PR/opentitan/scratch/email.html] 
ERROR: [dvsim] Errors were encountered in this run.

          [ legend ]: [Q: queued, D: dispatched, P: passed, F: failed,
K: killed, T: total]
00:18:51 [ build ]: [Q: 000, D: 000, P: 043, F: 001, K: 000, T: 044]
100%
00:46:00  [     run     ]: [Q: 000, D: 000, P: 115, F: 001, K: 003, T: 119] 100%
Jacob-Levy commented 2 years ago

@rswarbrick Alredy on the top of the issue... but attached the build.log build.log

Jacob-Levy commented 2 years ago

Also the run.log for the I2C errors attached run.log .

tjaychen commented 2 years ago

II just updated VCS&VERDI versions to S-2021.09-SP2 while waiting to our CAD to pull S-2021.09-SP2-1. I also pulled the opentitan latest master and run the following from opentitan (no AST PR files!):

% util/dvsim/dvsim.py hw/top_earlgrey/dv/top_earlgrey_sim_cfgs.hjson --purge -w -i smoke --max-parallel 3 --tool=vcs --fixed-seed=1

The result:

INFO: [FlowCfg] [scratch_path]: [chip] [/tanap1/proj_cd14/opentitan/jlevy/ot_PR/opentitan/scratch/master/chip_earlgrey_asic-sim-vcs]
## TOP_EARLGREY_BATCH_SIM Simulation Results (Summary)
### Wednesday May 11 2022 11:39:37 UTC
### GitHub Revision: [`6195eefb3`](https://github.com/lowrisc/opentitan/tree/6195eefb387fb0876c57c9db30333fb2fd9c1704)
### Branch: master

|       Name       |  Passing  |  Total  |  Pass Rate  |
|:----------------:|:---------:|:-------:|:-----------:|
|     TL_AGENT     |     1     |    1    |  100.00 %   |
|     ADC_CTRL     |     3     |    3    |  100.00 %   |
|       AES        |     0     |    3    |   0.00 %    |  <<<<<<<<
|       AES        |     3     |    3    |  100.00 %   |
|    AON_TIMER     |     3     |    3    |  100.00 %   |
|      CLKMGR      |     3     |    3    |  100.00 %   |
|      CSRNG       |     3     |    3    |  100.00 %   |
|       EDN        |     3     |    3    |  100.00 %   |
|   ENTROPY_SRC    |     3     |    3    |  100.00 %   |
|    FLASH_CTRL    |     3     |    3    |  100.00 %   |
|       GPIO       |     3     |    3    |  100.00 %   |
|       HMAC       |     4     |    4    |  100.00 %   |
|       I2C        |     2     |    3    |   66.67 %   |  <<<<<<<<
|      KEYMGR      |     3     |    3    |  100.00 %   |
|       KMAC       |     3     |    3    |  100.00 %   |
|       KMAC       |     3     |    3    |  100.00 %   |
|     LC_CTRL      |     4     |    4    |  100.00 %   |
|       OTBN       |     3     |    3    |  100.00 %   |
|     OTP_CTRL     |     3     |    3    |  100.00 %   |
|     PATTGEN      |     3     |    3    |  100.00 %   |
|    PRIM_ALERT    |     1     |    1    |  100.00 %   |
|     PRIM_ESC     |     1     |    1    |  100.00 %   |
|    PRIM_LFSR     |     1     |    1    |  100.00 %   |
|   PRIM_PRESENT   |     1     |    1    |  100.00 %   |
|   PRIM_PRINCE    |     1     |    1    |  100.00 %   |
|       PWM        |     3     |    3    |  100.00 %   |
|      PWRMGR      |     3     |    3    |  100.00 %   |
|     ROM_CTRL     |     3     |    3    |  100.00 %   |
| RSTMGR_CNSTY_CHK |     1     |    1    |  100.00 %   |
|      RSTMGR      |     3     |    3    |  100.00 %   |
|      RV_DM       |     7     |    7    |  100.00 %   |
|     RV_TIMER     |     3     |    3    |  100.00 %   |
|     SPI_HOST     |     3     |    3    |  100.00 %   |
|    SPI_DEVICE    |     3     |    3    |  100.00 %   |
|    SRAM_CTRL     |     3     |    3    |  100.00 %   |
|    SRAM_CTRL     |     3     |    3    |  100.00 %   |
|   SYSRST_CTRL    |     3     |    3    |  100.00 %   |
|       UART       |     3     |    3    |  100.00 %   |
|      USBDEV      |     2     |    2    |  100.00 %   |
|  ALERT_HANDLER   |     3     |    3    |  100.00 %   |
|       MAIN       |     1     |    1    |  100.00 %   |
|       PERI       |     1     |    1    |  100.00 %   |
|       CHIP       |     6     |    6    |  100.00 %   |

INFO: [FlowCfg] [results:email]: [/tanap1/proj_cd14/opentitan/jlevy/ot_PR/opentitan/scratch/email.html] 
ERROR: [dvsim] Errors were encountered in this run.

          [   legend    ]: [Q: queued, D: dispatched, P: passed, F: failed, K: killed, T: total]                                                                                        
00:18:51  [    build    ]: [Q: 000, D: 000, P: 043, F: 001, K: 000, T: 044] 100%                                                                                                        
00:46:00  [     run     ]: [Q: 000, D: 000, P: 115, F: 001, K: 003, T: 119] 100%

i'll have a loook and push a fix. just reproduced the aes error in my environment. Not really sure why this wasn't seen before..or breaking CI.

Jacob-Levy commented 2 years ago

@tjaychen Can you please add someone that can address the I2C failure (UVM_ERROR: uvm_test_top.env.scoreboard) I need to have a clean CI(private) run before I release the AST files...

weicaiyang commented 2 years ago

Not really sure why this wasn't seen before..or breaking CI.

CI is running with the signed-off tool now. Hence, i2c and aes are running with Xcelium.

tjaychen commented 2 years ago

@Jacob-Levy after talking to folks here, we think you should just push the PR first. The thing right now is for those two blocks (aes and i2c), the signoff tool is xcelium, and our guess is that xcelium doesn't flag these.

So we'll address them for vcs in the background, but you should be able to push a PR without issues (we hope).

Jacob-Levy commented 2 years ago

@tjaychen I can add a draft and you can add it to your draft PR #12570 and push them together as one PR. Is it OK with you?

Jacob-Levy commented 2 years ago

@tjaychen AES issue gone we now left with I2C run fail...

ERROR: [Scheduler] [00:24:14]: [run]: [status] [i2c:0.i2c_smoke.1: F]
ERROR: [dvsim] Errors were encountered in this run.

          [   legend    ]: [Q: queued, D: dispatched, P: passed, F: failed, K: killed, T: total]           
00:19:08  [    build    ]: [Q: 000, D: 000, P: 044, F: 000, K: 000, T: 044] 100%                      
00:45:56  [     run     ]: [Q: 000, D: 000, P: 118, F: 001, K: 000, T: 119] 100%
INFO: [FlowCfg] [results]: [i2c]:
## I2C Simulation Results
### Thursday May 12 2022 20:24:37 UTC
### GitHub Revision: [`1fbd3356e`](https://github.com/lowrisc/opentitan/tree/1fbd3356e57c5053ecdd0221d87c163741d3901f)
### Branch: master
### [Testplan](https://docs.opentitan.org/hw/ip/i2c/doc/dv/#testplan)
### Simulator: VCS

### Test Results
|  Milestone  |                   Name                    | Tests            |  Passing  |  Total  |  Pass Rate  |
|:-----------:|:-----------------------------------------:|:-----------------|:---------:|:-------:|:-----------:|
|     V1      |                host_smoke                 | i2c_smoke        |     0     |    1    |   0.00 %    |
|     V1      |               target_smoke                | i2c_smoke        |     0     |    1    |   0.00 %    |
|     V1      |               csr_hw_reset                | i2c_csr_hw_reset |     1     |    1    |  100.00 %   |
|     V1      |                  csr_rw                   | i2c_csr_rw       |     1     |    1    |  100.00 %   |
|     V1      | regwen_csr_and_corresponding_lockable_csr | i2c_csr_rw       |     1     |    1    |  100.00 %   |
|     V1      |                                           | **TOTAL**        |     2     |    3    |   66.67 %   |
|     V2      |          tl_d_outstanding_access          | i2c_csr_hw_reset |     1     |    1    |  100.00 %   |
|             |                                           | i2c_csr_rw       |     1     |    1    |  100.00 %   |
|     V2      |            tl_d_partial_access            | i2c_csr_hw_reset |     1     |    1    |  100.00 %   |
|             |                                           | i2c_csr_rw       |     1     |    1    |  100.00 %   |
|             |                                           | **TOTAL**        |     2     |    3    |   66.67 %   |

## Failure Buckets

* `UVM_ERROR (cip_base_scoreboard.sv:586) [scoreboard] tl_a_chan_fifos[i] item uncompared:` has 1 failures:
    * Test i2c_smoke has 1 failures.
        * 0.i2c_smoke.1\
          Line 49, in log /tanap1/proj_cd14/opentitan/jlevy/ot_PR/opentitan/scratch/master/i2c-sim-vcs/0.i2c_smoke/out/run.log

                UVM_ERROR @ 4108434364 ps: (cip_base_scoreboard.sv:586) [uvm_test_top.env.scoreboard] tl_a_chan_fifos[i] item uncompared:
                -------------------------------------------------------------------
                Name                           Type             Size  Value
                -------------------------------------------------------------------
                req                            cip_tl_seq_item  -     @528403

INFO: [FlowCfg] [scratch_path]: [i2c] [/tanap1/proj_cd14/opentitan/jlevy/ot_PR/opentitan/scratch/master/i2c-sim-vcs]
tjaychen commented 2 years ago

do you see this issue with xcelium @Jacob-Levy ? i think right now i2c CI is run with xcelium, so it doesn't show up for whatever reason (i do reproduce this under vcs).

Jacob-Levy commented 2 years ago

I run my CI(private) (using dvsim) locally so it uses only VCS. I did not have an issue with this run few weeks ago... Not sure if we have xcelium locally.

tjaychen commented 2 years ago

closing this for now since the issue is not observed in CI.