Problem with performance counters

[evitakogia]

Is there an existing CVA6 bug for this?

• [X] I have searched the existing bug issues

Bug Description

Hello,

I'm trying to read the performance counters(mhpcounter3-mhpcounter8), but I consistently get zeros as results. I've enabled the counters in the riscv-pk/machine/minit.c file and attempt to read them in my executable.

riscv-pk/machine/minit.c:

static void mstatus_init()
{
   if (supports_extension('S'))
      write_csr(scounteren, -1);
   if (supports_extension('U')) 
      write_csr(mcounteren, -1);

   write_csr(mhpmevent3, 1);
   write_csr(mhpmevent4, 2); 
   write_csr(mhpmevent5, 3); 
}

measure-mhp:

uint64_t hpmcounter3 = read_csr(hpmcounter3); 
uint64_t hpmcounter4 = read_csr(hpmcounter4);
uint64_t hpmcounter5 = read_csr(hpmcounter5);

Am I doing something wrong? How can I fix this issue from a hardware perspective?

[JeanRochCoulon]

Hi @evitakogia
The first question I would have is : "which cva6 configuration are you using? " because not all the configurations support the perf counter. I assign the issue ti @TulikaSi who has implemented the perf counter. She will surely can help you.

[TulikaSi]

Hi @evitakogia I completely agree with @JeanRochCoulon . Can you let us know which CVA6 configuration are you using? Because in some of the configurations the parameter CVA6ConfigPerfCounterEn has been tied to 0.

[evitakogia]

Hi @JeanRochCoulon and @TulikaSi , I'm using the cv64a6_imafdc_sv39_openpiton_config_pkg, so CVA6ConfigPerfCounterEn=1.

[TulikaSi]

Hi @evitakogia I have few questions wrt the snippet:

• write_csr(mhpmevent3, 1); --> Are you trying to monitor the L1 I-Cache Misses in the mhpmcounter3 ?
• write_csr(mhpmevent4, 2); --> Are you trying to monitor the L1 D-Cache Misses in the mhpmcounter4 ?
• write_csr(mhpmevent5, 3);--> Are you trying to monitor the ITLB Misses in the mhpmcounter5 ?

I am asking so beacuse the five LSB of the event selector CSRs(mhpmevent) are written to select the event that one needs to count from a particular generic event counter. And if the answer is yes then the event should have occured for the counter to count. Let me know if I understand it correctly.

Also you can refer the document for a better understanding https://github.com/openhwgroup/cva6/blob/master/docs/01_cva6_user/CSR_Performance_Counters.rst

[evitakogia]

@TulikaSi, the answer to all three questions is yes. I have tried for the first 6 events (L1 I-Cache Misses, L1 D-Cache Misses, ITLB Misses, DTLB Misses, Load Accesses, Store Accesses), but the result is zero for all of them. The exact executable I am running involves creating an array with a size equal to size * PAGE_SIZE * sizeof(char), with size being a power of 2 (16, 32, 64, 128, etc.), and PAGE_SIZE set to 4096. After that, I fill it with data and finally access it with a step equal to PAGE_SIZE for 100 iterations. Therefore, it seems quite challenging for the results I obtain to be valid for these events.

[evitakogia]

Hello @TulikaSi and sorry for the inconvenience, but I'm quite confused. Could you please tell me the exact steps you follow to get non-zero values for mhpcounter3-mhpcounter8?

[TulikaSi]

Hi @evitakogia I did manual simulation to test the counters in Vivado but as far as the behavioural functionality testing of the counters for each event is concerned using an application, I have not performed that. The CSR access tests i.e. read and write were also able to run properly. @JeanRochCoulon Do you have any idea who can probably help us in verifying it

[JeanRochCoulon]

@evitakogia is working on openpiton configuration which cannot be simulated (easily) using only cva6 repository. The issue need to be adapted to not openpiton configuration to be analyzed and fixed. As nobody is assigned to verify perf counter Today, I would encourage @evitakogia to adapt the test case to other configuration than openpiton, and @TulikaSi to understand the issue to fix it.

[evitakogia]

@JeanRochCoulon, if by the term "openpiton" you mean this, then I have never worked with this particular one. What I do is run the following commands.

make verilate
./work-ver/Variane_testharness $RISCV/riscv64-unknown-elf/bin/pk hello_world

hello_world.c:

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char* arg[]) {
    printf("Hello World !");
}

I notice that in the cva6/core/perf_counters.sv file, the variable addr_i never takes the values B03, B04, B05, B06, B07 and B08, which correspond to mhpmcounter3-mhpmcounter8. The addr_i takes its values from the cva6/core/csr_buffer.sv file and specifically from the variable csr_reg_n.csr_address. In this file, there is the following piece of code:

    //$fdisplay(fd, "%t fu_data_i.operand_b[11:0]: %h", $time, fu_data_i.operand_b[11:0]);
    if (csr_valid_i) begin
      //$fdisplay(fd, "%t valid fu_data_i.operand_b[11:0]: %h", $time, fu_data_i.operand_b[11:0]);
      csr_reg_n.csr_address = fu_data_i.operand_b[11:0];
      csr_reg_n.valid       = 1'b1;
    end

If I add a $display before the if statement and a $display inside the if statement (as shown above), Ι observe that we have the values B03, B04, B05, B06, B07, B08 before the if statement, but due to csr_valid_i, they are never transferred from fu_data_i.operand_b to csr_reg_n.csr_address. Therefore, I believe the issue lies in csr_valid_i. The csr_valid_i takes its values from the cva6/core/issue_read_operands.sv file.

If I make slight modifications to the hello_world.c, then I notice that there is an issue in the fu_data_i.operand_b[11:0] as well because the $display before the if statement does not print the values B03, B06, B07.

hello_world_counters.c:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <limits.h>

#define uint64_t __uint64_t
#define asm __asm__

#define read_csr(reg) ({ unsigned long __tmp; \
        asm volatile ("csrr %0, " #reg : "=r"(__tmp)); \
        __tmp; })

int main(int argc, char* arg[]) {
   uint64_t total_cycles = read_csr(cycle);
   uint64_t hpmcounter3 = read_csr(hpmcounter3);
   uint64_t hpmcounter4 = read_csr(hpmcounter4);
   uint64_t hpmcounter5 = read_csr(hpmcounter5);
   uint64_t hpmcounter6 = read_csr(hpmcounter6);
   uint64_t hpmcounter7 = read_csr(hpmcounter7);
   uint64_t hpmcounter8 = read_csr(hpmcounter8);
   uint64_t total_instr = read_csr(instret);

   printf("%lu %lu %lu %lu %lu %lu %lu %lu\n", total_cycles, hpmcounter3, hpmcounter4, hpmcounter5, hpmcounter6, hpmcounter7, hpmcounter8, total_instr);

   printf("Hello World !");

   total_cycles = read_csr(cycle);
   hpmcounter3 = read_csr(hpmcounter3);
   hpmcounter4 = read_csr(hpmcounter4);
   hpmcounter5 = read_csr(hpmcounter5);
   hpmcounter6 = read_csr(hpmcounter6);
   hpmcounter7 = read_csr(hpmcounter7);
   hpmcounter8 = read_csr(hpmcounter8);
   total_instr = read_csr(instret);

   printf("%lu %lu %lu %lu %lu %lu %lu %lu\n", total_cycles, hpmcounter3, hpmcounter4, hpmcounter5, hpmcounter6, hpmcounter7, hpmcounter8, total_instr);
}

Nevertheless, regarding the values C03, C04, C05, C06, C07, C08 (i.e. hpmcounter3-hpmcounter8), I think they work correctly, but since mhpmcounter3-mhpmcounter8 do not have values, it is logical for them to be zero as well.

@TulikaSi, as you can imagine, I ran much more complex executables, but the values continue to be zero. That's why I tried to run a simple program like hello_world.c.

[jlerhun]

Hello, I've made some tests of the performance counters, and I witness the same behaviour.

I'm using the CV64A6 on Genesys 2 FPGA board, in bare-metal mode, debug with gdb and a serial console. The config is cv64a6_imafdc_sv39, so the performance counters are enabled.

The test software runs in machine mode, so setting mcounteren and scounteren is not required, but it's done anyway. Then mcountinhibit is cleared, to enable all counters. I configure all six counters with various events by writing to mhpmevent*. Reading back those CSR shows the expected value.

Writing a specific value into a counter mhpmcounter* (with or without setting the corresponding mcountinhibit bit to stop counting) reads back as zero.

After reading the initial value of the counters, I run a loop intended to trigger in a controlled way a particular event to be counted, then read the counters again and subtract the initial value.

All counters read back as zero, for all events, except icache_access (16) that shows a strange behaviour (sometimes decreasing value after the loop). I did not test TLB-related counters.

I hope this gives some clues to find the issue.

[TulikaSi]

Hi @evitakogia @jlerhun I will try to reproduce the issue locally and then it will be better for me to debug the issue and find a fix for it.

[ncppd]

@TulikaSi I too have the same behavioural issue with counters not increasing and returning zero, do you have any updates on this? Thanks!