Confused about the results of "ipmctl show -dimm -performance MediaReads" ?

[Hotxuer]

Hi all,

I was using the commands "ipmctl show -dimm -performance" to measure the PM access amount of my program. But i found the results of MediaReads not as expected. It seems that performing a read and then a write to a 64-byte block in PM would result in a total of 2 XPLine (256 bytes) reads (i.e. the amount of change in MediaReads is 8, while I think it should be 4).

In my test program, I keep picking a random PM address, first perform a read on the 64-byte block at this address, then immediately perform a write to this block, and repeat the whole process, recording the total number of operations. At the beginning and end of the program, I also use "ipmctl show -dimm -performance" to record the amount of MediaReads, MediaWrites, ReadRequests, WriteRequests at that time, so that I can calculate the change of these data caused by each operation. I have disabled the hardware prefetcher of CPU when i run the program. The results are as follows:

[PM COUNTER] MediaReads: total = 67257116, average = 8.385
[PM COUNTER] MediaWrites: total = 35471668, average = 4.280
[PM COUNTER] ReadRequests: total = 7792815, average = 1.006
[PM COUNTER] WriteRequests: total = 7794509, average = 1.006

How should this result be explained? Why ReadRequests is 1 but it generates 2 of XPLine reads to PM media? Shouldn't the change of MediaReads be the same as the change of MediaWrites in this process? After all, the changes of ReadRequests and WriteRequests are the same. Or do writes to PM media repeatedly cause an increase in MediaReads, even if the same XPLine has already been read a while ago?

I also did a read-only test on random 64-byte blocks, and the results were fine, just as expected:

[PM COUNTER] MediaReads: total = 47807276, average = 4.121
[PM COUNTER] MediaWrites: total = 4188212, average = 0.087
[PM COUNTER] ReadRequests: total = 10871060, average = 1.005
[PM COUNTER] WriteRequests: total = 32133, average = 0.003

Here are the core parts of my test program:

void worker() 
{
        char* buffer = new char[4*chunk];
        while (done == 0)
        {
            volatile auto next_operation = benchmark->nextOperation();
            long long d = next_operation.second;
            uint64_t offset; 

            offset = (rdm.randomInt() % (SPACE / chunk)) * chunk;
            memcpy(buffer, (uint64_t*)(curr_addr + offset), chunk);
            memset((uint64_t*)(curr_addr + offset), d, chunk);
            flush_data((char*)(curr_addr + offset), chunk);
            mfence();

            result->throughput++;
        }
}

pm_count();
worker();
pm_count(true, result.throughput);

uint64_t pm_count_records[4] = { 0, 0, 0, 0 };
const std::string pm_count_cmds[4] = { "MediaReads", "MediaWrites", "ReadRequests", "WriteRequests" };

std::string get_cmd(int index) {
  return "sudo ipmctl show -dimm -performance " + pm_count_cmds[index];
}

std::string get_flag(int index) {
  return pm_count_cmds[index] + "=0x";
}

static void pm_count(bool end = false, uint64_t operation_num = 20000000) {
  mfence();
  char line[1024];
  FILE *fp;
  for (int i = 0; i < 4; i++) {
    std::string curr_cmd = get_cmd(i);
    std::string curr_flag = get_flag(i);
    uint64_t count_num = 0;
    if ((fp = popen(curr_cmd.data(), "r")) == NULL) {
        printf("[PM COUNTER] error\n");
        exit(0);
    }
    while (fgets(line, sizeof(line) - 1, fp) != NULL) {
        std::string line_str = line;
        uint64_t start_pos = line_str.find(curr_flag);
        if (start_pos != std::string::npos) {    
            char* tmp;
            std::string xxx = get_flag(i);
            uint64_t i = strtol(line_str.substr(start_pos + curr_flag.size(), 
                                line_str.size()).c_str(), &tmp, 16);
            count_num += i;
        }
    }
    pclose(fp);

    // check to print counting results since last time
    if (end) {
      printf("[PM COUNTER] %s: total = %llu, average = %.3f\n",
              pm_count_cmds[i].c_str(),
              (count_num - pm_count_records[i]),
              double(count_num - pm_count_records[i]) / operation_num);
    }
    pm_count_records[i] = count_num;
  }
}

Best,

[nolanhergert]

ipmctl is only reporting the fields from the PMem module here. I think you will have a better chance of response if you post at https://groups.google.com/g/pmem or similar.

However, if I were to guess, could the flush operation be doing a read afterwards to ensure it went correctly?

[Hotxuer]

Thanks a lot for your time and advice!