fpgasystems / Coyote

Framework providing operating system abstractions and a range of shared networking (RDMA, TCP/IP) and memory services to common modern heterogeneous platforms.
MIT License
207 stars 62 forks source link

OFFLOAD and SYNC behave strangely #42

Open gedatsu217 opened 1 year ago

gedatsu217 commented 1 year ago

OFFLOAD and SYNC behave unexpectedly. I understand OFFLOAD sends data from the host mem to the FPGA mem, and SYNC sends data from the FPGA mem to the host mem. Based on my understanding, I wrote the code below. The setting is the same as perf_mem example (EN_MEM is enabled).

cProcess cproc(0, getpid());

// Memory allocation 
int* fpga_mem1 = (int*)cproc.getMem({CoyoteAlloc::HUGE_2M, 1});
int* host_mem1 = (int*)cproc.getMem({CoyoteAlloc::HOST_2M, 1});
for (int i=0; i<32; i++) host_mem1[i] = i;
int* host_mem2 = (int*)cproc.getMem({CoyoteAlloc::HOST_2M, 1});
for (int i=0; i<32; i++) host_mem2[i] = i+1;

// Print host_mem1 and host_mem2
printf("host_mem1: ");
for (int i=0; i<32; i++) printf("%d ", host_mem1[i]);
printf("\n");
printf("host_mem2: ");
for (int i=0; i<32; i++) printf("%d ", host_mem2[i]);
printf("\n");

// Data transfer
cproc.invoke({CoyoteOper::OFFLOAD, host_mem1, fpga_mem1, 128, 128});
cproc.invoke({CoyoteOper::SYNC, fpga_mem1, host_mem2, 128, 128});
printf("----- Data Transfer -----\n");

// Print host_mem1 and host_mem2 after data transfer
printf("host_mem1: ");
for (int i=0; i<32; i++) printf("%d ", host_mem1[i]);
printf("\n");
printf("host_mem2: ");
for (int i=0; i<32; i++) printf("%d ", host_mem2[i]);
printf("\n");

// Memory free
cproc.freeMem(fpga_mem1);
cproc.freeMem(host_mem1);
cproc.freeMem(host_mem2);

I expect the data of host_mem1 be sent to host_mem2 via fpga_mem1, and then the output of host_mem1 and host_mem2 should be the same. However, the output is as below.

host_mem1: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 
host_mem2: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 
----- Data Transfer -----
host_mem1: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 
host_mem2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

The data of host_mem2 is changed, but I don't know why it is filled with zero. How can I send the data correctly so that the data of host_mem1 and host_mem2 is the same?