620.omnetpp_s_ref failed because PK doesn't provide sys_chdir()

s117 commented 4 years ago

Binary is compiled by https://github.com/s117/riscv-gnu-toolchain/commit/d0bdaa9a282a32cc68e6203098dc1162021ceba7

$ spike -m8192 pk -c omnetpp_s_base.riscv-m64 -c General -r 0

Requesting target memory 0x200000000
******* Resetting core ********** 
****Initializing the processor system****
******* Resetting core ********** 
******* Resetting core ********** 
****Initialization complete****
OMNeT++ Discrete Event Simulation  (C) 1992-2008 Andras Varga, OpenSim Ltd.
Version: 4.0, build: 090310-10709, edition: Academic Public License -- NOT FOR COMMERCIAL USE
See the license for distribution terms and warranty disclaimer
Setting up Cmdenv...
bad syscall array_size 1063 #49->0!
cycle = 3473449
instret = 3473456
******* Resetting core **********

syscall number 49 corresponds to sys_chdir, as defined here: https://github.com/s117/riscv-gnu-toolchain/blob/d0bdaa9a282a32cc68e6203098dc1162021ceba7/linux-headers/include/asm-generic/unistd.h#L167

s117 commented 4 years ago

It seems we have sys_chdir(), it just not exposed.

https://github.com/s117/riscv-fesvr/commit/1a1e918a00f94d287e5af94e646b1670b301f2ad https://github.com/s117/riscv-pk/commit/6da2960b4fdd389bcb112a3018d75fa0c37c762a

s117 commented 4 years ago

Exposed in https://github.com/s117/riscv-pk/commit/1f7fbdbb42d11bb2c0fd72a543e836650b0ee90f

s117 commented 4 years ago

Current sys_chdir will fail if the path pointer is not populated before called. A patch was made in https://github.com/s117/riscv-pk/commit/2d8c93248a2efdeeb2ef511b4b3040a020a50ff1

See the code comment there for why it will fail.

* Seems this issue also exists in the latest upstream code.

s117 commented 4 years ago

A better solution for chdir's unpopulated pointer problem: https://github.com/s117/riscv-pk/commit/eef623a97959315101aa0cf6f360ab860574f09e It still compatible with current FESVR https://github.com/s117/riscv-fesvr/commit/c30724e6407f75456bb77292ecc12ba28605021d, but FESVR can make use of the passed size information (https://github.com/s117/riscv-fesvr/commit/d9f1698425822bdc4dd270baab39b41eb8197c53).

s117 commented 4 years ago

Still doesn't work, even with PK eef623a

I tried to run this benchmark natively on my machine (amd64), and it works. By comparing the system calls trace between amd64 and rv64, it looks like the issue is that the rv64 one failed to list the dir ned.

$ spike -m8192 pk -c omnetpp_s_base.riscv-m64 -c General -r 0

Requesting target memory 0x200000000
******* Resetting core **********
****Initializing the processor system****
******* Resetting core **********
******* Resetting core **********
****Initialization complete****
OMNeT++ Discrete Event Simulation  (C) 1992-2008 Andras Varga, OpenSim Ltd.
Version: 4.0, build: 090310-10709, edition: Academic Public License -- NOT FOR COMMERCIAL USE
See the license for distribution terms and warranty disclaimer
Setting up Cmdenv...
Loading NED files from a SPEC directory 0

Preparing for running configuration General, run #0...
Scenario: $repetition=0
Assigned runID=speccpu-runid

<!> Error: Network `largeNet' not found, check .ini and .ned files.

End.
cycle = 43157479
instret = 43157486
******* Resetting core **********

$ strace -Tfe trace=open,openat,chdir,getcwd spike -m8192 pk -c omnetpp_s_base.riscv-m64 -c General -r 0

...
some unrelated trace
...
Requesting target memory 0x200000000
******* Resetting core **********
****Initializing the processor system****
openat(AT_FDCWD, "pk", O_RDONLY)        = 6 <0.000005>
******* Resetting core **********
******* Resetting core **********
****Initialization complete****
openat(AT_FDCWD, "omnetpp_s_base.riscv-m64", O_RDONLY) = 6 <0.000007>
OMNeT++ Discrete Event Simulation  (C) 1992-2008 Andras Varga, OpenSim Ltd.
Version: 4.0, build: 090310-10709, edition: Academic Public License -- NOT FOR COMMERCIAL USE
See the license for distribution terms and warranty disclaimer
getcwd("<RUN_DIR>/620.omnetpp_s_ref", 1024) = 126 <0.000005>
getcwd("<RUN_DIR>/620.omnetpp_s_ref", 255) = 126 <0.000004>
openat(AT_FDCWD, "omnetpp.ini", O_RDONLY) = 7 <0.000006>
Setting up Cmdenv...
getcwd("<RUN_DIR>/620.omnetpp_s_ref", 255) = 126 <0.000006>
openat(AT_FDCWD, "<RUN_DIR>/620.omnetpp_s_ref/ned/package.ned", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000009>
getcwd("<RUN_DIR>/620.omnetpp_s_ref", 1024) = 126 <0.000004>
chdir("<RUN_DIR>/620.omnetpp_s_ref/ned") = 0 <0.000006>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 255) = 130 <0.000004>
openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 8 <0.000006>
chdir("<RUN_DIR>/620.omnetpp_s_ref") = 0 <0.000005>
Loading NED files from a SPEC directory 0

Preparing for running configuration General, run #0...
Scenario: $repetition=0
Assigned runID=speccpu-runid
getcwd("<RUN_DIR>/620.omnetpp_s_ref", 255) = 126 <0.000004>
openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 9 <0.000007>
getcwd("<RUN_DIR>/620.omnetpp_s_ref", 255) = 126 <0.000005>
openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 10 <0.000007>

<!> Error: Network `largeNet' not found, check .ini and .ned files.

End.
cycle = 43157479
instret = 43157486
******* Resetting core **********
+++ exited with 0 +++

$ strace -Tfe trace=open,openat,chdir ./omnetpp_s_base.x64-m64  -c General -r 0

...
some unrelated trace
...
OMNeT++ Discrete Event Simulation  (C) 1992-2008 Andras Varga, OpenSim Ltd.
Version: 4.0, build: 090310-10709, edition: Academic Public License -- NOT FOR COMMERCIAL USE
See the license for distribution terms and warranty disclaimer
openat(AT_FDCWD, "omnetpp.ini", O_RDONLY) = 3 <0.000007>
Setting up Cmdenv...
openat(AT_FDCWD, "<RUN_DIR>/run_base_refspeed_x64-m64.0000/ned/package.ned", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000008>
chdir("<RUN_DIR>/run_base_refspeed_x64-m64.0000/ned") = 0 <0.000005>
openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4 <0.000006>
openat(AT_FDCWD, "<RUN_DIR>/run_base_refspeed_x64-m64.0000/ned/EtherAppCli.ned", O_RDONLY) = 4 <0.000006>
openat(AT_FDCWD, "<RUN_DIR>/run_base_refspeed_x64-m64.0000/ned/EtherAppSrv.ned", O_RDONLY) = 4 <0.000006>
openat(AT_FDCWD, "<RUN_DIR>/run_base_refspeed_x64-m64.0000/ned/EtherBus.ned", O_RDONLY) = 4 <0.000006>
openat(AT_FDCWD, "<RUN_DIR>/run_base_refspeed_x64-m64.0000/ned/EtherEncap.ned", O_RDONLY) = 4 <0.000006>
openat(AT_FDCWD, "<RUN_DIR>/run_base_refspeed_x64-m64.0000/ned/EtherHost.ned", O_RDONLY) = 4 <0.000006>
openat(AT_FDCWD, "<RUN_DIR>/run_base_refspeed_x64-m64.0000/ned/EtherHub.ned", O_RDONLY) = 4 <0.000006>
openat(AT_FDCWD, "<RUN_DIR>/run_base_refspeed_x64-m64.0000/ned/EtherLLC.ned", O_RDONLY) = 4 <0.000006>
openat(AT_FDCWD, "<RUN_DIR>/run_base_refspeed_x64-m64.0000/ned/EtherMAC.ned", O_RDONLY) = 4 <0.000006>
openat(AT_FDCWD, "<RUN_DIR>/run_base_refspeed_x64-m64.0000/ned/EtherSwitch.ned", O_RDONLY) = 4 <0.000009>
openat(AT_FDCWD, "<RUN_DIR>/run_base_refspeed_x64-m64.0000/ned/LargeNet.ned", O_RDONLY) = 4 <0.000009>
openat(AT_FDCWD, "<RUN_DIR>/run_base_refspeed_x64-m64.0000/ned/MACRelayUnit.ned", O_RDONLY) = 4 <0.000008>
openat(AT_FDCWD, "<RUN_DIR>/run_base_refspeed_x64-m64.0000/ned/MACRelayUnitNP.ned", O_RDONLY) = 4 <0.000008>
openat(AT_FDCWD, "<RUN_DIR>/run_base_refspeed_x64-m64.0000/ned/MACRelayUnitPP.ned", O_RDONLY) = 4 <0.000005>
chdir("<RUN_DIR>/run_base_refspeed_x64-m64.0000") = 0 <0.000008>
Loading NED files from a SPEC directory 13

Preparing for running configuration General, run #0...
Scenario: $repetition=0
Assigned runID=speccpu-runid
openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 4 <0.000009>
openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 4 <0.000005>
Setting up network `largeNet'...
Initializing...

Running simulation...
** Event #1   T=0   Elapsed: 0.000s (0m 00s)  0% completed   ev/sec=0
^C--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
strace: Process 120323 detached
** Event #2042694   T=0.00284808   Elapsed: 0.000s (0m 00s)  0% completed   ev/sec=0

<!> SIGINT or SIGTERM received, exiting.

Calling finish() at end of Run #0...

s117 commented 4 years ago

It's the glob() provided by the glibc doesn't work correctly.

A test program:

#include <stdio.h>
#include <glob.h>

void print_glob(const char* pattern){
  glob_t globdata;
  printf("globbing with patttern %s\n", pattern);
  int globret = glob(pattern, 0, NULL, &globdata);

  if (globret == GLOB_NOSPACE)
    printf("\tglob failed (GLOB_NOSPACE)\n");
  else if (globret == GLOB_ABORTED)
    printf("\tglob failed (GLOB_ABORTED)\n");
  else if (globret == GLOB_NOMATCH)
    printf("\tglob failed (GLOB_NOMATCH)\n");
  else if (globret != 0)
    printf("\tglob failed (reason unknown)\n");

  for (size_t globpos = 0; globpos < globdata.gl_pathc; ++globpos) {
    printf("\t%llu - '%s'\n", globpos, globdata.gl_pathv[globpos]);
  }
}

int main() {
  print_glob("*");
}

Compile with toolchain https://github.com/s117/riscv-gnu-toolchain/commit/d0bdaa9a282a32cc68e6203098dc1162021ceba7 and run it with Spike shows no match:

globbing with patttern *
        glob failed (GLOB_NOMATCH)

But it works correctly on my PC (gcc 10.2.0):

globbing with patttern *
        0 - 'glob_test_simple.c'
        1 - 'glob_test_simple.rv64'
        2 - 'glob_test_simple.rv64.dism'
        3 - 'glob_test_simple.x64'
        4 - 'glob_test_simple.x64.dism'

The call path (partial) to this function in 620.omnetpp_s_ref is:

6 - riscv-glibc/posix/glob.c:287 5 - simulator/fileglobber.cc:142 4 - simulator/nedresourcecache.cc:91 3 - simulator/nedresourcecache.cc:76 2 - simulator/csimulation.cc:217 1 - simulator/envirbase.cc:396

s117 commented 4 years ago

glob() failed because it calls into sys_getdents, which is not implemented yet.

s117 commented 4 years ago

Implemented new syscall sys_getdents64(): PK - https://github.com/s117/riscv-pk/commit/d97131e056c1a426ab4292a85b0c9b97b3d10f87 FESVR - https://github.com/s117/riscv-fesvr/commit/35813b013a61d2a0a62701bfa6c072bdea059dc2

With the changes above, now 620.omnetpp_s_ref can discover and load the .NED files correctly.

$ strace -Tfe trace=open,openat,chdir,getcwd,getdents64 spike -m8192 pk -c omnetpp_s_base.riscv-m64 -c General -r 0

...
some unrelated trace
...
Requesting target memory 0x200000000
******* Resetting core **********
****Initializing the processor system****
openat(AT_FDCWD, "pk", O_RDONLY)        = 6 <0.000005>
******* Resetting core **********
******* Resetting core **********
****Initialization complete****
openat(AT_FDCWD, "omnetpp_s_base.riscv-m64", O_RDONLY) = 6 <0.000006>
OMNeT++ Discrete Event Simulation  (C) 1992-2008 Andras Varga, OpenSim Ltd.
Version: 4.0, build: 090310-10709, edition: Academic Public License -- NOT FOR COMMERCIAL USE
See the license for distribution terms and warranty disclaimer
getcwd("<RUN_DIR>/620.omnetpp_s_ref", 1024) = 126 <0.000005>
openat(AT_FDCWD, "omnetpp.ini", O_RDONLY) = 7 <0.000006>
Setting up Cmdenv...
openat(AT_FDCWD, "<RUN_DIR>/620.omnetpp_s_ref/ned/package.ned", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000010>
getcwd("<RUN_DIR>/620.omnetpp_s_ref", 1024) = 126 <0.000004>
chdir("<RUN_DIR>/620.omnetpp_s_ref/ned") = 0 <0.000006>
openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 8 <0.000006>
getdents64(8, 0x55c0c824e0f0 /* 15 entries */, 32768) = 528 <0.000013>
getdents64(8, 0x55c0c824e0f0 /* 0 entries */, 32768) = 0 <0.000007>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000004>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000004>
openat(AT_FDCWD, "<RUN_DIR>/620.omnetpp_s_ref/ned/EtherAppCli.ned", O_RDONLY) = 9 <0.000007>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000004>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000003>
openat(AT_FDCWD, "<RUN_DIR>/620.omnetpp_s_ref/ned/EtherAppSrv.ned", O_RDONLY) = 10 <0.000006>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000004>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000004>
openat(AT_FDCWD, "<RUN_DIR>/620.omnetpp_s_ref/ned/EtherBus.ned", O_RDONLY) = 11 <0.000009>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000004>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000004>
openat(AT_FDCWD, "<RUN_DIR>/620.omnetpp_s_ref/ned/EtherEncap.ned", O_RDONLY) = 12 <0.000007>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000005>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000005>
openat(AT_FDCWD, "<RUN_DIR>/620.omnetpp_s_ref/ned/EtherHost.ned", O_RDONLY) = 13 <0.000008>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000006>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000005>
openat(AT_FDCWD, "<RUN_DIR>/620.omnetpp_s_ref/ned/EtherHub.ned", O_RDONLY) = 14 <0.000011>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000045>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000005>
openat(AT_FDCWD, "<RUN_DIR>/620.omnetpp_s_ref/ned/EtherLLC.ned", O_RDONLY) = 15 <0.000008>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000004>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000003>
openat(AT_FDCWD, "<RUN_DIR>/620.omnetpp_s_ref/ned/EtherMAC.ned", O_RDONLY) = 16 <0.000008>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000005>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000004>
openat(AT_FDCWD, "<RUN_DIR>/620.omnetpp_s_ref/ned/EtherSwitch.ned", O_RDONLY) = 17 <0.000009>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000037>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000004>
openat(AT_FDCWD, "<RUN_DIR>/620.omnetpp_s_ref/ned/LargeNet.ned", O_RDONLY) = 18 <0.000008>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000004>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000004>
openat(AT_FDCWD, "<RUN_DIR>/620.omnetpp_s_ref/ned/MACRelayUnit.ned", O_RDONLY) = 19 <0.000008>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000005>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000005>
openat(AT_FDCWD, "<RUN_DIR>/620.omnetpp_s_ref/ned/MACRelayUnitNP.ned", O_RDONLY) = 20 <0.000009>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000005>
getcwd("<RUN_DIR>/620.omnetpp_s_ref/ned", 1024) = 130 <0.000007>
openat(AT_FDCWD, "<RUN_DIR>/620.omnetpp_s_ref/ned/MACRelayUnitPP.ned", O_RDONLY) = 21 <0.000011>
chdir("<RUN_DIR>/620.omnetpp_s_ref") = 0 <0.000008>
Loading NED files from a SPEC directory 13

Preparing for running configuration General, run #0...
Scenario: $repetition=0
Assigned runID=speccpu-runid
openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 22 <0.000009>
openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 23 <0.000038>
Setting up network `largeNet'...
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
strace: Process 608993 detached

s117 commented 4 years ago

620.omnetpp_s_ref finished.

$ spike -m8192 pk -c omnetpp_s_base.riscv-m64 -c General -r 0

Requesting target memory 0x200000000
******* Resetting core ********** 
****Initializing the processor system****
******* Resetting core ********** 
******* Resetting core ********** 
****Initialization complete****
OMNeT++ Discrete Event Simulation  (C) 1992-2008 Andras Varga, OpenSim Ltd.
Version: 4.0, build: 090310-10709, edition: Academic Public License -- NOT FOR COMMERCIAL USE
See the license for distribution terms and warranty disclaimer
Setting up Cmdenv...
Loading NED files from a SPEC directory 13

Preparing for running configuration General, run #0...
Scenario: $repetition=0
Assigned runID=speccpu-runid
Setting up network `largeNet'...
Initializing...

Running simulation...
** Event #1   T=0   Elapsed: 0.000s (0m 00s)  0% completed   ev/sec=0
** Event #467312603   T=2.25   Elapsed: 0.000s (0m 00s)  100% completed   ev/sec=0

<!> Simulation time limit reached -- simulation stopped.

Calling finish() at end of Run #0...

End.
cycle = 1118421022034
instret = 1118421022041
******* Resetting core **********

s117 commented 4 years ago

Commit d0401da incorporated this patch.

s117 / anycore-riscv

620.omnetpp_s_ref failed because PK doesn't provide sys_chdir() #13