test signatures

[algrobman]

I'm new to this repo, and failed to find the tests signatures, results. I think these were provided at some point .. What do I check/compare with my run results?

[allenjbaum]

Test signatures are now generated dynamically by the riscof test framework, based on the specific configuration of your hart, which is described by a riscv-config YAML formatted file. This enables a single set of test to be run regardless of the architectural options supported by your hart (for the basic RV32I/RV64I ISA, that's mostly just HW misalign support)

The short form is that riscof selects, configures, and runs tests based on the config file on both the Sail formal model, and on your hart, and compares the results.

Riscv-config is described by the documentation here: https://github.com/riscv-software-src/riscv-config Riscof is described by the documentation here:https://github.com/riscv-software-src/riscof

[eroom1966]

Hi @allenjbaum Is there a step-by-step guide tutorial showing all stages required

• packages to install
• repositories to clone
• execution flow

In order to get to the point where a set of programs and signatures are available to run on the RTL

what privileges are required for the above steps, many professional companies will not provide the users with root level privileges to perform these tasks where programs/packages are installed globally

Also, is there any document detailing which devices have so far successfully run through the process, and are allowed to use the logo indicating compliance with the tests

[UmerShahidengr]

Hi @eroom1966, Riscof documentation (available here) is comprehensive, and contains a quick start guide which will give you a step by step instruction to set up the tools, configure a project, set up DUT and ref (Sail), run basic tests, and explain the execution flow. For most of the packages, root/privilege access will not be needed. If you follow the quick start procedure, you will be able to run tests easily without having any trouble of privilege access.

[stnolting]

Here is a simple port of the RISCOF framework I made for a custom RISC-V core: https://github.com/stnolting/neorv32-riscof

Maybe this can be helpful for you to get started 😉

[allenjbaum]

Those are interesting questions, especially the last.

The documentation for packages needing installation, repos that need to be cloned, etc. can be found in the links I shared above, and some of the replies above/below. The procedures for self-certification are listed in this policy document here: https://docs.google.com/document/d/1bXzONWVxXCp0wUigVDE2bQDU13uQRsZM80pmbXbERQc . See the section "Where to Report Test Results"

I'll clean up the README.md in that directory to add that information.

As of this date, no PR has been submitted, though there are several implementations that have run through the entire flow (and there is a github repo with scripts that can ease the process here: https://github.com/InspireSemi/riscof_install_example_DUT

Note that, as you well know, that the existing tests are woefully short of complete (especially compared to yours), but are slowly getting built up.

On Tue, Oct 25, 2022 at 12:08 AM Lee Moore @.***> wrote:

Hi @allenjbaum https://github.com/allenjbaum Is there a step-by-step guide tutorial showing all stages required

• packages to install
• repositories to clone
• execution flow

In order to get to the point where a set of programs and signatures are available to run on the RTL

Also, is there any document detailing which devices have so far successfully run through the process, and are allowed to use the logo indicating compliance with the tests

— Reply to this email directly, view it on GitHub https://github.com/riscv-non-isa/riscv-arch-test/issues/290#issuecomment-1290086733, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXVJXHDPJSSP5RYFBWU53WE6BNBANCNFSM6AAAAAARNNRAIA . You are receiving this because you were mentioned.Message ID: @.***>

[MarcKarasek]

I would clone the repo Allen mentioned https://github.com/InspireSemi/riscof_install_example_DUT

This has scripts to install all dependencies on ubuntu and rhel8. It saves a lot of time. It also has a sample DUT dir and scripts to run the framework. Also saves time.

On Tue, Oct 25, 2022 at 6:44 PM Allen Baum @.***> wrote:

Those are interesting questions, especially the last.

The documentation for packages needing installation, repos that need to be cloned, etc. can be found in the links I shared above, and some of the replies above/below. The procedures for self-certification are listed in this policy document here:

https://docs.google.com/document/d/1bXzONWVxXCp0wUigVDE2bQDU13uQRsZM80pmbXbERQc . See the section "Where to Report Test Results"

I'll clean up the README.md in that directory to add that information.

As of this date, no PR has been submitted, though there are several implementations that have run through the entire flow (and there is a github repo with scripts that can ease the process here: https://github.com/InspireSemi/riscof_install_example_DUT

Note that, as you well know, that the existing tests are woefully short of complete (especially compared to yours), but are slowly getting built up.

On Tue, Oct 25, 2022 at 12:08 AM Lee Moore @.***> wrote:

Hi @allenjbaum https://github.com/allenjbaum Is there a step-by-step guide tutorial showing all stages required

• packages to install
• repositories to clone
• execution flow

In order to get to the point where a set of programs and signatures are available to run on the RTL

Also, is there any document detailing which devices have so far successfully run through the process, and are allowed to use the logo indicating compliance with the tests

— Reply to this email directly, view it on GitHub < https://github.com/riscv-non-isa/riscv-arch-test/issues/290#issuecomment-1290086733 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AHPXVJXHDPJSSP5RYFBWU53WE6BNBANCNFSM6AAAAAARNNRAIA

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/riscv-non-isa/riscv-arch-test/issues/290#issuecomment-1291217797, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD72ZM55FGHTW35V24CKA53WFBPEDANCNFSM6AAAAAARNNRAIA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- /**** Marc Karasek ASM T648 Herder of Cats ****/

[algrobman]

BTW, is this repo the "official RISCV compliance test" suit ? what is the status of this repo: https://github.com/riscv-software-src/riscv-tests ?

[allenjbaum]

Those are much earlier tests and are not part of the official RISC Conmpatibility (not Compliance) suite. They are (currently) more comprehensive, but also inadequate coverage, and support fewer optional behaviors allowed by the spec. i.e. they assume specific options that may not be present in a DUT (including which bits of a CSR are RW vs RdOnly, and their allowed ranges)

On Thu, Oct 27, 2022 at 10:53 AM algrobman @.***> wrote:

BTW, is this repo the "official RISCV compliance test" suit ? what is the status of this repo: https://github.com/riscv-software-src/riscv-tests ?

— Reply to this email directly, view it on GitHub https://github.com/riscv-non-isa/riscv-arch-test/issues/290#issuecomment-1293874392, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXVJTXGRD75XZDK3MDHADWFK6TFANCNFSM6AAAAAARNNRAIA . You are receiving this because you were mentioned.Message ID: @.***>

[algrobman]

Honestly, this new RISCOF framework is pretty troublesome from installation point of view. It would be much easier for users like me to have self-checking tests, containing predicted results, at least for user spec instructions, which is not that hard to create. I understand , that it is more difficult to provide privileged spec tests results and it's much easier for creators to rely on the framework, but this is not such for users of this repo. I would like you guys to take better care of your clients ...

[allenjbaum]

I think you just answered your own question. Yes, we could (usually) have tests for the bare user mode stuff - but there will be very, very few implementations that don't use the priv spec. And once you're committed to that, self testing goes out the window, and the cost of also doing unpriv is negligible. But, just the existence of Zicsr (which is required for several unpriv extensions) and the open-ended definition of a WARL field makes self-checking difficult.

But even the simplest unpriv tests (load, store) can't have precomputed signatures, because misaligned accesses can be handled either in hardware or by trap and emulate - and in the general case it is not possible to know which. Welcome to our world.

We are moving to get the installation to be much easier. Marc's repo is one step. We also intend to have a "Reference signature as a Service" implementation, which will make getting those reference signatures a much simpler process. All of this depends on the kindness of volunteers to implement. If you want to throw a pile of money at RVI to speed this up, it probably could be.

[algrobman]

Allen, thanks for explanation, our main obstacle is reluctance of IT to install all these service packages. I couldn't make them to install opam, for example . Another roadblock is Linux versions. We have 7.x something and there is no way I can make them to switch to something more modern. Can you consider to use some more common tools and not the latest and greatest?

[MarcKarasek]

I am assuming you are using RHEL 7.

You don't need to install a lot of packages. Have you looked at the repo I setup to help get up and running? https://github.com/InspireSemi/riscof_install_example_DUT

It has scripts for what you need to install as sudo on a rhel 8 system. (Should be similar to RHEL 7). Along with scripts to install the rest of the packages to your home dir under ~/.local/ opam for example is not available as a package for RHEL 8, so I just grabbed the linux binary from the opam site and one of the scripts puts it into ~/.local/bin..

Marc

On Thu, Oct 27, 2022 at 6:35 PM algrobman @.***> wrote:

Allen, thanks for explanation, our main obstacle is reluctance of IT to install all these service packages. I couldn't make them to install opam, for example . Another roadblock is Linux versions. We have 7.x something and there is no way I can make them to switch to something more modern. Can you consider to use some more common tools and not the latest and greatest?

— Reply to this email directly, view it on GitHub https://github.com/riscv-non-isa/riscv-arch-test/issues/290#issuecomment-1294196719, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD72ZM24OP6BIYMLMA7UVELWFL7STANCNFSM6AAAAAARNNRAIA . You are receiving this because you commented.Message ID: @.***>

-- /**** Marc Karasek ASM T648 Herder of Cats ****/

[allenjbaum]

The reference signature as a service is being developed exactly because of the IT environment. Now we just need someone to go implement it; RIOS Labs in China has volunteered machine and disk resources for it. Linux versions: I actually don't know what breaks if you use an older Linux version, so I can't say.

On Thu, Oct 27, 2022 at 3:35 PM algrobman @.***> wrote:

Allen, thanks for explanation, our main obstacle is reluctance of IT to install all these service packages. I couldn't make them to install opam, for example . Another roadblock is Linux versions. We have 7.x something and there is no way I can make them to switch to something more modern. Can you consider to use some more common tools and not the latest and greatest?

— Reply to this email directly, view it on GitHub https://github.com/riscv-non-isa/riscv-arch-test/issues/290#issuecomment-1294196719, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXVJS5JN3DIVLNPCAS4VDWFL7STANCNFSM6AAAAAARNNRAIA . You are receiving this because you were mentioned.Message ID: @.***>

[algrobman]

Allen, Marc, Thanks again for your responses.

1) Marc, I did not try your repo - I'm not IT person and have no permissions/knowledge to install anything on company servers. I referred opam as I had installation problems with it in another company for litmus tests evaluation. IT couldn't install it due older Linux version. 2) Allen, what is approximate time frame, when this service be available?

3) could the test suite run against spike step by step locally be considered as compliance certification ?

[allenjbaum]

2: we need to find a (volunteer) resource to set up the web pages and

infrastructure. We've not identified anyone to do that, so I can't give you.a timeline.

3 You might be able to use Spike as a reference model, instead of Sail

• but if you don't have permissions/knowledge to install Spike (it's outside code from a git repo), then that is a rather pointless aternative.

The OPAM issue is a real issue, and that's why Marc, who was under similar constraints, came up with his repo. We intend to make it even easier with the RSaaS approach, but we don't have a timeline.

We've said we will Spike allow to be a reference model IFF Sail doesn't support the extension (new extensions usually go into Spike first) The ACT framework allows that; its a simple configuration option. But Spike may not give the same answers that Sail will give, because it isn't designed to be as configurable (specifically weird architectural options for privileged modes, but especially WARL CSR semantics), so yea, you could select that, and you'd find they don't match.

We are not set up to accept a cycle by cycle comparison period (let's not even get into the ambiguity of what a "cycle is) DV - which only needs to deal with a single model at a time, can get away with exposing internal state and using it. But it doesn't scale to hundreds of implementations, each of which has their own microarchitecture and their own timings.

Our mandate is testing that anyone's DUT that is thrown at us is architecturally compliant, andti do that at scale we look at only the least common denominator for all models: the contents of memory

The only custom support that the ACT framework requires is to be able to:

• link and load an ELF formatted binary,
• start it executing,
• recognize when a test is complete (typically a write to a model specific MMIO address signals that to the RTL simulator)
• extract the contents of a memory range. We expect all DV environments can be set up to perform each of those functions already, That's as non-invasive as we can make it.

The framework takes that memory region signature, and generates a test report that we can audit, - otherwise,

• we don't know where it's coming from, and we can't depend on it.

This will be harsh:: If your IT organization won't allow you to install RISC-V tools, and your management agrees with that restriction, then frankly it sounds like they're not interested in certifying that you're RISC-V compatible. It seems a bit disingenuous that you might be able to run Spike.. but not other RISC-V tools,

If Eventually you won't have to run the Sail reference model (you can just specify the particulars of your core, and we'll generate it)- but that isn't going to happen overnight. But even if you did, you still need to run our tests on your DUT, and use our framework to do the comparison. You may not have permission/knowledge to install/run the framework, but if your management is serious about being able to use RISC-V trademarks, they should grant that permission.

On Fri, Oct 28, 2022 at 9:05 AM algrobman @.***> wrote:

Allen, Marc, Thanks again for your responses.

1.

Marc, I did not try your repo - I'm not IT person and have no permissions/knowledge to install anything on company servers. I referred opam as I had installation problems with it in another company for litmus tests evaluation. IT couldn't install it due older Linux version. 2.

Allen, what is approximate time frame, when this service be available? 3.

could the test suite run against spike step by step locally be considered as compliance certification ?

— Reply to this email directly, view it on GitHub https://github.com/riscv-non-isa/riscv-arch-test/issues/290#issuecomment-1295175599, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXVJRDOXWHH3IHRONSBYLWFP2ULANCNFSM6AAAAAARNNRAIA . You are receiving this because you were mentioned.Message ID: @.***>

[algrobman]

A few comments:

we have spike and riscv tools as part of GCC compiler , installed already. We do run your tests in our DV environment, and compare DUT instruction trace with spike trace, or compare CPU state at every instruction commit with spike during simulation. Can we be certified with such approach?

BTW, what's ACT?

[algrobman]

Just tryied to install riscof: " % pip3 install git+https://github.com/riscv/riscof.git Defaulting to user installation because normal site-packages is not writeable Collecting git+https://github.com/riscv/riscof.git Cloning https://github.com/riscv/riscof.git to /tmp/pip-req-build-nqqyryar Running command git clone --filter=blob:none --quiet https://github.com/riscv/riscof.git /tmp/pip-req-build-nqqyryar /grid/common/pkgs/git/v2.30.0/libexec/git-core/git-remote-https: /grid/common/pkgs/openssl/v1.0.2h/lib/libcrypto.so.1.0.0: no version information available (required by /lib64/libldap-2.4.so.2) /grid/common/pkgs/git/v2.30.0/libexec/git-core/git-remote-https: /grid/common/pkgs/openssl/v1.0.2h/lib/libcrypto.so.1.0.0: no version information available (required by /lib64/libldap-2.4.so.2) /grid/common/pkgs/git/v2.30.0/libexec/git-core/git-remote-https: /grid/common/pkgs/openssl/v1.0.2h/lib/libssl.so.1.0.0: no version information available (required by /lib64/libldap-2.4.so.2) /grid/common/pkgs/git/v2.30.0/libexec/git-core/git-remote-https: /grid/common/pkgs/openssl/v1.0.2h/lib/libcrypto.so.1.0.0: no version information available (required by /lib64/libldap-2.4.so.2) /grid/common/pkgs/git/v2.30.0/libexec/git-core/git-remote-https: /grid/common/pkgs/openssl/v1.0.2h/lib/libcrypto.so.1.0.0: no version information available (required by /lib64/libldap-2.4.so.2) /grid/common/pkgs/git/v2.30.0/libexec/git-core/git-remote-https: /grid/common/pkgs/openssl/v1.0.2h/lib/libssl.so.1.0.0: no version information available (required by /lib64/libldap-2.4.so.2) Resolved https://github.com/riscv/riscof.git to commit 4f8b28afb60bf55fdc1bf22b3e8ec146dc14e6cd Preparing metadata (setup.py) ... error error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [1 lines of output] ERROR: Can not execute setup.py since setuptools is not available in the build environment. [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed

× Encountered error while generating package metadata. ╰─> See above for output.

note: This is an issue with the package mentioned above, not pip. hint: See above for details. WARNING: There was an error checking the latest version of pip. " here our python versions:

% python --version Python 3.7.0 % pip3 --version pip 22.1 from /grid/common/pkgsData/python-v3.7.0/Linux/RHEL6.0-2013-x86_64/lib/python3.7/site-packages/pip (python 3.7)

what's wrong?

[pawks]

Looks like you are installing from git. In which case there is an inherent requirement for the setup tools package to be installed. You will need to run the following command to support installing from source directly:

pip3 install setuptools

PS - The recommended method of installing packages for users is from the pypi directly. I would advise you to use that if possible. You can install riscof using the command: pip3 install riscof

[algrobman]

great! got this: % pip3 install riscof Defaulting to user installation because normal site-packages is not writeable Collecting riscof Downloading riscof-1.25.2-py2.py3-none-any.whl (45 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.2/45.2 kB 2.2 MB/s eta 0:00:00 Collecting pytz>=2019.1 Downloading pytz-2022.5-py2.py3-none-any.whl (500 kB)

.... Collecting Cerberus>=1.3.1 Downloading Cerberus-1.3.4.tar.gz (63 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.8/63.8 kB 2.6 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... error error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [65 lines of output] running dist_info creating /tmp/pip-modern-metadata-oebx0euy/Cerberus.egg-info writing /tmp/pip-modern-metadata-oebx0euy/Cerberus.egg-info/PKG-INFO ....

[algrobman]

end of log: File "/tmp/pip-build-env-bkc4337g/overlay/lib/python3.7/site-packages/wheel/macosx_libfile.py", line 41, in import ctypes File "/grid/common/pkgsData/python-v3.7.0/Linux/RHEL6.0-2013-x86_64/lib/python3.7/ctypes/init.py", line 7, in from _ctypes import Union, Structure, Array ImportError: libffi.so.7: cannot open shared object file: No such file or directory [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed

× Encountered error while generating package metadata. ╰─> See above for output.

note: This is an issue with the package mentioned above, not pip. hint: See above for details. WARNING: There was an error checking the latest version of pip.

[pawks]

Looks like a missing dependency(libffi). You could install it using: yum install libffi or yum install libffi-devel.

[algrobman]

% yum install libffi Loaded plugins: aliases, changelog, kabi, langpacks, product-id, search-disabled-repos, subscription-manager, tmprepo, verify, versionlock Loading support for Red Hat kernel ABI You need to be root to perform this command.

[algrobman]

Seems there is an installation problems here for multiuser corporate environments. Your stuff is working relatively well for single users environments , like personal Ubuntu. But how do I convince IT of the company to go through all this hassle to upgrade Linux to the state required for your tools? They keep Linux common for all users, which may not want any changes in the OS.

It should be some way to provide pretty confined tools, won't require such drastic measures. (maybe, Perl based, which works everywhere)..

why one needs to reinstall corporate OS to compile and run a few tests?

[algrobman]

speaking of which .. Allen, Marc, could you help to certify SweRV EL2? https://github.com/chipsalliance/Cores-SweRV-EL2 this is simple M-mode rv32imc core, opensource, can run with free tools (verilator) or with Cadence, Synopsys or Mentor simulators. It has simple Demo TB to run a few tests I think, step by step guide /report how to run your compliance on this design. could be beneficial for future users ...

[MarcKarasek]

Tell you what clone the repo for example DUT. I think I posted it in a repy. Add this sim using the example_dut in the repo.

I will help you and answer any questions you have.

Marc

On Sat, Oct 29, 2022 at 1:52 PM algrobman @.***> wrote:

speaking of which .. Allen, Marc, could you help to certify SweRV EL2? https://github.com/chipsalliance/Cores-SweRV-EL2 this is simple M-mode rv32imc core, opensource, can run with free tools (verilator) or with Cadence, Synopsys or Mentor simulators. It has simple Demo TB to run a few tests I think, step by step guide /report how to run your compliance on this design. could be beneficial for future users ...

— Reply to this email directly, view it on GitHub https://github.com/riscv-non-isa/riscv-arch-test/issues/290#issuecomment-1295917553, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD72ZMYZ5YLZH2GH55P6U73WFVP43ANCNFSM6AAAAAARNNRAIA . You are receiving this because you commented.Message ID: @.***>

-- /**** Marc Karasek ASM T648 Herder of Cats ****/

[MarcKarasek]

for one you do not have openssl installed. This is used for any https// access like cloning a repo.

again if you would bother to clone the example dut repo a lot of this is already spelled out for you. What needs to be installed using sudo (your IT ) and then everything else including riscof can be installed toi you local home dir. under ~/.local/bin

If you do not mind me asking what company/organization are you with?

On Sat, Oct 29, 2022 at 11:52 AM algrobman @.***> wrote:

Just tryied to install riscof: " % pip3 install git+https://github.com/riscv/riscof.git Defaulting to user installation because normal site-packages is not writeable Collecting git+https://github.com/riscv/riscof.git Cloning https://github.com/riscv/riscof.git to /tmp/pip-req-build-nqqyryar Running command git clone --filter=blob:none --quiet https://github.com/riscv/riscof.git /tmp/pip-req-build-nqqyryar /grid/common/pkgs/git/v2.30.0/libexec/git-core/git-remote-https: /grid/common/pkgs/openssl/v1.0.2h/lib/libcrypto.so.1.0.0: no version information available (required by /lib64/libldap-2.4.so.2) /grid/common/pkgs/git/v2.30.0/libexec/git-core/git-remote-https: /grid/common/pkgs/openssl/v1.0.2h/lib/libcrypto.so.1.0.0: no version information available (required by /lib64/libldap-2.4.so.2) /grid/common/pkgs/git/v2.30.0/libexec/git-core/git-remote-https: /grid/common/pkgs/openssl/v1.0.2h/lib/libssl.so.1.0.0: no version information available (required by /lib64/libldap-2.4.so.2) /grid/common/pkgs/git/v2.30.0/libexec/git-core/git-remote-https: /grid/common/pkgs/openssl/v1.0.2h/lib/libcrypto.so.1.0.0: no version information available (required by /lib64/libldap-2.4.so.2) /grid/common/pkgs/git/v2.30.0/libexec/git-core/git-remote-https: /grid/common/pkgs/openssl/v1.0.2h/lib/libcrypto.so.1.0.0: no version information available (required by /lib64/libldap-2.4.so.2) /grid/common/pkgs/git/v2.30.0/libexec/git-core/git-remote-https: /grid/common/pkgs/openssl/v1.0.2h/lib/libssl.so.1.0.0: no version information available (required by /lib64/libldap-2.4.so.2) Resolved https://github.com/riscv/riscof.git to commit 4f8b28afb60bf55fdc1bf22b3e8ec146dc14e6cd Preparing metadata (setup.py) ... error error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [1 lines of output] ERROR: Can not execute setup.py since setuptools is not available in the build environment. [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed

× Encountered error while generating package metadata. ╰─> See above for output.

note: This is an issue with the package mentioned above, not pip. hint: See above for details. WARNING: There was an error checking the latest version of pip. " here our python versions:

% python --version Python 3.7.0 % pip3 --version pip 22.1 from /grid/common/pkgsData/python-v3.7.0/Linux/RHEL6.0-2013-x86_64/lib/python3.7/site-packages/pip (python 3.7)

what's wrong?

— Reply to this email directly, view it on GitHub https://github.com/riscv-non-isa/riscv-arch-test/issues/290#issuecomment-1295879731, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD72ZM6UMLZIEXN4BYMDGJLWFVB4HANCNFSM6AAAAAARNNRAIA . You are receiving this because you commented.Message ID: @.***>

-- /**** Marc Karasek ASM T648 Herder of Cats ****/

[eroom1966]

I think @algrobman has underlined a real problem here. I work with a lot of corporate CAD/IT departments, It is always difficult getting corporate wide installations through these groups. In my experience none of these groups I have worked with would be prepared to go through the trial/error process we are seeing unfold here.

Either everything in the flow needs to be reproducible with user level permissions, OR, a well defined description of what needs to be installed- prior to installing the tools required to use this flow.

It seems implausible to me, that an approach of trial/error would be acceptable to a corporate CAD group

I am truly sorry if this sounds difficult to hear by those who have architected this system, I don’t underestimate the amount of effort expended, but I believe we have to deal with the world (and corporate CAD/IT groups!) as it is, not how we would like it to be.

[algrobman]

I have a few follow up questions.

1) what is legality of this repo? How a user can be sure that this repo represents official view of RISCV International or riscv.org? I couldn't find any reference of this repo on their web site. Google search brings multiple "compliance tests suits" repos - all with different tests ... 2) How can a user trust the software from this and any referred sites in sense of cyber security? (are all these required installations spyware free? (what if "Russian hackers", disguised as Czechs, had planted a spyware, will steal my company tread secrets 😊 ?
We just got a corporate email, warning from installing SW on our computers by ourselves ...

regarding Lee's comment , I'm not sure I can convince our IT or even guide them with all these installations ... In best case they will just give up after a few attempts ...

You guys better provide some self contained solution, if you want be essential part of the RISCV industry proliferation process. Again, at least user ISA spec tests can be self-checking so community could certify at least non-privileged part of their designs

[MarcKarasek]

If your company (who is it btw??) would like to provide some resources ($$, people, etc..) to help that would be great!

Remember everyone that you are talking with has a day job and is VOLUNTEERING their time on this committee as well as other committees. We do not work for RVI.

Marc

On Sun, Oct 30, 2022 at 1:03 PM algrobman @.***> wrote:

I have a few follow up questions.

1. what is legality of this repo? How a user can be sure that this repo represents official view of RISCV International or riscv.org? I couldn't find any reference of this repo on their web site. Google search brings multiple "compliance tests suits" repos - all with different tests ...
2. How can a user trust the software from this and any referred sites in sense of cyber security? (are all these required installations spyware free? (what if "Russian hackers", disguised as Czechs, had planted a spyware, will steal my company tread secrets 😊 ? We just got a corporate email, warning from installing SW on our computers by ourselves ...

regarding Lee's comment , I'm not sure I can convince our IT or even guide them with all these installations ... In best case they will just give up after a few attempts ...

You guys better provide some self contained solution, if you want be essential part of the RISCV industry proliferation process. Again, at least user ISA spec tests can be self-checking so community could certify at least non-privileged part of their designs

— Reply to this email directly, view it on GitHub https://github.com/riscv-non-isa/riscv-arch-test/issues/290#issuecomment-1296302890, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD72ZMZUFMAJKYU5XLZ2VIDWF2S5RANCNFSM6AAAAAARNNRAIA . You are receiving this because you commented.Message ID: @.***>

-- /**** Marc Karasek ASM T648 Herder of Cats ****/

[MarcKarasek]

That is precisely the issue that the repo I put together addresses. It spels out exactly what needs to be installed from a sudo point of view and what you can install in your local home directory, under ~/.local/bin/... What needs to be installed via sudo is a very small set of rpms. sudo yum groupinstall 'Development Tools' -y sudo yum install gmp-devel pkgconfig zlib-devel cmake3 -y

The rest are installed locally to a person's home dir.

There are scripts for both rhel and ubuntu to use to do these installs (or to give to a IT person to install for you).

Look at this repo : https://github.com/InspireSemi/riscof_install_example_DUT/blob/main/install_scripts/rhel/sudo_install_sail_rhel8.sh My company put this together as a give back to the RVI community.

And as I have stated before, would love some help with this. Roll up your sleeves and join in the fun..

On Sun, Oct 30, 2022 at 5:03 AM Lee Moore @.***> wrote:

I think @algrobman https://github.com/algrobman has underlined a real problem here. I work with a lot of corporate CAD/IT departments, It is always difficult getting corporate wide installations through these groups. In my experience none of these groups I have worked with would be prepared to go through the trial/error process we are seeing unfold here.

Either everything in the flow needs to be reproducible with user level permissions, OR, a well defined description of what needs to be installed- prior to installing the tools required to use this flow.

It seems implausible to me, that an approach of trial/error would be acceptable to a corporate CAD group

I am truly sorry if this sounds difficult to hear by those who have architected this system, I don’t underestimate the amount of effort expended, but I believe we have to deal with the world (and corporate CAD/IT groups!) as it is, not how we would like it to be.

— Reply to this email directly, view it on GitHub https://github.com/riscv-non-isa/riscv-arch-test/issues/290#issuecomment-1296176759, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD72ZM4GJFAKNOVT5QOVZP3WFYTVLANCNFSM6AAAAAARNNRAIA . You are receiving this because you commented.Message ID: @.***>

-- /**** Marc Karasek ASM T648 Herder of Cats ****/

So far I do not believe anyone has actually looked at this repo.

On Sun, Oct 30, 2022 at 5:03 AM Lee Moore @.***> wrote:

I think @algrobman https://github.com/algrobman has underlined a real problem here. I work with a lot of corporate CAD/IT departments, It is always difficult getting corporate wide installations through these groups. In my experience none of these groups I have worked with would be prepared to go through the trial/error process we are seeing unfold here.

Either everything in the flow needs to be reproducible with user level permissions, OR, a well defined description of what needs to be installed- prior to installing the tools required to use this flow.

It seems implausible to me, that an approach of trial/error would be acceptable to a corporate CAD group

I am truly sorry if this sounds difficult to hear by those who have architected this system, I don’t underestimate the amount of effort expended, but I believe we have to deal with the world (and corporate CAD/IT groups!) as it is, not how we would like it to be.

— Reply to this email directly, view it on GitHub https://github.com/riscv-non-isa/riscv-arch-test/issues/290#issuecomment-1296176759, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD72ZM4GJFAKNOVT5QOVZP3WFYTVLANCNFSM6AAAAAARNNRAIA . You are receiving this because you commented.Message ID: @.***>

-- /**** Marc Karasek ASM T648 Herder of Cats ****/

[algrobman]

Marc, thanks for all your help .

[allenjbaum]

Pointers to all these repos can all be found in https://wiki.riscv.org/display/HOME/GitHub+Repo+Map. All repos pointed to from that page are owned by RISC-V International. If you want me to get a pointer to these on some other page, I can get that done; it's not a big deal, and should not be a basis of making a reasonable decision.

Regarding security: how does your company know if GCC is spyware free? Haven't you heard of supply chain poisoning? How does it know your verilog is spyware free? They either accept that they are or they don't, and if they don't , then you can't use it. And if you can't use it, then you don't necessarily get to use the RISC-V trademarks. What it sounds like you are saying is that you can only trust software that you've paid for. Except, of course, Linux and all the tools associated with it. Oops.

I think if this is a problem, you should have your IT department get in touch with us directly.

But. you're worried about hackers and security. How can we accept that your cycle-by-cycle dump isn't faked? You didn't use our tools and environment, so who knows what it is really doing. More to the point, RISC-V doesn't, whereas if you used our tools, then we would.

We are working on making this a lot friendlier - but, as I have stated, and Marc has stated, this is a volunteer effort; we don't get paid for doing this work, and RISC-V not a for-profit enterprise. If you wish to become part of the solution, keep finding issues, and we'll try to keep up fixing them. I don't know if the reference-signature-as-a-service will completely make this user mode only or not, but it should remove a major pain point.

And if you want to submit your cycle-by-cycle comparison dump and apply for a waiver, give it a try. The RISC-V TSC will vote on whether to accept it or not, No guarantees - it might be a precedent too far.

On Sun, Oct 30, 2022 at 10:03 AM algrobman @.***> wrote:

I have a few follow up questions.

1. what is legality of this repo? How a user can be sure that this repo represents official view of RISCV International or riscv.org? I couldn't find any reference of this repo on their web site. Google search brings multiple "compliance tests suits" repos - all with different tests ...
2. How can a user trust the software from this and any referred sites in sense of cyber security? (are all these required installations spyware free? (what if "Russian hackers", disguised as Czechs, had planted a spyware, will steal my company tread secrets 😊 ? We just got a corporate email, warning from installing SW on our computers by ourselves ...

regarding Lee's comment , I'm not sure I can convince our IT or even guide them with all these installations ... In best case they will just give up after a few attempts ...

You guys better provide some self contained solution, if you want be essential part of the RISCV industry proliferation process. Again, at least user ISA spec tests can be self-checking so community could certify at least non-privileged part of their designs

— Reply to this email directly, view it on GitHub https://github.com/riscv-non-isa/riscv-arch-test/issues/290#issuecomment-1296302890, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXVJR7WUJXTNU55BCFLPDWF2S5TANCNFSM6AAAAAARNNRAIA . You are receiving this because you were mentioned.Message ID: @.***>

[algrobman]

Thanks guys again. I'd figured out how to run your tests in self-checking mode - appears the run instruction macros have predicted value. With custom "plugin" macro I could make the test "fail" if there is a result mismatch.

[pawks]

Thanks guys again. I'd figured out how to run your tests in self-checking mode - appears the run instruction macros have predicted value. With custom "plugin" macro I could make the test "fail" if there is a result mismatch.

Those values are just place holders and not to be relied upon, examples - (link1), (link2).

[neelgala]

@algrobman can this issue be closed?

[algrobman]

yes, I was able to get what I needed, at least for "integer" tests (have no need in F/D tests for now), but will really appreciate if you could "return" self-checking capability to the suite ( may be excluding CSRs issues)

[allenjbaum]

We have never had self checking tests. We have tests that put expected results in comments, but that isn't my definition of self checking. And that doesn't always work because too many instructions have results dependent on microarchitecture, or the exact unnamed architectural options that are implemented (load and store ops are the poster child for that). We will have some self checking tests in extraordinary circumstances (e.g. when we have results that are restricted non-deterministic), but that support hasn't been implement yet (it's a bit tricky) Because there are many tests that won't have a single unique expected result, we instead get it from running a reference model configured with the same unnamed architectural options as your DUT. While there are tests that do have deterministic results, but we are not going to implement two different ways to do the compatibility checks.