Add JIT for RISC-V

[janvrany]

The aim of this issue is to track progress on implementing a full JIT for RISC-V architecture. The OMR part of the JIT is reasonably stable. For the J9 part of the JIT, the idea is to essentially to follow the path of AArch64 port as it proved successful.

FYI: @ChengJin01 , @0xdaryl

[ChengJin01]

FYI: @DanHeidinga, @tajila, @pshipton, @knn-k, @mstoodle

[janvrany]

As first steps, I mean to:

1. Set up (and document) a development environment, including IDE, to work on J9 JIT. The issue is that there's no CMake support for RISC-V and generally no cross-compilation support for CMake. Some work has been done (#10915, #11054) but I'd probably have to resort to using UMA for compilation and testing and CMake for IDE.

2. Go over all PRs adding AArch64 port to better understand what has been done and in what order.

3. Have an initial kick-off hacking session in couple weeks to get more hands-on experience with the code (as opposed to just read existing code)

[0xdaryl]

Thanks for creating this issue @janvrany! As we discussed last week, I'm happy to meet with you regularly (and open it up to anyone else who wants to participate) to help you understand the code, answer questions, and guide your implementation. If interested, ping me on Slack and we'll set something up and announce it here.

Looking at the list of all arch:aarch64-tagged PRs might be a bit overwhelming and the detail too fine-grained to help you understand the bigger picture of what needs to be done and how it all fits together. Which is why I committed to producing a higher-level roadmap like we followed for AArch64. This issue might be the perfect spot for that.

[pshipton]

@sxa fyi

[knn-k]

Look at the following for the list of PRs that were merged for AArch64 JIT in OpenJ9:

https://github.com/eclipse/openj9/pulls?q=is%3Apr+is%3Amerged+label%3Aarch%3Aaarch64+label%3Acomp%3Ajit+sort%3Acreated-asc

[sxa]

@pshipton Thanks for the tag.

I'd be interested in at least following any progress on this so if you have any calls @0xdaryl I'd appreciate an invite too as I'd love to have a working JIT in OpenJ9 :-)

[DingliZhang]

Hi @janvrany , Thanks for this issue, I want to know about the hacking session you said and I will very honoured to have the opportunity to participate in this session. Is this session a zoom meeting and how can I join this session? Thanks a lot!

[janvrany]

Hi, @zdlgv5

I'm sorry, I did not mean anything big on zoom, merely that I'll lock myself up in the office and play with the code o my own to get better understanding. After that, I hope can have meaningful call with @0xdaryl and anyone interested where we can discuss next steps. Is that OK?

[janvrany]

After a discussion with @0xdaryl, we decided to have RISC-V OpenJ9 JIT meeting on Tuesday, Dec 8 2020, 9:30 AM - 11:30 AM EST: https://ibm.webex.com/ibm/j.php?MTID=mdcdaf7a65f60dbf9038484bda47f6245 Everyone interested is welcome to join this meeting.

Also OpenJ9 slack, channel #risc-v for more interactive conversations. Again, everyone interested is welcome to join!

[cobyforrester]

Hello @janvrany! I am having trouble accessing the slack channel, and it appears locked, is there any way I would be able to join it with the email cobyforrester@gmail.com?

[DingliZhang]

hi @janvrany , Thanks for your reply! May I be able to join the OpenJ9 slack with the email dingli@iscas.ac.cn?

[ChengJin01]

Hi @cobyforrester & @zdlgv5 ,

The #risc-v channel in Eclipse OpenJ9 (Slack) is open for public. Please feel free to join with the e-mail you prefer to offer.

[ChengJin01]

Hi @cobyforrester,

I tried to add you in the channel but ended up with error like "already in the workspace". Could you offer another e-mail address for me to try again?

[mayeths]

Hi @janvrany , I cowork with @zdlgv5 now, may I join the slack with wow@mayeths.com? Thanks!

[janvrany]

@mayeths , @cobyforrester , @zdlgv5:

I'm sorry I do not know how I can add you to that channel. You can join the slack instance by clicking to the link on "Join OpenJ9 slack" on https://www.eclipse.org/openj9/

From there, you should be able to join the channel.

[ChengJin01]

Hi @mayeths, I just added you in the channel.

[mayeths]

Hi @mayeths, I just added you in the channel.

Received invite, thanks @ChengJin01

[DingliZhang]

@mayeths , @cobyforrester , @zdlgv5:

I'm sorry I do not know how I can add you to that channel. You can join the slack instance by clicking to the link on "Join OpenJ9 slack" on https://www.eclipse.org/openj9/

From there, you should be able to join the channel.

I have joined in the channel, thanks @ChengJin01 @janvrany

[janvrany]

Last week I managed to compile openk9-openjdk-11 with openj9 JIT component (libj9jit29.so) using CMake. It is just a skeleton, but it does compile and the VM starts (and segfaults in JIT thread because of lack of private linkage).

It is not much, but at least gives us a defined starting point that compiles.

Some changes to OMR were required, they're reasonably OK so I'll start upstreaming them this and next week.

I'd like to thank to @knn-k , @ChengJin01 and @dnakamura who helped me with this first little step.

[sxa]

@janvrany Great to hear! When you put the changes in can you link to the PRs in this issue too?

[janvrany]

@sxa : sure thing!

[janvrany]

@sxa: first few: https://github.com/eclipse/omr/pull/5698, https://github.com/eclipse/omr/pull/5699, https://github.com/eclipse/omr/pull/5700

[0xdaryl]

FYI… Apologies for the late notice, but I need to push the start time of our discussion tomorrow out 30 minutes to 10am EST. I have a scheduling conflict that unfortunately can’t be changed.

[0xdaryl]

Our next scheduled discussion will be Tuesday Jan 12 @ 7am EST. https://ibm.webex.com/ibm/j.php?MTID=m37dc73c6c1ff8e79c39c18364d66df73 All are welcome.

[janvrany]

For those interested, here you may find my work-in-progress code on RISC-V JIT:

• JDK: https://github.com/janvrany/openj9-openjdk-jdk11/tree/jv/riscv-devel

  git checkout -b jv/riscv-devel https://github.com/janvrany/openj9-openjdk-jdk11 openj9-openjdk-jdk11

• OpenJ9: https://github.com/janvrany/openj9/tree/jv/riscv-devel

  git checkout -b jv/riscv-devel https://github.com/janvrany/openj9 openj9-openjdk-jdk11/openj9

• OMR: https://github.com/janvrany/omr/tree/jv/riscv-devel

  git checkout -b jv/riscv-devel https://github.com/janvrany/omr openj9-openjdk-jdk11/omr

  • Build scripts (not necessary, but shows how to build the JDK from the above): https://github.com/janvrany/openj9-openjdk-jdk11-devscripts/tree/jv/riscv-devel

    git checkout -b jv/riscv-devel https://github.com/janvrany/openj9-openjdk-jdk11-devscripts

The above has enough to JIT-compile a simple method like:

public static meaningOfLife() {
    return 42;
}

It uses CMake to build OpenJ9 and support cross-compilation (though I have tried only cross-compiling from x86_64 Debian 11).

In next weeks, I'll try to upstream all the changes.

[janvrany]

...or slightly more complicated method like:

static double jitDadd(double a, double b) {
    return a + b;
}

[0xdaryl]

We are scheduled to have a RISC-V JIT discussion call tomorrow.  Given @janvrany's focus on upstreaming the changes he has completed so far we don't have anything to discuss from his perspective tomorrow.

However, I am willing to keep the call if anyone wants to discuss anything or ask any questions.  Just reply here...

In the meantime, for those that are interested in contributing to this effort and need more background on general OpenJ9 JIT code generator concepts, there are a couple of videos available (approx 1 hour each) that should help to familiarize you with some of the concepts, terminology, and components you'll need to better understand the code.

https://www.youtube.com/watch?v=PRphJuGg3bs

https://www.youtube.com/watch?v=2r4u0RKPTps

Questions are welcome any time.

[0xdaryl]

No discussion topics or questions have come forward. We’ll cancel this session. I’m happy to schedule a new one when there is interest.

[janvrany]

My idea is to first upstream changes to OMR, bit by bit and then start upstreaming OpenJ9 changes. Curently these are in a queue: https://github.com/eclipse/omr/pull/5740 and https://github.com/eclipse/omr/pull/5741.

[janvrany]

There's one thing I'd welcome a help. As I said, I use CMake to cross-compile OpenJ9 for RISC-V. I did the initial work to make it working (for me), see:

• https://github.com/janvrany/openj9/commit/4e48535a51743e32b48a9b988f8e8054109fea44
• https://github.com/janvrany/openj9/commit/dfbe95b212d7522f30e4dd00bd49278ae8beb4f3
• https://github.com/janvrany/openj9/commit/1141514a1e3e44c5abb8bc2a5cc8dd2a399e2664
• https://github.com/janvrany/openj9-openjdk-jdk11/commit/a2aead6adea2f988933c1f1148ca1abc7aafad4f
• https://github.com/janvrany/openj9-openjdk-jdk11/commit/15ca3628290de022dce80c188bf87488c8c7a9da

I'd appreciate if someone would take these as and inspiration and take care of upstreaming these changes. I do not have knowledge of the build system and all the little details and other platform limitations...

[ChengJin01]

Hi @janvrany, I will discuss this with @0xdaryl, @mstoodle and the VM leaders next week how to deal with all these changes on your side. Theoretically, I will take over all your work from there as I am in charge of this project but some details need be figured out as how to get this working in the scheduled meeting.

[advancedwebdeveloper]

Ping

[janvrany]

Today, last change required by OpenJ9 has been merged into OMR so all is set to start upstreaming initial JIT work to OpenJ9. I'll try to have a look at this next week.

[janvrany]

I have updated https://github.com/janvrany/openj9/tree/jv/riscv-devel to contain up-to-date RISC-V JIT code. This can be compiled using OMR master as all OMR changes have been merged.

Currently waiting for CMake support for RISC-V (cross)compiling to be finished, after that I'll start upstreaming OpenJ9 changes.

[janvrany]

I just started pushing remaining changes that add basic skeleton of JIT to OpenJ9: #12685 I excluded changes to CMake - they can be found in https://github.com/eclipse-openj9/openj9/pull/12685

[janvrany]

Another small PR: #12982

[janvrany]

Another (this time rather big) PR contributing initial skeleton of a JIT component - #15374 - and a couple of smaller ones -#15375 , #15376, #15377

[janvrany]

Another wee update on RISC-V JIT:

I have implemented a new JIT option - cancelCompilationOnAssume - which, when enabled, causes the compiler to give up compilation of method which causes assertion failure. This allowed me to mark not-yet-implemented parts of the JIT with TR_UNIMPLEMENTED() and let the JIT to compile what it can.

After a couple hours with GDB (stack walking gave me especially hard time :-) and fixing few things here and there I'm able to reproducibly run:

java -Xjit,verbose,cancelCompilationOnAssume -version

on a real hardware without crashing and producing plausible output. Indeed, there's whole bunch of methods that failed to compile, but those that were compiled appear to work fine.

I managed to run MinimalTest.java as:

./jdk/bin/java '-Xjit:verbose,cancelCompilationOnAssume' -cp /tmp/jdk/test/classes:/tmp/jdk/test/libs/junit.jar MinimalTest

But that finishes without crash only in GDB - when run on real hardware it crashes randomly in compilation thread. This needs to be investidated.

While this is still far from being usable, for me it is a major milestone since now I have real system (for some definition of "real") with GC kicking and so on that runs compiled code (as opposed to compiling only carefully crafted tests).

It is also easy to see what is the most frequent reason for compilation to fail and therefore what it the most important bit to fix next. Now it looks it's JNI calls.

As always, the code available in my branches:

• https://github.com/janvrany/openj9-openjdk-jdk11/tree/jv/riscv-devel
• https://github.com/janvrany/openj9/tree/jv/riscv-devel
• https://github.com/janvrany/omr/tree/jv/riscv-devel
• https://github.com/janvrany/openj9-openjdk-jdk11-devscripts/tree/jv/riscv-devel

And, as always, all this would not be possible without numerous core OpenJ9 devs who patiently answered all my questions. Thanks!

[sxa]

@janvrany Are there any plans to make this available for JDK17 yet or is all the work still being targetted at JDK11? (Obviously with OpenJ9 most of the JIT changes are a lot easier to make work across multiple versions!)

[janvrany]

@sxa There are no specific plans for JDK17, nor there's any reason to target JDK11 other than it was the natural choice when I started. Since then I stuck to it as to not having too much moving targets (there's quite a lot of out-of-tree code already since PR review process stalled a bit - for understandable reasons! - and keeping up with OMR and OpenJ9 is enough already :-)

That being said, If you're interested in JDK17 I'm all for giving it a stab together some time second half of January. If you're up to it, ping me in January : - )

But, keep expectations (very) low: right now it bails out to interpreter even for simple things like invokeinterface or instanceof.

[janvrany]

when run on real hardware it crashes randomly in compilation thread. This needs to be investidated.

I've hunted down few bugs and now it runs just fine - it does not crash, but still there's too much missing to make it practical.

Code in my fork is updated.