Inclusion of Eclipse MAT framework components question

jasonk000 commented 4 years ago

Have you explored alternative packaging methods , or patching methods , for MAT & JIFA?

Currently, in order to build MAT and JIFA from source, I need to maintain a fork of Eclipse MAT. I noticed that most of the changes to MAT core are to allow use of MAT outside of Eclipse Platform.

I think maybe there are two things we could do:

Upstream the changes to MAT so that the code can be used from source?
Build JIFA closer to the Eclipse runtime framework (maybe build it as an Eclipse Application) so it gets access to Eclipse Platform services and the MAT code can be loaded natively?

y1yang0 commented 4 years ago

Good proposal. Comparing to use a fork of Eclipse MAT, we'd better do some modification of JIFA so that it can be directly built with upstream Eclipse MAT. We are investigating the second approach.

jasonk000 commented 3 years ago

Looking for OSGI help on this ticket!

After some discussion on Eclipse MAT list it looks like the right solution is to modify JIFA to allow it to import MAT directly. I have started working with goomph and bnd-platform plugins to configure the worker to run standalone OR as an Eclipse plugin.

To experiment with the branch

$ # git clone and checkout
$ mkdir -p backend/worker/build/headlessAppSetup/configuration/
$ cp config.ini backend/worker/build/headlessAppSetup/configuration/
$ rm -rf backend/worker/build/headlessAppSetup/plugins/*.jar
$ ./gradlew bundles && ./gradlew updateSite && ./gradlew launchInEquinox

It compiles and boots but does not quite work .. curl connects and dies

jkoch@jkoch-desktop:~/code/eclipse-jifa$ curl -vvvv http://localhost:8102/
*   Trying 127.0.0.1:8102...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8102 (#0)
> GET / HTTP/1.1
> Host: localhost:8102
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server

If we again run with:

$ rm -rf backend/worker/build/headlessAppSetup/plugins/*.jar
$ ./gradlew launchInEquinox  | grep -a50 -b50 -i io.netty.handler.codec
....
4169553:Caused by: java.lang.ClassNotFoundException: io.netty.handler.codec.ByteToMessageDecoder cannot be found by io.netty.codec-http_4.1.49.Final
....

Which seems to indicate that netty-codec-http cannot see io.netty.handler.codec, but I have checked the OSGi manifests and it all seems OK - so I think the issue is perhaps related to the vertx verticle classloader trying to read io.netty.handler.codec but not having the required osgi import-package.

Any extra help would be appreciated, I am not familiar with OSGI.

jasonk000 commented 3 years ago

Some extra comments based on the changes I have made so far:

The Eclipse MAT parser works ok, and can process files. So, we can be sure that the Eclipse MAT core is being loaded in and should work OK.
There are some areas where JIFA uses reflection to access internal properties in MAT, which would be rejected by OSGI.
I did not set up SSHj but I think it should be OK
We might consider moving some of the custom query objects into separate JIFA module
We might consider moving the Eclipse wrapper to a separate OSGI module to keep the worker clean

D-D-H commented 3 years ago

When I ran

./gradlew bundles && ./gradlew updateSite && ./gradlew launchInEquinox

got the following error

Caused by: java.lang.ClassNotFoundException: org.eclipse.mat.query.IResultTree cannot be found by org.eclipse.mat.api_1.11.0.202012021418

D-D-H commented 3 years ago

When I ran

./gradlew bundles && ./gradlew updateSite && ./gradlew launchInEquinox

got the following error

Caused by: java.lang.ClassNotFoundException: org.eclipse.mat.query.IResultTree cannot be found by org.eclipse.mat.api_1.11.0.202012021418

But sometimes it doesn't happen :(

D-D-H commented 3 years ago

@jasonk000 @kelthuzadx I push my solution to https://github.com/D-D-H/jifa/tree/integrate-mat

run with

./gradlew backend:worker:run

Looking forward to your suggestion :)

jasonk000 commented 3 years ago

@D-D-H this looks great. Much cleaner separation than my implementation, and sidestep Eclipse framework difficulty.

I have a couple of thoughts:

I really like the separation of mat-deps folder. It looks like I can use this to (1) download default JARs during build and, (2) at deployment time, provide my own JARs with some customization, do not need to wait for quarterly release from upstream. This is really great!
We could use p2asmaven plugin to perform the download of JAR files for the default case, which makes it easy to ensure version matches with MAT and uses the Eclipse P2 sites. Javadoc for the plugin describes how to use it and an example here.

I had a couple of small tweaks here and here

Some question, on how I think I would use this. Could you let me know your thoughts?

1) It seems we can point this at any Eclipse MAT installation by use of the -Dmat-deps={path} property, is that correct? This would be perfect. I can do a local install of MAT build and configure JIFA to use it. Are there any constraints?

2) In the case that we build custom Eclipse MAT w/ patches before JIFA that requires an API change in MAT (for example, new query implementations), and we can include these JARs at build time and use -xdownloadDepsOfMAT in gradle build.

An alternative would be that the local MAT build process produces a local P2 repository, and we could provide this classpath or a property to configure the P2 location during builds.

D-D-H commented 3 years ago

@D-D-H this looks great. Much cleaner separation than my implementation, and sidestep Eclipse framework difficulty.

I have a couple of thoughts:

I really like the separation of mat-deps folder. It looks like I can use this to (1) download default JARs during build and, (2) at deployment time, provide my own JARs with some customization, do not need to wait for quarterly release from upstream. This is really great!

We could use p2asmaven plugin to perform the download of JAR files for the default case, which makes it easy to ensure version matches with MAT and uses the Eclipse P2 sites. Javadoc for the plugin describes how to use it and an example here.

Good idea, I think so too

I had a couple of small tweaks here and here

fixed.

Some question, on how I think I would use this. Could you let me know your thoughts?

It seems we can point this at any Eclipse MAT installation by use of the -Dmat-deps={path} property, is that correct? This would be perfect. I can do a local install of MAT build and configure JIFA to use it. Are there any constraints?

yes, use the system property mat-deps to specify the path of all mat dependencies. I am not sure if there are any restrictions, I tested with the two versions of MAT, it seems to work.

In the case that we build custom Eclipse MAT w/ patches before JIFA that requires an API change in MAT (for example, new query implementations), and we can include these JARs at build time and use -xdownloadDepsOfMAT in gradle build.

An alternative would be that the local MAT build process produces a local P2 repository, and we could provide this classpath or a property to configure the P2 location during builds.

Both ways are okay for me. And we also could add another task(e.g. replaceDepsOfMAT) to achieve this purpose. We can discuss it after fully solving the problem of integrating MAT.

D-D-H commented 3 years ago

Because we have other follow-up features to develop, we integrate this patch into the main first. if there is any problem exist in this patch, plz contact me on slack, thanks.

eclipse / jifa

Inclusion of Eclipse MAT framework components question #2