Pinpoint 1.8.2 - MongoDB calls not intercepted when using 2.x Driver classes

[asgs]

1. Created a sample application that connects to a local MongoDB instance, queries a collection, loops through the cursor and print each document.
2. Run the application by passing the options below. -javaagent:/home/asgs/pinpoint-agent/pinpoint-bootstrap-1.8.2.jar -Dpinpoint.agentId=gowri-local-test1 -Dpinpoint.applicationName=gowri-local-test1

What did you do to trigger the bug?

We're not able to see MongoDB component on the remote Pinpoint Web server.

Expected behavior

We should be able to see the MongoDB as one of the external components that the application running the agent is connecting to

Logs

gowri-local-test1-pinpoint.log.zip

Additional context

Tried running the application with both mongodb-driver and mongo-java-driver (versions 3.5.0 and other versions below 3.9.0)

[RoySRose]

Hello, @asgs @ychandu

I've created the plugin since after 3.0 has released. Since, Mongo teams recommendation on accessing Database and Collection was through below classes

import com.mongodb.client.MongoCollection
import com.mongodb.client.MongoDatabase

I didn't instrument the ones used in 2.x

com.mongodb.DB
com.mongodb.DBCollection

So, if you are using DB and DBCollections classes to access DB and the Collections, it won't be traced. (I'm not sure I can work on this at the time being)

Two solutions. You can you the APIs as Mongo Team recommended. Or little modification to Mongodb plugin (add DB,DBCollection to the plugin)

I'd love to guide your way if you are willing to upgrade the plugin and contribute :)

[asgs]

@RoySRose you're right, tracing works when the 3.x classes are used. Let us know how we could update the plugin to trace the 2.x classes as well.

[RoySRose]

@asgs

I've looked into the library of Mongo-driver to check classes

com.mongodb.DB
com.mongodb.DBCollection

There are 2 parts that you need to instrument.

1. Connection part (getting MongoClient, DB, Collection)
2. Session part (CRUD) You can assume what the classes are doing in Pinpoint-Mongodb-Plugin.

The bad news is, I thought it would need LITTLE modification. I thought only 1. Connection part needs to be supplemented. But after getting into the details. I found out the name of methods that manage CRUD is different from mongo-java-driver 3.x. So, 2. Session part (CRUD) also need to be supplemented, too.

If you still are willing to proceed on this. Please check out the codes in Pinpoint-Mongodb-Plugin. and let me know if you have further questions.

Plugin dev guide may also help you if you got stuck on the way.

[asgs]

@RoySRose we're trying to make some changes related to the Connection Part and rebuild the plugin to test in the local environment. But running mvn -e clean package from the root of the pinpoint repository provided warnings on missing Environment variables for Java 6, 7, 8, and 9. Since only Java 8 is available, all four environment variables were set to the same Java 8 path. After rerunning the mvn command, this warning is now gone. But a compilation error is seen. It's not clear what the compilation error is or how to debug it further.

Also, if there is a way to just build the mongodb plugin, it would be much easier

[asgs]

Ignore the above, please. The environment variable set was incorrect.

Will let you know about the progress further, @RoySRose

[RoySRose]

@asgs

Also, if there is a way to just build the mongodb plugin, it would be much easier

You can build mongo plugin module and after that pinpoint-agent module

[asgs]

@RoySRose we've made some changes to the MongoDB plugin to trace the 2.x version of the classes you mentioned above. It appears to work, but with the following issues.

1. The MongoDB component shows up empty in the main view. In the timeline view, it's listed as UNKNOWN_DB
2. The name of the MongoDB database shows as unknown, too.

Thankfully, the call tree does show "MONGO-JSON" for now, so we could get the JSON query and the bind var values.

Please review the change and let us know what is missing here

[RoySRose]

Oh~:) great.

Still, there are things to take care of. But I think you're on the right track! First. The reason for the unknowns. You would probably need to add

• driverGetDatabaseInterceptor
• driverGetCollectionInterceptor

it's not separated for 3.0 and 3.7 is because they use identical data structure of storing database and collection.

and modify

• MongoDriverConnectInterceptor2_X.java

check getting database part. It should be different from 3.x.

R session and cud session interceptors... I'm didn't look into it in details. But for now, I think you can pass.

Last. WriteContext should be modified for showing json query. This needs to be modified to show with more details.

I think you got here very fast :) Hope to hear more from you~

[asgs]

@RoySRose after instrumenting the right method to get the DB instance from the Mongo class, the Unknowns have now resolved appropriately.

I'm not sure what you mean by

WriteContext needs to be modified to show with more details

We're able to see the Mongo JSON along with the bind var values like below.

Are there more details that could be obtained?

[RoySRose]

Hello, @asgs

Wow, You've taken care of it very fast!!

I've fully looked into mongo-driver library, to answer your questions. TLTR; modification of WriteContext is not necessary for the sample application.

Since you're a fast learner and looks like you are gonna understand this very quickly. I'll tell you why I've thought WriteContext needs to be modified.

I saw your sample application uses DBObject as below.

DBObject query = new BasicDBObject("zzz", "aaa");

I didn't implement any code to take care of DBObject interface in

• WriteContext.parse()
• WriteContext.writeValue() <-- needs of modification differs by cases So, at first, I thought JSON wouldn't be properly shown in Pinpoint-Web But, since BasicDBObject implements Bson interface. It was fine.

If there are any implementations of DBObject which doesn't implement Bson. It will not be shown properly in the future. (I don't know if these kind of implementation exists in 3.x library)

By the way, what do you think about contributing your work?

I'll be happy to review your code and help the contribution

[asgs]

@RoySRose Closed PR https://github.com/naver/pinpoint/pull/5429 and created another PR https://github.com/naver/pinpoint/pull/5435

We're unable to verify this change on the master (1.9.x) as our Pinpoint collector is currently running on 1.8.2 (Docker version). The MongoDB doesn't show up even with the test app mentioned above.

Is there a Docker version of 1.9.x available?

[asgs]

@RoySRose any update on the question above?

[RoySRose]

@asgs

About docker, I don't create the docker images since we don't have any snapshot jar available. 1.9.x docker image will be created after the release of 1.9.0.

About the compatibility. Officially, 1.8.x collector is not compatible with 1.9.x agent. But it may work since there aren't any significant changes in the communication between them.

To get the docker image of 1.9.x(collector). You can always create the image by yourself. Or if you don't have much time. You can wait for the release of 1.9.0

[asgs]

@RoySRose

1. When is the 1.9.0 planned for release?
2. Is there a build guide on Dockerization of Pinpoint?

[RoySRose]

@asgs

1. There isn't an exact plan yet.
2. There aren't any guide on dockerization on Pinpoint But you can look at the example from here https://github.com/naver/pinpoint-docker

[asgs]

@RoySRose the supported tags on that git repo lists versions upto 1.8.3. will building from master be equivalent to 1.9.0-SS?

[RoySRose]

@asgs

I'm not sure what you're talking about, supported tags on docker repo. is built from the release of Pinpoint.

There isn't any open docker image built from 1.9.0-SS. You need to make one if you need an image built from 1.9.0-SS. for now.

[asgs]

@RoySRose got it. the term "supported tags" seems to be a bit misleading

[RoySRose]

@asgs

Really?! Never thought it would be confusing. How did you understand the term. It may help me change the term

[asgs]

@RoySRose by "supported tags", I was assuming it meant that the master branch is the same as the last created tag (1.8.3) and 1.9.x is still WIP.

I'd prefer a term like "Released tags"

[RoySRose]

Oh. master branch is WIP.(we don't have 1.9.x branch yet, it will be created after master becomes 1.10SS) This may differ from open source projects to projects.