How to get current mouse position on surface viewer?

[dzhi1993]

Hi,

I'm currently reprogramming the viewer for our version. Basically, I reallocate the surface viewer (lowerImageBot2) to make it bigger as shown in the figure. The right cerebullem flatmap is actually the lowerImageBot2 which is a webgl object. I have reprogrammed it using different shaders to draw in 2D and want to add a "mousedown" event listener when I click this viewer to get the current mouse position. Because I want the crosshair in this webgl can move with my mouse down so that I have to pass this value to render the scene. I know this webgl on the canvas is inherited from viewer class and there are some mouse event listeners are defined. Can you please specify which variables should I look at? Thank you in advance!

Best

[martinezmj-ims]

Nice work. There's one set of mouse listeners set up for the entire viewer. When you mouse down, it first detects which slice the mouse was in:

papaya.viewer.Viewer.prototype.findClickedSlice()

... which becomes selectedSlice in the viewer.

From there, it converts the screen coordinates to image coordinates using:

papaya.viewer.Viewer.prototype.convertScreenToImageCoordinate(xLoc, yLoc, screenSlice)

I'm not sure how best to do what you want to accomplish. Maybe set up your own listener and use the above methods to convert to image coordinates? You may also want to look at:

papaya.viewer.Viewer.prototype.updatePosition()

papaya.viewer.Viewer.prototype.mouseDownEvent()
papaya.viewer.Viewer.prototype.mouseMoveEvent()
papaya.viewer.Viewer.prototype.mouseUpEvent()

[dzhi1993]

Thank you, sir. I tried to add own event listener but it doesn't work. I noticed that there are two variables called this.contextMenuMousePositionX and this.contextMenuMousePositionY in papaya.viewer.Viewer.prototype.mouseDownEvent().

Are they the current mouse position coordinate on canvas when mouse down and drag? I think this pair of xy coordinate is what I want if I'm right, then I can calculate the current mouse position related to webgl by using surface's offsets.

[martinezmj-ims]

contextMenuMousePositionX/Y would be appropriate to use if you need to read a pixel in the webgl canvas. They are the coordinate in the webgl canvas, not the coordinate in image volume space.

[dzhi1993]

Thanks for the help, and now it works. The crosshair is rendered at the correct mouse position when mousedown and drag event. But there is a further question, how to synchronize this mouse position in webgl with the other three viewers.

So far, the 3D coordinates in the left three viewers can be correctly transferred to webgl coordinates by calling papaya.viewer.Viewer.prototype.getCurrentValueAt(), because my .nii file stores index information for each of the 3D coords. Then I have a mapping from this index to the vertices coords in webgl so that the webgl crosshair is synchronized with the crosshairs of left three viewers.

Now, I want to achieve the same thing on the other hand, from webgl to axial/coronal/sagittal slice. Is there any function I can use to get the currentCoord in the three viewers when I mouse clicked on the webgl canvas?

Thank you

[martinezmj-ims]

I'm not sure it will work with your model, but this is the closest thing to what you're looking for that already in Papaya:

papaya.viewer.ScreenSurface.prototype.findPickedCoordinate()

You might also find it useful to follow this flow:

papaya.viewer.Viewer.prototype.updateCursorPosition()
papaya.viewer.ScreenSurface.prototype.pick()
... draws ...
papaya.viewer.ScreenSurface.prototype.findPickedCoordinate()

[dzhi1993]

Thank you. I think I'm doing this in a stupid way, which I made a mapping of pixel to 3D volumes coords, and pass this coords in screensurface to viewer's coords like this.viewer.currentCoord.x = x;. It works not very properly since the three viewers only synchronized when mouse click but not for dragging and missing the crosshairs shown. Since our web release due date is approaching, we think it's ok for the short term. But for the long term, I still want to know what the best way to achieve this extra cerebellum flatmap viewer that could synchronize with the other three viewers perfectly. Maybe I shouldn't use webgl to render this and just make it as a canvas? Because we want to dynamically change the overlays of this flatmap and It could be very time-consuming if we re-rendering the whole flatmap everytime we change the overlay contrasts.

Another question is about the CSS part that I'm modifying the menu bars. I want to add a sub-menu for the label in File menu so that I've added a list of items for the label. But when I click this label MDTB contrast, the sub-menu is aligned to the right of File button, as shown below. How can I change this sub-menu position that makes it aligned to the left like its parent menu, and make it has a scroll bar? We know little about web development and sorry for the frequent asking, hope you don't mind. Thank you.

[martinezmj-ims]

On the CSS question, add "icons": null. That should make the menu align correctly. For example:

{"label": "MDTB", "action": "something", "icons": null, "items": [

I'm not sure I can be much help with the other issue. It sounds like an interesting extension you are trying to build. I don't think there's much difference between doing it in webgl vs canvas -- and it doesn't sound like a huge leap to get it work with mouse dragging, if you've already got it working with click.

If you can come up with more specific questions on where you're hitting snags, I'll do what I can to help.

[dzhi1993]

Hi Michael, thanks for your help! Now, what if I want to directly load a specific .nii file by click one condition in this submenu? My code is like this

{"label": "MDTB contrasts - 47 conditions", "icons": null, "items": [
                {"label": "Cond01_No-Go", "action": "Open-data/onlineAtlas/Cond01_No-Go.nii"},
                {"label": "Cond02_Go", "action": "Open-data/onlineAtlas/Cond02_Go.nii"},
                ...
                {"label": "Cond47_Response_Alternatives_Hard", "action": "Open-data/onlineAtlas/Cond47_Response_Alternatives_Hard.nii"}
        ]},

due to this code block in doAction function

} else if (action.startsWith("Open-")) {
            imageName = action.substring(action.indexOf("-") + 1);
            this.viewer.loadImage(imageName);

All my .nii files are stored in ../tests/data/onlineAtlas/ folder. But after doing so, an error returns

Uncaught Error: Syntax error, unrecognized expression: #Open-data/onlineAtlas/Cond01_No-Go.nii0

Could you give me any clue? Thank you!

[dzhi1993]

Hi Michael, please ignore my last comment. I've solved the previous question by adding a new action, like this

{"label": "MDTB contrasts - 47 conditions", "icons": null, "items": [
                {"label": "Cond01_No-Go", "action": "OpenBoth-Cond01_No-Go"},
                {"label": "Cond02_Go", "action": "OpenBoth-Cond02_Go"},
                ...

and this action OpenBoth- will load both nifti volume and gifti surface at same time, by this

else if (action.startsWith("OpenBoth-")) {
            if (this.container.viewer.screenVolumes.length > 2) {
                this.container.viewer.removeOverlay(2); // Always remove the previous one, index = 2
            }
            imageName = action.substring(action.indexOf("-") + 1);
            let NiifileName = ["data/onlineAtlas/" + imageName + ".nii"];
            let GiifileName = "data/onlineAtlas/" + imageName + ".func.gii";
            this.viewer.loadImage(NiifileName, true, false, false);
            this.viewer.loadSurface(GiifileName, true, false);

It always keep two overlays on the current viewer, bu the surfaces are keep appending by each click of the contrasts. Do you have similiar function for removing previous loaded surfaces just like papaya.viewer.Viewer.prototype.removeOverlay did for the overlays?

Another question comes up is how to use papaya.viewer.ColorTable? By doing above, my func.gii has been loaded (I've changed the source code to make it func.gii support) and pass the value to surface.colorsData. However, it just the cdata from the func.gii (In my case is a float32array with length 28,935 ranged around [-2, 2], these values are the color information of that 28,935 vertices) and I want to convert this value to the RGB color value for my webgl rendering. Can papaya.viewer.ColorTable makes it? Thank you!

Best

[kapilvirasat]

Hello Developers,

I need to get the value of overlay on mouse button click. In the Papaya viewer visual get 1 on overlay click and 0 outside overlay.

Please can you suggest me how can we get this value.

[kapilvirasat]

Hello @dzhi1993

Can you please guide me how to proceed. I need to get the co-ordinates and overlay on clicking the Papaya image screen.

My Code:

<!DOCTYPE html>

<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>

        <!-- iOS meta tags -->
        <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no"/>
        <meta name="apple-mobile-web-app-capable" content="yes">
        <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">

        <link rel="stylesheet" type="text/css" href="papaya.css?build=1449" />

        <!-- dependencies -->
        <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
        <script type="text/javascript" src="papaya.js?build=1449"></script>
        <script type="text/javascript" src="daikon.js"></script>

        <title>Medical Image Viewer</title>

        <script type="text/javascript">

            // Images
            let currentImg = "<?php echo $currenturl;?>";
            let overlayImg = "<?php echo $overlayurl;?>";

            // // handeling click events
            // $(document).ready(function(){
            //     $(".papaya-main-swap").click(function(){
            //             papayaContainers[0].viewer.prototype.mouseDownEvent();    
            //             // papayaContainers[0].viewer.rotateViews(); // working rotation
            //             // papaya.viewer.Viewer.prototype.rotateViews();
            //     });
            // });

            // medical case
            var params = [];
            params["worldSpace"] = true;
            params["images"] = [currentImg, overlayImg];
            params[overlayImg] = {"min": 4, "max": 10};
            papaya.Container.syncViewers = true;
            params[currentImg, overlayImg] = {"min": 4, "max": 8, "lut": "Red Overlay"};
            params["showOrientation"] = true;
            // params["surfaces"] = ["data/mySurface.surf.gii"];
            // params["coordinate"] = [0, 0, 0];
            // params["rotation"] = [0,0,45];

            // ============ Testing ============
            // let a = papaya.viewer.Viewer.prototype.convertScreenToImageCoordinate(xLoc, yLoc, screenSlice)

            // let b = this.contextMenuMousePositionX;
            // let b = this.viewer.currentCoord.x = x

            let a = papaya.viewer.Viewer.prototype.findClickedSlice();  // not working
            alert(a);
   </script>
</head>

    <body>
       <div style="width:60%; margin-left: auto; margin-right: auto; height: auto">
            <button id="swap-slice" class="papaya-main-swap" type="button" style="display: block; top: 475px; left: 700px; position: relative;">Test Button</button>
            <div class="papaya" data-params="params"><b>Medical Image:</b></div><br><br><br>
            <!-- <div class="papaya" data-params="params2"><b>Overlay Image:</b></div> -->
       </div>
    </body>
</html>

How to include below codes and get co-ordinates and overlay.

papaya.viewer.Viewer.prototype.updateCursorPosition()
papaya.viewer.ScreenSurface.prototype.pick()
... draws ...
papaya.viewer.ScreenSurface.prototype.findPickedCoordinate()      // need cordinate and over true/false

I need to click on Papaya image viewer image and get the co-ordinate/sliced image of overlay. Please suggest me.