Audit GSV Explorer Window Should Scale Larger Automatically Based on Windows Size

[jonfroehlich]

I've long wanted the GSV audit interface to scale automatically based on the open browser window's size (though with a minimum and maximum size set). And now some users are asking for this:

@kotarohara dismissed this as too much work, especially because of the algorithm for placing labels on interface (& other things, which may be hard-coded positionally like stuff in the tutorial). @kotarohara, can you provide more details on why you didn't want to do this?

So, rather than solving this with dev time, I always just use the web browser's built-in zoom feature to zoom the window in.

For example, with no browser zoom:

With browser zoom to 150%:

We could add this solution (with directions and maybe an animated gif or video) to the FAQ. Alternatively, we could actually try to update the code to resize with window size.

[jonfroehlich]

Aha, looks like I already entered in something about this previously: https://github.com/ProjectSidewalk/SidewalkWebpage/issues/178

[kotarohara]

I don't think that's worth implementing because (i) it's hard and (ii) you can zoom using the browser feature Jon mentioned above.

The label placement/tracking uses some constants (alpha_x and alpha_y) tied to the Street View pane size and aspect ratio. This tracking feature is a hack and not a documented feature in the Street View API. So I don't know how these constants are determined, which means I can't adjust these constants based on the users' browser settings. There must be equations to determine these constants, but I don't think it's worth looking into.

[jonfroehlich]

Thanks @kotarohara for the details on this. It does sound like this is also a candidate for the FAQ rather than an actual technical fix. We could have a nice animated gif (or video) that shows off my solution.

[jonfroehlich]

We've discussed this as a team and based on our user testing, we now think this is important. Put simply, the size of the GSV window is important and greatly affects usability. A small GSV windows makes it much harder to audit.

[jonfroehlich]

@manaswis had to rewrite some of this code, I believe, due to changes in Google Street View (maybe ~10 months ago). Thus, I wonder if @kotarohara's classification of 'hard' is still relevant. I think auto-resizing to fit the user's browser is still highly preferable to having them use the built-in zoom function on browser. In addition, to simplify, we could maintain the same aspect ratio of the GSV pane size--just increase everything proportionally.

[jonfroehlich]

I just spoke to @tongning about this and he's going to consider it as a post-Fall'18 semester task once his finals are done. I think it may also be useful for us to post screenshots of what the Project Sidewalk interface looks like on our laptop screens since most people these days are using laptops. I'm an anomaly that mostly uses a desktop with large, 4k screens and thus, the labeling interface looks super duper small.

[misaugstad]

This is what it looks on my laptop without zooming in, which is pretty sad considering it has a 13-inch screen :sweat_smile:

[manaswisaha]

Looks fine on my 13-inch screen:

Looks like the zoom% is <100 for your browser screen. Is it?

@manaswis: Can you add in the resolution of your screen and which laptop you have?

[jonfroehlich]

Good catch Manaswi. Looks like you’re right about Mikey’s screenshot. I’ll post mine soon.

Sent from my iPhone

On Nov 29, 2018, at 5:21 PM, Manaswi Saha notifications@github.com wrote:

Looks fine on my 13-inch screen:

Looks like the zoom% is <100 for your browser screen. Is it?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[misaugstad]

I'm at 100%

[jonfroehlich]

Here's my MacBook Pro:

[jonfroehlich]

Here's my Windows machine at work with a 3840 x 2160 Resolution but the OS is scaling the display 150% (see second screenshot below)

[aileenzeng]

Here's my laptop (1280 x 800) at 100% zoom.

[jonfroehlich]

@aileenzeng, that makes sense because @kotarohara originally designed Project Sidewalk for laptops of that screen size I believe (because this was his screen resolution at the time!).

[misaugstad]

Oh yeah my resolution is 1920x1080 that'd do it :grin:

[tongning]

Thanks everyone for the screenshots! Definitely looks important to address this.

[jonfroehlich]

Here's what it looks like on my Microsoft Surface Book 2:

While the resolution of the Surface Book is super high (3,000 x 2,000), by default Windows scales everything by 200%.

If I turn this scaling off then the Project Sidewalk webpage looks like the following (hahaha):

[jonfroehlich]

Hey @tongning, any updates on this? Also, make sure you check out Sage's post: https://github.com/ProjectSidewalk/SidewalkWebpage/issues/794

[tongning]

I've done some investigation and unfortunately it looks like the issues that @kotarohara pointed out still exist (https://github.com/ProjectSidewalk/SidewalkWebpage/issues/303#issuecomment-245344290). I've tested a variety of methods to scale up the auditing interface proportionally (eg. 1, 2, 3), but all of them cause labels to be dropped onto the wrong locations. Even simulating a full-page browser zoom by upscaling the whole page causes the same problem.

My feeling so far is that there isn't an easy surface-level fix for this, so I'm currently looking into whether the math for the label positioning can be updated to support scaling. Open to hearing suggestions!

[jonfroehlich]

Thanks for the update Anthony!

@kotarohara said (https://github.com/ProjectSidewalk/SidewalkWebpage/issues/303#issuecomment-245344290):

The label placement/tracking uses some constants (alpha_x and alpha_y) tied to the Street View pane size and aspect ratio. This tracking feature is a hack and not a documented feature in the Street View API. So I don't know how these constants are determined, which means I can't adjust these constants based on the users' browser settings. There must be equations to determine these constants...

Hrm, how did we come up with the constants in the first place?

In terms of ideas, do we use the official Google Street View API for overlaying labels (link)? If not, why not? Wouldn't this solve the problem as the GSV engine itself would then handle visualizing the markers in the right place... (And the higher level point here is maybe we need to fundamentally question how we are visualizing labels in the first place and attaching them to GSV).

Also, here are some other potential links to investigate:

• http://projects.teammaps.com/projects/streetviewoverlay/streetviewoverlay.htm
• Example GSV app that shows different types of visual markers seemingly attached to the GSV world: https://www.youtube.com/watch?v=EHKEvdZnJH0

[tongning]

Per offline recommendation from @jonfroehlich I am looking into the possibility of replacing the label attachment code, which seems to be custom, with the PanoMarker utility that @aileenzeng used for the admin panel in PR #1308. So far it looks promising, here is the before/after of using PanoMarker:

Before

Observe that after widening the GSV pane from 720 to 920px, the label "jumps" after panning:

After

Using PanoMarker, widening the GSV pane does not interfere with the label's sticking:

Even with PanoMarker I still have not been able to perform a scale-up/zoom of the audit interface without affecting label sticking, but I think the ability to change the width and height of the GSV area should be sufficient. Next steps are reimplementing the hover/click actions of the original labels using PanoMarker, followed by screen size detection.

[misaugstad]

I am looking into the possibility of replacing the label attachment code, which seems to be custom, with the PanoMarker utility that @aileenzeng used for the admin panel in PR #1308

From the PanoMarker utility, do we get sufficient information to place the label on the panorama in the exact same place later? I remember a long time ago looking into PanoMarkers (I think that's what it was) and finding that I could only get the lat-lng from them, not the full info needed to place it in the exact place on the pano for future crops...

I may be misremembering or maybe things have been updated since then. But I wanted to check in to make sure this is possible given some past issues with that.

[aileenzeng]

Even with PanoMarker I still have not been able to perform a scale-up/zoom of the audit interface without affecting label sticking

I ran into label sticking problems when I was working with PanoMarkers too! I think you might be able to fix it by setting the anchor so that the google.maps.Point is at the center of the PanoMarker. Here's an example from the admin interface:

self.labelMarker = new PanoMarker ({
    [...]
    size: new google.maps.Size(20, 20),
    anchor: new google.maps.Point(10, 10)
});

[tongning]

Thanks for the suggestion @aileenzeng! I have set the anchor but I think I didn't describe the problem clearly enough: Zooming inside the GSV pane is fine, but programmatically upscaling the GSV pane (i.e., simulating the user adjusting the browser's zoom) causes some issues. I'm guessing this sounds like a different problem?

@misaugstad that is a good point, I will make sure to view the labels in the admin interface to check that they're still in the right spot. Regardless of what new label attachment method we switch to (PanoMarker or otherwise) I think this is a fairly high-risk change, so it will need extensive testing.

[jonfroehlich]

Love this discussion. Thanks all for contributing. I agree that this is a high-risk change but potentially with high reward! I hope @kotarohara gets back to you @tongning on providing more context and rationale for implementing things the way he did (it may be as simple as "well, I did what was fastest and it worked!").

[kotarohara]

Hello and happy 2019 :)

@aileenzeng, that makes sense because @kotarohara originally designed Project Sidewalk for laptops of that screen size I believe (because this was his screen resolution at the time!).

The GSV pane size was informed by the browser window size data that we collected from AMT workers (I think we still keep track of this information). We set the size to the current value so that most people could see all the UI components on the browser window. But that's like 5-6 years ago, so I guess people are using larger displays.

My feeling so far is that there isn't an easy surface-level fix for this, so I'm currently looking into whether the math for the label positioning can be updated to support scaling. Open to hearing suggestions!

One hack-y way to find the equations for alpha_x and alpha_y is incrementing the pane width and height from the current values, and interpolate to get the equations. I don't think Google engineers who are designing Street View APIs do something really complicated, so my hunch is that the alphas are (inversely) proportional to the pane width and height. But I could be wrong!

In terms of ideas, do we use the official Google Street View API for overlaying labels (link)? If not, why not? Wouldn't this solve the problem as the GSV engine itself would then handle visualizing the markers in the right place... (And the higher level point here is maybe we need to fundamentally question how we are visualizing labels in the first place and attaching them to GSV).

I hope @kotarohara gets back to you @tongning on providing more context and rationale for implementing things the way he did (it may be as simple as "well, I did what was fastest and it worked!").

I think this was the only possible way to implement, at least at that time. The official API on the linked page only allows you to specify a latitude-longitude coordinate for a marker. I don't think this is useful to put a stamp on a Street View image, which requires a precise x and y image coordinates. I haven't checked Street View APIs for a while, so maybe there are newer APIs that allow us to do what we do on the Project Sidewalk interface (i.e., putting a stamp on a specific point on a Street View image)... @misaugstad @aileenzeng PanoMarkers looks cool! I wonder how they are doing it.

@tongning if you manage to figure out the equations to determine alphas and let the program dynamically adjust the Street View pane size, please make sure that the correct alphas are kept track in the database so that labels can be put in the right spot as @misaugstad mentioned. I think these alphas are stored in the interaction table.

[tongning]

Thanks @kotarohara, this is very helpful! So if I'm understanding correctly, the process for determining alpha_x and alpha_y is essentially changing the GSV pane size, and then adjusting them by trial and error until the labels stick correctly?

I can definitely see why the official API is insufficient now, and it sounds like what @misaugstad was referring to? PanoMarker tracks the label's heading and pitch, which to me sounds sufficient to locate a point on a panorama but I could be wrong. There is a process for convering canvas coordinates to panorama heading and pitch described here. However I don't see PanoMarker keeping track of depth...@kotarohara is this something you feel might cause issues?

[jonfroehlich]

However I don't see PanoMarker keeping track of depth...@kotarohara is this something you feel might cause issues?

If we know where the PanoMarker is on the pano, can't we just use the same code we are using to extract the depth position (see CHI'19 paper). We need depth data for each label because we use this in our CV algorithms.

[tongning]

If we know where the PanoMarker is on the pano, can't we just use the same code we are using to extract the depth position (see CHI'19 paper). We need depth data for each label because we use this in our CV algorithms.

My impression (let me know if this isn't the case) is that the depth data is also used by the tool itself to compute lat/lng of the label. So I'm concerned that replacing the label attachment code with PanoMarker might make it appear that the everything is working, but cause lat/lng to be computed inaccurately when the GSV pane is resized.

[jonfroehlich]

Ah, I see. Yes, lat/lng computation is absolutely critical as we need to know real-world position of these problems. Can you start experimenting or are you bottlenecks on Kotaro (because he’s in Singapore and it’s 3:36am there right now)?

Sent from my iPhone

On Jan 2, 2019, at 11:30 AM, Anthony Li notifications@github.com wrote:

If we know where the PanoMarker is on the pano, can't we just use the same code we are using to extract the depth position (see CHI'19 paper). We need depth data for each label because we use this in our CV algorithms.

My impression (let me know if this isn't the case) is that the depth data is also used by the tool itself to compute lat/lng of the label. So I'm concerned that replacing the label attachment code with PanoMarker might make it appear that the everything is working, but cause lat/lng to be computed inaccurately when the GSV pane is resized.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[tongning]

I can experiment; just want some eventual confirmation from Kotaro as to whether the depth calculation code will need to be updated 🙂

[kotarohara]

Thanks @kotarohara, this is very helpful! So if I'm understanding correctly, the process for determining alpha_x and alpha_y is essentially changing the GSV pane size, and then adjusting them by trial and error until the labels stick correctly?

Yes, you are right :)

PanoMarker tracks the label's heading and pitch, which to me sounds sufficient to locate a point on a panorama but I could be wrong.

I think you are right, @tongning ; the heading and pitch should be sufficient to compute the image coordinates and canvas coordinates. But take it with a grain of salt as I don't really know how PanoMarker works.

My impression (let me know if this isn't the case) is that the depth data is also used by the tool itself to compute lat/lng of the label.

You are right that the current labeling tool uses the depth information to compute latlng. It's used to put label on the Google Maps pane (apart from sending that information back to the server). But you only need image coordinates to find the depth information, and technically you should be able to acquire the necessary information from heading and pitch.

[tongning]

Thanks @kotarohara! Sounds like I should be able to continue with the PanoMarker migration; I'll report back here if I hit any blocks.

[jonfroehlich]

Any updates on this?

[tongning]

So far, I think the PanoMarkers are working fine for marking the labels themselves. The difficult part of the migration has been ensuring that all of the items surrounding the label also stay at consistent positions - this includes the delete icon, severity tag, missing severity badge, and severity selection box. Based on the PanoMarker documentation, the PanoMarker libraries don't seem to support anything more complex than placing an icon. Though we might be able to stretch it a little by adding the delete icon and missing severity badge as separate PanoMarkers, there doesn't seem to be an elegant way to use PanoMarker to render the severity selection box.

I was reading a bit more into the PanoMarker code and documentation and found this comment:

/*
 * Custom StreetView panoramas usually do not have any geographical information
 * (e.g. inside views), thus a different method of positioning the marker has to
 * be used. This class takes simple heading and pitch values from the panorama's
 * center in order to move the marker correctly with the user's viewport
 * changes.
 *
 * Since something like that is not supported natively by the Maps API, the
 * marker actually sits on top of the panorama, DOM-wise outside of the
 * actual map but still inside the map container.
 */

There is also a detailed blog post here about how PanoMarker works. It looks like PanoMarker is just drawing the label on top of the street view pane and recomputing the position to draw the label as the street view is panned. The approach I am currently trying is to use the PanoMarker library for computing the position to draw the label, while still drawing everything manually. If this works, we might be able to keep a lot of existing code for surrounding items like the severity selection box.

[jonfroehlich]

This is also related to https://github.com/ProjectSidewalk/SidewalkWebpage/issues/1287. @tongning, any updates on this one?

[tongning]

I put this aside after AccessScore started but I'll pick it back up now that I'm working on some miscellaneous issues. If I'm understanding #1644 correctly, this issue is now higher priority because label stickiness isn't working in Safari, but is still working in Chrome and Firefox?

[jonfroehlich]

Actually, I believe @aileenzeng fixed the label stickiness issue from https://github.com/ProjectSidewalk/SidewalkWebpage/issues/1644. But she can weigh in on this...

[aileenzeng]

Yep, it's fixed now! It should be working in Chrome/Firefox/Safari.

[jonfroehlich]

What's the status of this? @tongning, do you have cycles to come back to this? Given that you made significant progress on this--I would hate for that work to get wasted. You are in the best position to close the loop on this and actually solve the problem! :)

[tongning]

Still in finals but hopefully I'll have cycles soon :) Should I prioritize this over autocropper debugging and general scraper improvements?

[jonfroehlich]

Auto-cropping and scraper are more important.

Sent from my iPhone

On May 20, 2019, at 6:02 PM, Anthony Li notifications@github.com wrote:

Still in finals but hopefully I'll have cycles soon :) Should I prioritize this over autocropper debugging and general scraper improvements?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[misaugstad]

@tongning how far did you get on this, and how hard do you think it would be for another student to pick up where you left off?

[jonfroehlich]

Yes, getting a full status report on how far you made it on this, what you learned, and what's left to do would be great. Based on my memory, Anthony was attempting to use Aileen's Panomarker strategy and made significant progress...

On Mon, Jul 29, 2019 at 12:07 PM Mikey Saugstad notifications@github.com wrote:

@tongning https://github.com/tongning how far did you get on this, and how hard do you think it would be for another student to pick up where you left off?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ProjectSidewalk/SidewalkWebpage/issues/303?email_source=notifications&email_token=AAML55OTL4VW3WBGGMI2AITQB4WXBA5CNFSM4COVAE3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3BRJQI#issuecomment-516101313, or mute the thread https://github.com/notifications/unsubscribe-auth/AAML55ONAVBSCRZ6VSQ4V6LQB4WXBANCNFSM4COVAE3A .

-- Jon Froehlich Associate Professor Paul G. Allen School of Computer Science & Engineering University of Washington http://makeabilitylab.io @jonfroehlich https://twitter.com/jonfroehlich - Twitter Help make sidewalks more accessible: http://projectsidewalk.io

[tongning]

The approach of using the math in the Panomarker library without using Panomarkers themselves was looking fairly promising, but I think the trickiness is in the details - for example, making sure that zoom works, that labels reappear after moving to another pano and returning, that interaction logging still works, etc. That requires deep-diving into the existing code, but I think it's totally reasonable for someone else to give it a try. I pushed some experimentation code to branch 303-autoresize-auditpane.

[jonfroehlich]

This is still a problem and our users are still complaining (note that the admin panel now shows the feedback column in a really condensed fashion for some reason):

[jonfroehlich]

Since we discussed this recently, just tagging you on it @hoominchu. It's another one of those blue sky, persistent problems we've had. IIRC, in our most recent conversation, @misaugstad suggested possibly switching everything to PanoMarker.

This may also be related to https://github.com/ProjectSidewalk/SidewalkWebpage/issues/2660

[misaugstad]

We continue to get requests for something like this through our survey on the Explore page.

On July 13, 2023 on Zurich server (translated from German)

Simply make everything larger, especially the font. It is so unclear that you have to go so close to the screen, which is not normal.

On Aug 20 on our Burnaby server:

would be nice to have a fullscreen option, can see stuff more clearly

[misaugstad]

I think I may have figured out a way to automatically zoom..? You can do a sort of CSS zoom, though there are may be some issues that can arise from that that we should look into more fully. Partially as a help from chatGPT, you can add this function in your browser console:

function isElementVisible(elementId) {
  var myElement = document.querySelector(elementId);
  var zoomFactor = parseFloat(document.body.style.zoom) / 100.0 || 1;
  var scaledRect = myElement.getBoundingClientRect();
  if (zoomFactor !== 1) {
    scaledRect = {
      top: scaledRect.top * zoomFactor,
      left: scaledRect.left * zoomFactor,
      bottom: scaledRect.bottom * zoomFactor,
      right: scaledRect.right * zoomFactor
    };
  }

  var isVisibleWithZoom =
    scaledRect.top >= 0 &&
    scaledRect.left >= 0 &&
    scaledRect.bottom <= (window.innerHeight || document.documentElement.clientHeight) &&
    scaledRect.right <= (window.innerWidth || document.documentElement.clientWidth);

  console.log('Is element fully visible with zoom?', isVisibleWithZoom);
}

Then if you call this, for example, it should return true if the whole UI is visible, and false otherwise.

isElementVisible('#pop-up-message-background');

If you use browser zoom to zoom in far enough, it should start to return false. And you can then use CSS to zoom using this:

document.body.style.zoom = '120%';

I've played around with this, and it seems to correctly check if the entire element is visible within the viewport, taking into account the zoom! Given this, we should be able to automatically increase the zoom level by increasing size until we detect a size where we don't have everything visible, then we just go to the next highest size!

We should run some tests to make sure that the GSV window works correctly and that all the data is saved appropriately using this method :)

[misaugstad]

Okay one issue is that this property is not available on Firefox :grin: