Fast Scroll Can Cause HTMX reveal event To Fail

wiverson commented 3 years ago

HTMX fails with the following error in the console:

Uncaught TypeError: Cannot read property 'toUpperCase' of undefined
    at $t (htmx.min.js:1)
    at Fe (htmx.min.js:1)
    at htmx.min.js:1
    at X (htmx.min.js:1)
    at htmx.min.js:1

...when scrolling quickly. The code to replicate can be found at https://github.com/wiverson/htmx-demo - and the specific code generating the view can be found at:

Starting Thymeleaf View Java Spring Boot Controller

For reference, this is using the WebJAR version of HTMX, v1.3.2.

Replicated on macOS on both Safari and Chrome.

wiverson commented 3 years ago

Stack from the console using the htmx code from the non-minified version of htmx on GitHub:

Uncaught TypeError: Cannot read property 'toUpperCase' of undefined
    at issueAjaxRequest (htmx.js:2144)
    at maybeReveal (htmx.js:1066)
    at htmx.js:1055
    at forEach (htmx.js:222)
    at htmx.js:1054

wiverson commented 3 years ago

FWIW as an experiment I tried changing to hx-swap="afterend settle:1s" and adding the settle:1s made it much, much quicker to die with the same stack.

Changed to settle:0s and it won't generate the error no matter how fast I scroll.

1cg commented 3 years ago

@benpate You interested in looking into this one?

benpate commented 3 years ago

Yes. I will give it a try. Thank you for the test code. That should make this easier to find/fix.

I experienced a similar issue with an older version of this code and went looking for a solution. It seems like it works "mostly" but not all the way.

benpate commented 3 years ago

First, I'm sorry it's taken me some time to take a look at this. But, I finally did, and here's what I found.

1) It looks like the "revealed" trigger is working correctly. I've tried the manual test page on Firefox/Safari/Chrome, and I'm able to quickly scroll through pages and pages of mocked HTTP requests on all three browsers. This is still true when I add hx-swap=""innerHTML settle:1s" to the manual test page.

2) Looking at your error message, it seems like the problem is actually an undefined is being passed in to the issueAjaxRequest function (which is what calls the server after the revealed trigger is fired). This makes me guess that one of the other hx- parameters is missing.

I'm not very familiar with the Java templating engine you're using, but it seems like the issue is on line 17 of your demo code which uses hx-trigger="revealed" but then uses th:hx-get="" instead of simply hx-get. Again, maybe Java is manipulating this before it gets to the browser, but it looks like the place to start. Is it possible to remove the th: from this attribute?

wiverson commented 3 years ago

RE: taking time, no worries. Fix my bug in your open source project now! haha no.

The infinite-scroll.html is a Thymeleaf template. All the th:hx-get version does is rewrite the URL inside a bit for server config. I just checked in a version that doesn't do that - for a local test it makes no difference.

The infinite-scroll.html version is the initial load. The version that comes back to the server is actually coming from the controller.

Looking at it more closely, I think it might be as dumb as malformed html coming back from the controller. Hmm.

benpate commented 3 years ago

Cool :). Kick it around and let me know if you need more help?

wiverson commented 3 years ago

Ok, I poked at it some more, and the HTML in the current version is now valid.

Can you check on your end using a version that doesn't use the built-in JS local server? I suspect that might be causing the threading to sync in the browser in a fashion that's not representative of a "real" server.

You can replicate with my demo example by installing JDK 16 & Maven and just running mvn spring-boot:run but I suspect you could also replicate with any other server, including just a dumb local micro http server.

Here's the entire rendered html base:

<!doctype html>
<html lang="en" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://www.ultraq.net.nz/thymeleaf/layout ">

<head>

    <title>Infinite Scroll</title>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

    <title>Infinite Scroll</title>

    <script src="/webjars/jquery/jquery.min.js"></script>
    <!--Bootstrap CSS-->
    <link rel="stylesheet" href="/webjars/bootstrap/4.6.0-1/css/bootstrap.min.css"/>
    <!--Bootstrap JS-->
    <script type="text/javascript" src="/webjars/bootstrap/4.6.0-1/js/bootstrap.min.js"></script>

    <!-- Debug HTMX -->
    <!-- <script type="text/javascript" th:src="@{/js/htmx.js}"></script> -->

    <!--HTMX-->
    <script type="text/javascript" src="/webjars/htmx.org/1.3.3/dist/htmx.min.js"></script>

    <!-- Hyperscript for fancier stuff -->
    <script type="text/javascript" src="/webjars/hyperscript.org/0.0.9/dist/_hyperscript.js"></script>

    <style>
        .htmx-indicator {
            background-color: black;
            position: fixed;
            bottom: 0;
            right: 0;
            width: 60px;
            padding: 5px 5px 5px 5px
        }
    </style>
</head>

<body>
<nav class="navbar navbar-expand-lg navbar-light bg-light">
    <div class="container-fluid">
        <a class="navbar-brand" href="/">HTMX + Spring Boot Demo</a>
        <p class="navbar-text">Rendered at <span>15/May/2021 14:10</span></p>
    </div>
</nav>

<section>
    <div class="container alert alert-secondary" role="alert">
        <p>This is heavily tweaked version of the <a href="https://htmx.org/examples/infinite-scroll/" target="_blank">htmx
            Infinite Scroll demo</a>, modified for vanilla Bootstrap and Spring Boot.</p>
    </div>
    <div class="container">
        <table hx-indicator=".htmx-indicator" hx-get="/infinite-scroll/page/1" hx-trigger="revealed" hx-swap="innerHTML settle:0s">
            <tr>
                <td>Loading...</td>
            </tr>
        </table>
    </div>
</section>
<img alt="Loading Indicator"
     class="htmx-indicator"
     src="https://raw.githubusercontent.com/SamHerbert/SVG-Loaders/master/svg-loaders/spinning-circles.svg"/>
</body>

</html>

... and here is a sample of the response...

 <tr>
     <td>Madalene</td>
     <td>Weber</td>
     <td>wesley.oconnell@example.com</td>
 </tr>
 <tr>
     <td>Sonny</td>
     <td>Turner</td>
     <td>jasmin.pouros@example.com</td>
 </tr>
 <tr>
     <td>Fernando</td>
     <td>Kassulke</td>
     <td>francisco.fadel@example.com</td>
 </tr>
 <tr>
     <td>Herb</td>
     <td>Satterfield</td>
     <td>porfirio.borer@example.com</td>
 </tr>
 <tr>
     <td>Drucilla</td>
     <td>Prosacco</td>
     <td>hank.beer@example.com</td>
 </tr>
 <tr>
     <td>Mac</td>
     <td>Denesik</td>
     <td>paul.hansen@example.com</td>
 </tr>
 <tr>
     <td>Marion</td>
     <td>Lesch</td>
     <td>colby.nader@example.com</td>
 </tr>
 <tr>
     <td>Moshe</td>
     <td>Corwin</td>
     <td>mike.klein@example.com</td>
 </tr>
 <tr>
     <td>Darla</td>
     <td>Glover</td>
     <td>madalyn.bode@example.com</td>
 </tr>
 <tr hx-get="/infinite-scroll/page/2"
     hx-trigger="revealed"
     hx-swap="afterend">
     <td>Marlin</td>
     <td>Jacobs</td>
     <td>isaiah.hills@example.com</td>
 </tr>

Let me know if that is enough to replicate. Probably could literally replicate with these as raw HTML in an python3 -m http.server instance if that's easier.

wiverson commented 3 years ago

Oh, and the web jars stuff is just a fancy Java way of managing/referencing NPM libraries that have been turning into Maven (Java build tool) dependencies.

wiverson commented 3 years ago

So, I can't replicate it with my MacBook Pro trackpad scrolling, but I can with an eternal Logitech scrolling mouse.

What.

benpate commented 3 years ago

What.

Yeah, that's pretty weird, but we're looking at a pretty specific issue. I'll try to replicate on my machine using a separate web server. I also have a MacBook, but have only tested with an Apple Mouse. I'll try with a couple of PC mice, to see if that affects things. We might be dealing with a specific incompatibility related to smooth scrolling.

Separately, I really like the way htmx gives us a "revealed" trigger to work with, but its essentially a synthetic event on that node that's generated inside the library. To maintain compatibility with IE11, we're using a less efficient way of watching for scroll events. I may be needing something more powerful soon (for instance, to let me know when something scrolls OFF of the page, too) and I'm thinking about an extension or separate JS library that would trigger events. If I end up with anything useful, I'll try to publish it for the htmx community, too.

benpate commented 3 years ago

Hey @wiverson - I'm still getting the same results no matter what I try.

I've reworked the demo to load 100 page fragments from an external web server (not sinon.js). The code looks like this

<div class="panel" hx-get="http://localhost/page/0" hx-trigger="revealed" hx-swap="innerHTML settle:1s"></div>
<div class="panel" hx-get="http://localhost/page/1" hx-trigger="revealed" hx-swap="innerHTML settle:1s"></div>
<div class="panel" hx-get="http://localhost/page/2" hx-trigger="revealed" hx-swap="innerHTML settle:1s"></div>
<div class="panel" hx-get="http://localhost/page/3" hx-trigger="revealed" hx-swap="innerHTML settle:1s"></div>
<div class="panel" hx-get="http://localhost/page/4" hx-trigger="revealed" hx-swap="innerHTML settle:1s"></div>
<div class="panel" hx-get="http://localhost/page/5" hx-trigger="revealed" hx-swap="innerHTML settle:1s"></div>
<div class="panel" hx-get="http://localhost/page/6" hx-trigger="revealed" hx-swap="innerHTML settle:1s"></div>
<div class="panel" hx-get="http://localhost/page/7" hx-trigger="revealed" hx-swap="innerHTML settle:1s"></div>
<div class="panel" hx-get="http://localhost/page/8" hx-trigger="revealed" hx-swap="innerHTML settle:1s"></div>
<div class="panel" hx-get="http://localhost/page/9" hx-trigger="revealed" hx-swap="innerHTML settle:1s"></div>
...

I tried it with three different mice (my Apple Mouse, a Razor gaming mouse, and a crappy iHome USB mouse from last millennium)
I've tried it with three different browsers (Firefox, Safari, and Chrome)
I've tried with and without hx-swap="innerHTML settle:1s"

Everything still loads quickly after I scroll to a new section of the page. Sorry, I'm going to need to kick this back to you. I think something may still be funny with your code.

Perhaps you could put a warning on your website: "not compatible with Logitech mice." Maybe that would work? /s

wiverson commented 3 years ago

Yeah, external mice (LogiTech or otherwise) are pretty popular, so it's still worth triaging IMHO.

I'm using a LogiTech MX Anywhere 3, FWIW. Best external mouse I've ever used, ha.

I can't replicate the issue with the online demo that uses the test JS server, but I have replicated it with both a Java Spring Boot and a dumb Python HTTP server.

Here is the dumb Python server version:

htmx-463.zip

run.sh to launch the server, http://localhost:8000/ and spin the mouse wheel and can instantly replicate. But only with the mouse.

Observations:

Changing the settle to zero makes it go away. Increasing settle makes the bug appear faster.
The error seems to be a missing property in the Ajax issue under the maybe reveal?

One interesting thing I didn't know about until now: WheelEvent is a thing. So perhaps this is all due to some events getting fired related to wheel events?

Some potentially interesting links I found just now looking at this a bit.

Wheel event
Scroll events
Fast scrolling
Weird scroll wheel bug - btw this issue I can replicate in Safari as well as Chrome, but interesting code on scroll wheels.
Hilariously complicated overview of different event issues

Perhaps this is an issue with HTMX and scrolling/mouse wheel events?

wiverson commented 3 years ago

Event trace that fails (using the LogiTech mouse wheel):

failing events Screen Shot 2021-05-16 at 11 00 18 AM

Event trace that works (using the MacBook Pro trackpad scrolling):

working trackpad events Screen Shot 2021-05-16 at 11 03 24 AM

Generated the traces by using monitorEvents(window)

Want me to buy you a mouse? Or if you can find the bug in the two HTML files I just sent over I'll send HTMX a donation equal to the $ for a mouse? :)

benpate commented 3 years ago

I'll try to replicate it with your example html. That seems like the best way forward.

You're very kind to offer a donation to htmx. They would go to @1cg, not to me, so I'd say do it anyway because he's awesome. But thank you, no, I'm good, I don't need a donation 😎

wiverson commented 3 years ago

Use the HTML in the zip - I cleaned those out to make the smallest repro case I could.

TBH these kinds of issues are why I prefer the JVM - I know all the gory threading internals and tools to fix there. Guess I should get off my high horse, buckle down and get into some JS analysis after all these years, ha. ;)

Let me know if there is anything else I can do to help. I'm really, really curious now. My brain naturally leaps to exotic stuff like event buffer overflows or some kind of event thread sequencing issue, but who knows. 🤷‍♂️

wiverson commented 3 years ago

Went ahead and fired up the debugger. Looks like the call to getInternalData is expecting a node with a verb, but for some reason that verb isn't defined.

My guess is that when settle is set to zero, getInternalData is able to correctly find the result, but when settle is set to (for example) 1s there is something about the sequence of events being fired/parsing the new data/etc that's out of sync.

benpate commented 3 years ago

Gotcha. Will do :). We'll find it.

benpate commented 3 years ago

I've been able to reproduce this problem on my machine -- specifically using my keyboard to jump all the way to the bottom of the page while existing pages are loaded. it is failing on the same line that you've highlighted (2147 on that build, 2144 using the non-minified https://unpkg.com/htmx.org@1.3.3/dist/htmx.js)

It's going to take some digging, because (as you mentioned) it's failing during the issueAjaxRequest() function, which gets run after the revealed trigger has done its job. I'm going to look at this a little more, but we're probably going to need hit the red button to call in @1cg on this one.

I think my first guess is the same as yours @wiverson -- that the revealed trigger is running before the node has been fully processed. This also fits with your experience that using settled:1s makes this problem easier to replicate. The fix is probably too deep into htmx's internals for me to figure out, but it will probably require something like holding on to that trigger for a moment, or having issueAjaxRequest do some on-the-fly "settling" of its own to make the request go through.

Either of these "solutions" might work great, but will depend on some deep understanding of the underlying architecture.

wiverson commented 3 years ago

I've been able to reproduce this problem on my machine

Woot! ;)

1cg commented 3 years ago

thank you for looking into this so deeply @benpate !

I'll take over from here and see if I can come up w/ a fix. Both of you are probably onto the root cause: the request is happening before the element is fully processed.

create an open source web framework they said...

it'll be fun they said...

:beers:

1cg commented 3 years ago

@wiverson can you grab the latest htmx.js from the dev branch and try out my possible fix in https://github.com/bigskysoftware/htmx/commit/4dd50048510a583a1dfb6d7ace1787fdfcd4140b

wiverson commented 3 years ago

try out my possible fix in 4dd5004

Done - works great now!

Tested with no settle set, 0s and 2s. All works fine now.

create an open source web framework they said... it'll be fun they said...

Fame! Fortune! Glory! ^_^

...

Or at least popping up on HN once in a while. ;)

FWIW, I'm working on a deck, tentatively titled "Full Stack Java Development in 2021" and htmx is going to be a cornerstone. Likely some combination of presenting at the local Java user group and/or a YouTube video. :)

1cg commented 3 years ago

great to hear this is now working, closing this issue out

bigskysoftware / htmx

Fast Scroll Can Cause HTMX reveal event To Fail #463