The algorithm itself is described in the HTML5 specification, and various browser engines only implement it. There are two stages, tokenization (where each tag and data string is recognized) and the tree building. During the second stage, a stack of opened elements (among other ones), which were detected in the first step, is being kept. Using that stack, the engine will be able to detect mismatches.
How does browsers handle unclosed elements?
The algorithm itself is described in the HTML5 specification, and various browser engines only implement it. There are two stages, tokenization (where each tag and data string is recognized) and the tree building. During the second stage, a stack of opened elements (among other ones), which were detected in the first step, is being kept. Using that stack, the engine will be able to detect mismatches.
There are more details in the W3C specs: https://dev.w3.org/html5/spec-preview/Overview.html#an-introduction-to-error-handling-and-strange-cases-in-the-parser
"Tag Soup"
In Web development, "tag soup" refers to syntactically or structurally incorrect HTML written for a web page.
https://en.wikipedia.org/wiki/Tag_soup
Further readings