Open bemson opened 9 years ago
bemson
commented
9 years ago So the event.timeStamp property is undefined in IE10 - at least. Here's a quick jsbin page to see the results yourself. This has tripped up the meta data I attempt to validate, causing localStorage messages to fail.
The fix is simple. Just capturing the problem for posterity.
bemson
commented
9 years ago storage eventsAccording to my tests: all versions of IE drop storage events. (I guess that's why they don't even bother firing them in IE 11...)
To observe the dropped events: open this jsbin storage reader in a new window, then this jsbin storage writer in a different window - place them side by side if you like. The writer simpler upticks a storage key every n milliseconds. The reader observes this change and outputs the value it sees.
Now, in IE because the storage event has zero information about the updated value, we're forced to retrieve it at the time of the event. This already allows for the value to change before the event can even be processed, and tha's exactly whats happening. We effectively lose value changes. If you set the interval to something like 25, you'll see that the reader outputs reams of lines showing the same value being "changed" (each line is a storage event).
Based on my rudimentary IE emulator, the threshold is just above 30ms. Meaning, changes occurring faster than that will get lost (rather, overwritten). Unless there is a way to coordinate the cadence at which every window writes to localStorage, I don't see a way to capture changing values.
There isn't much hope for SubEtha on IE, folks.
cc: @williamkapke - You mentioned this ages ago, when you first told me about scomm... How did your team end up ensuring dropped messages were received? How did that work on scale?
cc: @cnojima - Any ideas on how to work within a network with dropped packets? :-D
Any insights welcome folks. Oh and happy holidays!
bemson
commented
9 years ago IE 9 & 10 have empty storage events, which informs when to read storage keys but also allows a race-condition, where the change which triggers the event gets overwritten by the time it is read.
Since writing to a single key can not be coordinated between n number of bridges, the solution must first allow for each bridge to have it’s own writable key. This way each bridge can throttle it’s own writes, and not be concerned about blowing away messages from another bridge.
But since the IE solution means bridges no longer share keys, there would be no manifest (or map) listing all known bridges. The race-condition simply prevents using localStorage for syncing data across runtimes. Instead, a more concurrent, shared data store must be employed: cookies.
A single cookie can be shared between bridges (iframes). Upon initializing, each bridge will have a random delay before accessing this cookie, in order to retrieve and update a “bridge manifest”. This could be a simple hash, listing each bridge id. Bridges would need to update this cookie, in order to inform other bridges that they have exited the network.
The bridge manifest would also serve as a change manifest, tracking the last time each bridge updated it’s single storage key. Thus, when a given bridge receives a storage event, it first references the cookie in order to compare timestamps and determine which bridge sourced the event.
This solution presumes that cookie access is faster than localStorage. It also presumes that localStorage reads and writes are either blocking or immediately shared between window instances. That is, how soon is a set storage value available to other iframes?
I'll be working with on feasibility tests for this change-manifest strategy. I plan to use jsbin and test pages side by side in IE9-10. I also plan to test IE11, since the tests will be not be observing events - instead they will be testing latency thresholds with extreme polling.
I'd like to verify the following assumptions:
Finally, I'll check if any performance improvements can be had by using Web Workers. These are supposed to have access to localStorage as well, but I'm very wary of how they will perform in a "separate" thread - specifically on IE.
cc: @cnojima
bemson
commented
9 years ago To test how long it takes for updated cookie and storage values to be available to other windows, I created test pages in jsbin. The writer displays a timestamp of when the value is set (i.e., when you click a button). The reader observes changes every given number of milliseconds (5, by default), and outputs when the value changes along with a timestamp. The storage-reader also observes the storage event.
The cookie writer and cookie reader jsbins.
The storage writer and storage reader jsbins.
In IE 9 (via emulation), both cookies and storage are available in the range of sub-milliseconds to 4 milliseconds. There is no distinct difference in speed, at least for small values.
Also, the storage event does occur 5ms or so after the storage value has changed.
bemson
commented
9 years ago It appears that IE fires events long after the storage value has changed. Based on these tests, both the cookie and storage values will have been updated in time, before the storage event occurs.
However, though storage keys will only be updated every 30ms or so per iframe, that doesn't synchronize writes to their shared cookie.
bemson
commented
9 years ago The cookie-manifest approach attempts to reduce the i/o needed to observe changes in localStorage. Below is an illustration of the expected sequence. In brief, setting and getting localStorage values will involve setting and reading a shared cookie, first.
Based on earlier tests, new storage and cookie values would be available when the corresponding storage event occurs. Combined with throttling writes by ~40ms, the architecture should ensure messages are not dropped. As well, this avoids having to store previous messages or unnecessarily read storage-keys (i.e., the cookie will inform which keys have been updated).
IE 11 can use this architecture, except it's trigger for reading storage-keys would be changes in the cookie-manifest - not the storage event. This is because IE11 doesn't support storage events for iframes in different domains (some forking may be done, to support the best change monitor/management approach).
The cookie-manifest does rely on reading and writing to a cookie that will be shared by n number of iframes. This is understood to be a risk, but is the only idea I've found that avoids any kind of heartbeat style monitoring. The latter requires each bridge track the presence of other bridges, using timestamps. It may be worth looking into, but definitely requires more overhead and guarantees more latency... will look into after testing the cookie-manifest approach.
bemson
commented
9 years ago Making definite progress with IE implementation. As suspected, the unload logic is either (1) not firing, or (2) not cleaning up the cookie and storage keys as needed.
I'm trying to use an cleanup routines for "expired" bridges - bridges that are not in the manifest, but have a tracking message. It will make everything rather slow, but necessary to avoid "ghost" peers that hang around per client.
bemson
commented
9 years ago Works in IE11 too. Gonna clean up for IE9 next. Final test is to see if the winfo visualizer works as expected.
bemson
commented
9 years ago There is an issue with the visualizer, where either clients or client events are dropped. But, I fear it's in that code, not due to the bridge itself. I won't close this until the visualizer works as expected in IE browsers.
bemson
commented
9 years ago The IE manifest solution began to have race-condition failures from too many browser windows attempting to read and write to one cookie. A heartbeat approach will ensure bridges can track when other bridges drop and join the network.
The approach calls for each bridge to update a tracked cookie value, every X number of milliseconds. Other bridges will check the cookie every X/2 milliseconds, to ensure each bridge is captured and removed as needed. A "dead" bridge would be one that has not updated it's ticker after 1.5-2X seconds.
The heartbeat addition is rather minor, but the working code has many moving parts and will still take some time to implement. Below is the latest sequence diagram, demonstrating more of the full network actions and logic behind SubEtha so far.
updated sequence diagram 2/27
bemson
commented
9 years ago Just chatted with @SerpentJoe, who came up with a brilliant method for ensuring zero dropped messages in IE: self-receipts. The idea is that the message manifest would serve as both a signal for new messages (to other bridges) and a confirmation that messages have been "received" (for the same bridge).
After bridge A updates it's message ticker (in the message manifest/cookie), it's next check of the cookie would confirm that the changed value is present. If not, the cookie is again set, to the last ticker value. This handles the case where a cookie write may be blown away by another bridge that happened to be set the cookie at the same time.
The only case where this doesn't work, is when the last message is dropped, before the bridge exits. It requires bridges to write near each other, but that should be generally avoidable.
There's more to this idea, but it adds a great deal of confidence to this headache laden attempt to get SubEtha to work on IE. Thanks again, to Peter Jones, and I'll post another sequence diagram in a few days.
bemson
commented
9 years ago Ok, this looks like it should work. This sequence diagram may be updated in the future, but it best captures the "self-receipt" concept, inspired by Peter Jones (@serpentjoe). I'll be coding this for the next week or so, in order to eliminate dropped messages and ghost peers.
bemson
commented
9 years ago Okay. I've clearly been discouraged by this polling/cookie/localStorage approach, up till now required for SubEtha to work in IE. Forking the implementation between it and standards compliant-er browsers simply blows chunks. It hurt my brain trying figure out a timing strategy, based on the number of active bridges. (I'm terrible with numbers.)
But then, @gothy takes note of this project and stars it. I was inspired to push onwards, presuming folks want a working idea, not just a good one. Turns out, he left a comment on crosstab (a similarly purposed library), issue tejacques/crosstab#26 ... Not surprisingly that project has equally difficult localStorage challenges in IE.
His comments links to a localStorage bug against IE 11. The bug lists some workarounds, the last of which recommends using indexedDB (IDB).
I had heard of this API, and confused it for WebSQL, a narrowly supported client-storage feature. IDB on the other hand, has wider browser support. Even better news: preliminary tests indicate how IndexedDB will work between different domain iframes in IE 10.
That said, put simply, I disliked their API. It took a day for me to nail down the choreography of objects and methods needed for minimal usage. Nonetheless, with my newfound understanding, I'm excited to code the SubEtha-Bridge module using IDB exclusively - removing any browser forks.
The manifest/msg concept will be preserved for IDB. The main difference is that IDB will not dispose messages. The localStorage approach uses a single, designated key, for different messages - relying on events to broadcast the new value. The former means I can use IDB as a message log, and no message can get dropped!
IDB is not an event-based system, such as localStorage. There is no way for one window to notified of changes to IDB storage. Thus, the first pill to swallow is that IDB will require polling. The good news is that the each poll has a response event (i.e., "success"). This means I can tightly poll after each response event, instead of periodically (with setInterval or setTimeout). This should improve performance and reduce computational resources somewhat, akin to running code per requestAnimationFrame cycle.
bemson
commented
9 years ago Looks like IndexedDB handles all the concurrency issues introduced by the peerless'ish network I'm attempting to create with SubEtha. It's the centralized logic I was always missing!
So what's the bad news? FireFox and Safari don't allow IndexedDB within different domain iframes. That Safari config may allow it, but there's no corresponding config in FF. This is a true bummer, but I'll take Chrome, IE 10, and IE 11, over Safari and Firefox.
The best news is that the architecture will be much cleaner, codified into IndexedDB stores ("tables" in SQL parlance). The database schema will actually offload much of the logic currently handled by each bridge. For example, IndexedDB will obviate the need for network messages, since each bridge can access the same record of changes.
These are the tables and columns I envision thus far. (It's been some time since I've used SQL lingo, so forgive my rough edges.)
gothy
commented
9 years ago Glad to hear that starring this repo was inspirational :)
bemson
commented
9 years ago Below is the new sequence diagram for SubEtha, using IndexedDB. It mentions Promises, since the SubEtha-Client will be undergoing changes to support them (see bemson/subetha-client#3).
This new architecture will rely on IDB for unique ids. That is, both bridges and clients will get their unique ids from the ones auto-generated by IDB. This should save a few bytes and processing, since no guids are required - the ids no longer need to be random, since they are canonical to the IDB object store indice.
However, in order to prevent unsanctioned messages, where one client fakes it's id, a local index must also exist. The local index will simply increment ids of SubEtha clients in the browser. The host will map local and "global" ids (for lack of a better term), using the local one for public consumption and the global one for bridge communications.
Clients that no longer have a synced local id, will be removed from the network.
tejacques
commented
9 years ago This is very interesting! Please let me know if you end up getting this working, and I may adopt this or a similar strategy to deal with IE10/11 iframes for crosstab.
bemson
commented
9 years ago Thanks very much for the interest @tejacques !
My efforts are split between trying to plough through the code and keeping perspective on the approach. I really feel that if I could better document the "protocol" I'm trying to codify, the simpler it'd be to code it and get peer feedback.
Below are the basic payloads I envision for this IDB variation.
I'm working on publishing this to the wiki, but docs are not my strong suit. Also, the wiki would provide a roadmap that puts standards-based encryption on the schedule - which will become more and more important to devs.
This is the minimum message structure required by the bridge (for the IDB strategy).
{
mid: <int>, // message identifier
type: <string> // message type
}From the host or bridge, requirements for the remaining message structure would depend on the following message types.
Handlers for join or event requests, would require an additional request identifier. The bridge response would use this identifier in order to link it's message to the request.
{
mid: <int>, // message identifier
type: <string>, // message type
data: { // message payload
rid: <int> // request identifier
}
}Note: The wiki will go into details for both request types.
A bridge response adds to the host's request structure. To link a response to a request, it would reference the same identifier given by the host.
The "status" field allows for sending HTTP-style response code, so clients could glean more details about the result (specifically in cases of failure). I have yet to list these codes, but they're occurring as I find exit points in the code (e.g., while parsing an incoming message).
{
mid: <int>, // message identifier
type: 'rsp', // message type
data: { // message payload
rid: <int>, // request identifier
ok: <bool>, // request result
status: <int> // response status
}
}Okay - not the update I'd hoped to post, but I am making progress in planning the work for this bridge refactor. I'm making changes, but have yet to publish my branch since the code isn't in "commitable" shape, right now.
So, despite having sequence-diagram-fatigue, below is my latest understanding of how this architecture will play out with IndexedDB. Of note, I've added details on cancelling a join request - previously called "auth" request. There are some other linguistic changes, like calling client events client messages. But, the basic flows are the same.
What I would like to do soon (besides progress in actually coding this), is reconcile these flows with the various message types.
bemson
commented
8 years ago It has been an uncomfortably long while since updating this thread. After so much support and interest from peers, it felt prudent to capture something now, before addressing the next challenge. So, forgive me in advance for this laborious review and reflection of the past ten odd months of frustratingly slow progress.
As one would hope of any self-respecting technology article, there is a SubEtha Bridge demonstration. The demo is very rudimentary: rendering but a single number on the page. That number reflects how many other bridges are on the veritable localStorage "network"... Riveting stuff, isn't it?
To see it in action, load the url in multiple windows (tabs, are fine, but far less interesting). As you open and close them, each window will increment and decrement it's number. Latency aside, the synchronized updating of separate windows shows how they are in fact linked.
Here is a short screen capture of the demo, using four IE 10 windows.
Though it uses my Bridge module, the demo is meant to example how localStorage can be used in IE, for inter-window communications. The demo code actually co-opts a specific version of the Bridge module, and fools it to work outside an iframe and with a mock Client module.
Because this demo is not in an iframe, it should work in Firefox and Safari browsers. However, the larger SubEtha project will continue to target Chrome, IE 10, and IE 11 only. (Respectively, by policy and default, Firefox and Safari do not allow storage access in third-party iframes.)
IE Edge was the outlier for this solution. The approach may work with non-iframed, same-origin windows in IE Edge. Unfortunately, for my purposes, IE Edge is a lost cause...
Storage events offer the ability for separate browser instances to observe changes to their shared, origin-based localStorage items. The event would describe when and how a given item (or "key") changed. Withstanding Firefox and Safari's paranoid security policies, IE is the only browser to get this wrong.
IE's infamous, abysmal, storage event implementation provides zero information on the nature of a storage change. At best, IE storage events inform you that a change occurred. The result is out of sync logic and an impossibly disconnected "network".
Without useful storage events in IE, localStorage should be treated as the mundane (albeit persisted) hash object it always was. Normal best-practices can be applied, as if you were dealing with any old shared object. Also, considering that this hash will be touched by several untrusted routines, you might then imagine measures to monitor changes made by other routines.
Once it became clear that I could never artificially broadcast localStorage changes to other windows, figuring out a way to communicate became more straightforward (less "feature magic" from the browser). I used key additions, updates and removals, to send messages between windows.
In IE, for any routine that uses localStorage, you should use standard namespacing. The difference is that each namespace would need to be wholly unique, in order to avoid sharing keys (akin to namespace collisions). In my module, bridges generated a unique/random id, and prefixed their storage items with that id.
By not sharing keys, you avoid data racing. A data race is similar to a race condition, except it's the data which is not concurrent (not the execution). For instance, multiple windows writing to the same key will leads to data loss, since each window can only read the latest value.
The magic of storage events ameliorated these data race/loss scenarios. Logic no longer relied on the current value, but transitional one provided by the storage event.
With localStorage - an ever changing shared hash - you will need to observe object member changes yourself. However you implement it (i.e., the events you define and their payloads), this boils down to periodically reconciling keys with some cached value.
That effort led to a monitoring routine and some custom events. My work was tweaked for my module, but it could be extracted as a separate module.
Monitoring any object this way is absolutely non-performant. But it does offer the next best thing to native "storage events". The primary difference being that you can only catch some changes with this method - storage events (are meant to) capture every transition of a storage item.
So, to measure bad and good news, this is not a way to monitor fast changing storage items. Nor does it guarantee that a changed or removed item will be observed at all. More so, the term "fast" is relative, and complicated by browser-specific i/o details that I loathe to dive into here. (Not that I know about them, mind you - just that low-level details always have a long tail.)
Ultimately, in order for this error prone approach to work, you'll need to adjust how you use localStorage entirely...
With a poor-mans object monitor, it's best to change values as little as possible. So instead of capturing an object in localStorage, capture changes to the object's state. Said another way, you want to capture your custom storage events (that you don't receive in IE) in localStorage.
For example, here's how you might trace the on/off state of a light-bulb over time. Instead of making a key reflect the bulb's current state, you would add keys which reflect changes to that bulbs state.
| Key | Value |
|---|---|
light-<guid> |
{"watts": "60", "room": "den", "created": 12310920} |
light-<guid>-change-1 |
{"on":1,"ts":12345678} |
light-<guid>-change-2 |
{"on":0,"ts":12346678} |
light-<guid>-change-3 |
{"on":1,"ts":12347678} |
light-<guid>-change-4 |
{"on":0,"ts":12348678} |
By designating a key-naming pattern that identifies the bulb's static characteristics, you can effectively communicate it's dynamic characteristics to other windows. In this example, changes to the on/off state are sequenced in the key name. The actual value further communicates the timing of the state change.
With this approach, any slower performing windows can catch up to every transition you expected with normal storage events.
Regarding key-naming patterns, it's important to use a scheme that prioritizes the messages you want to communicate between two windows. Continuing with the bulb example, a bulb that breaks, might use the key break-<guid>, where the actual "guid" matches a key named light-<guid>. Before or after reconciling which keys changed, you would then process broken bulb events before bulb change events. The importance of these details depend on your goals.
This approach doesn't mean never remove a key. But it does mean that you want to design how your system should respond when certain keys are removed. Some keys won't matter - like the bulb state change keys. Some would - like the bulb identity keys.
In my case, I used the removal of a value to communicate meaning to other windows. My particular IndexedDB cleanup process required the aid of the remaining Bridges.
When a window closes, the Bridge's identity key is set to "0". Other bridges see this, then line up to determine who will clean up the expired bridge. Once one of those bridges finish, it removes the identity key of the expired bridge. The removal of the key informs the bridges that the deed is done.
In this way, I update a bridge's value once to communicate a binary state change: alive or dead. Then I use the removal of the key to further communicate to bridges that the expired bridge has been cleaned up.
While this strategy for addressing IE's localStorage shortcomings works, there are a few surprises that I wanted to relay as well.
IE 10 and IE 11 are completely different, when it comes to localStorage events: One has them, the other doesn't. My poor shim had to use different techniques for assessing data changes.
With IE 10, I let the events trigger scans. However, since each write results in several events, I chose to debounce them a bit. I will likely throttle them in the future, to avoid having perpetual events preventing otherwise timely scans.
In IE 11, there are no events to speak of, so I simply scan localStorage at a reasonable interval. Because scans are costly, I plan to make that interval dynamic, based on the frequency of changes encountered.
I must warn against these two methods of reading localStorage keys in IE: Object.keys(localStorage), and for (key in localStorage) {...}. Using either will likely result in missing keys that have been added by other windows.
The safest way to retrieve all localStorage keys/values, is to use the .length property along with the .key() and .getItem() methods.
var
ln = localStorage.length,
cache = {},
key
;
while (ln--) {
key = localStorage.key(ln);
cache[key] = localStorage.getItem(key);
}Even with the recommended approach for reading data, I've been able to retrieve a key who's value is null. The ability to read deleted keys, is perhaps the most notable, damning evidence that IE's localStorage implementation sucks cow balls.
So don't assume that a key's existing means it is valid. A simple string-type check should separate the dead from the living. This will avoid emitting an update-like event on keys that have actually removed.
The most surprising discovery came from observing that some windows simply stopped observing localStorage changes. Like forcing a CSS redraw, localStorage must also be kicked in to gear, so it's data stays in sync.
Though an expensive operation, syncing was mercifully simple: just write something to localStorage. Thankfully, so far, I've only observed this in IE 10.
function ieKickStorage() {
localStorage.setItem('somekey', 1);
}Based on my experiments, neither the key nor value has to change between writes - which is somewhat counterintuitive. I'm unsure if removing a key would have the same effect.
In my Bridge module, I "kick" localStorage every 3 seconds with the same key and value. I plan to add some finesse this, like resetting my interval after actually writing data.
Firstly, this issue is not solved in my opinion. However, my progress does allow moving on to other topics. The lack of IE Edge support will likely be addressed separately as a new bug. (Hasn't this one been through enough?)
Secondly, I'd like to put time towards a demo that doesn't include my Bridge module. It's really a distraction from what was learned here. By extracting my localStorage monitor, one could easily write logic and a key structure that monitors the addition and removal of windows... Any takers?
I grew fond of the diagrams I created in this thread, and have updated them - but they mainly describe SubEtha, not the Bridge. I hope to post them in the SubEtha wiki (which is empty now).
I also need to update the Client module to work with the new Bridge module. While these tasks require time, neither should take too long.
Thanks to the following individuals for seeing me through this insane issue. At some critical moment, these folks offered their time and attention, if not enthusiasm and interest in this issue. For that, I thank you - in order of your user name:
ryangiomi
commented
8 years ago Great write up, and congrats! Had a laugh about IE and cowballs. :D
--removed quoted post (bemson)--
bemson
commented
8 years ago Thanks mayne. More to come! Enjoy the Halloween weekend!
--removed quoted post (bemson)--
puzrin
commented
8 years ago @bemson , i'd say some solid buzzwords like "eventual consistency" and "append-only oplog" will be good here :) . At least that will make more clear, where to steal borrow dataflow principles from (some DB design lectures).
bemson
commented
8 years ago Good to know @puzrin . These are definitely things I don't know of, and they capture the message well. If I ever get time, this will go into a wiki/blog somewhere. Anyone is free to repurpose this thread for the greater good!
puzrin
commented
8 years ago @bemson before reading your posts i could not understand, that familiar patterns are ok for our case :) . Thanks for your work.
Bad news is that implementation requires a bit more that couple of code lines. Good news are that things become more clear & predictable:
tejacques
commented
8 years ago @bemson really interesting. I did something very similar to try and get a polyfill for IE8 StorageEvents -- IE8-EventListener. It's nice to see a lot of similar concepts going on which validate some design decisions I made there. I really hope you didn't have to spend as much time messing around figuring out the details there as I did, but I suspect you may have spent more.
That said, I re-read most of this thread, and noticed you had made some claims about IE9/10/11 with regards to StorageEvents firing and their contents. IE9-11's StorageEvent do have the standard key, oldValue and newValue on them, and running your jsbin reader and writer I was unable to get it to exhibit any dropped events on any of IE9/10/11, however because you're reading the value from localStorage (which hasn't necessarily updated yet by the time the event is received in IE) it shows the previous value. If you instead log the event's value, it gives the proper one data. This can be done with this snippet:
window.addEventListener('storage', function(e) {
console.log(e);
});And that works in IE9/10/11. I've only been able to get the issue of localStorage events fire twice or not at all in iframes to happen in IE11 -- I haven't replicated the behavior in IE9/10. The issue of events firing twice happens when localStorage is written to from an iframe, and the event not firing at all happens when an iframe points to Domain B with parent frame Domain A writes to localStorage.
The biggest obstacle I've encountered with using localStorage as a changelog in IE is that it can become expensive to look through every key in localStorage every time. If you have several tabs/iframes/windows open it quickly adds up. I've noticed that in IE, the relative order of keys returned by index from the key function does not appear to change if you add, remove, or change a value in localStorage. Because of this, I believe there is a lot of room for performance improvements by binary searching for a sentinel-value and reading new events from that marker.
bemson
commented
8 years ago @puzrin , my strategy has been to use LS for window registration and IDB for messaging. I do share one key for messaging, since all windows would then only have to inspect the latest IDB entries to catch up.
@tejacques , two points... ok, three.
key, newValue and oldValue properties??!tejacques
commented
8 years ago @bemson
bemson
commented
8 years ago @tejacques - I'm completely befuddled by your findings, since they're counter to this 2009 article. You wouldn't be talking about your library's approach, where you retrieve the current value and capture it as the new storage item's value, right?
That said, I'll be sure to write a jsbin that simply dumps the storage event (from another window). I'll then place that in an iframe of another jsbin page, to see if the event details are still present. Unfortunately, I've got paid work to finish first! (I should have an update over the weekend.)
tejacques
commented
8 years ago IE8 was released March 19 2009, IE9 wasn't released until March 14 2011. Since that article is from 2009 and mentions IE8, I've got to believe it's referring specifically to IE8. I do want to stress however that a method similar to what you've written is necessary with IE11 to be fully compliant when working with iframes.
bemson
commented
8 years ago @tejacques Sweet ba-jesus, thank you! You're right in that it doesn't remove the need for my findings, but it feels good to know that IE10 might be more standards compliant than I'd thought. Plus, I wouldn't have to fork my code as much as I have already. I will definitely research this further!
bemson
commented
8 years ago @tejacques , IE9 & IE10 show no poor storage event objects or performance. It's amazing how my confusion led me to down such a dark path. Even my first on this topic got it wrong... Geez.
I wrote two test files on cssdeck.com. One writes to storage, and the other captures storage events. I even allowed for the event to be captured from the document or window - since that confused me as well.
Playing with these two pages, it's clear that IE9 and 10 do emit proper storage events! The capturing page shows how all the necessary keys are present. Woot!
I then, embedded these pages into jsbin.com, so as to mimic a third-party iframe scenario. The iframe writer and event listener both also work in IE9 & 10.
The cssdeck.com pages don't work in IE11; the user content is actually already in a different domain iframe (due to their setup). Therefore, I tested non-iframed jsbins versions of the same storage writer and event-listener. Thankfully, IE11 did fire the expected storage events when on the same domain.
Finally, to test polling IE 11 and iframes. I wrote a polling storage reader test (via cssdeck.com). It shows that IE11 can read localStorage via third-party iframes - but still doesn't receive storage events.
Ok, so IE 11 (and perhaps Edge) are the outliers to this problem. That is better news than before, and thank you for catching my error. On the down side, the lack of events in IE11 still means plenty of code forking. Meanwhile, I'll update my IE10 implementation, to rely on the standards-based approach, going forward.
Whew! Are we there yet? :-p
puzrin
commented
8 years ago @bemson see https://github.com/nodeca/tabex/labels/note for our records. Problems are with IE11 (cross-domain events) and iOS (timers + private mode). Also, probably you should know that iOS has problems with IndexedDB, but is ok with deprecated WebSQL.
bemson
commented
8 years ago @puzrin , thanks much. I'm aware of iOS and private mode, but my library (rather, my intended use of it) targets desktops and separate windows. If my project ever did target iOS, the IndexedDB note is good to know.
bemson
commented
8 years ago Ok. So, for my own sake, I checked the demo on IE11. The polling is definitely not great, but latency can be adjusted along with expectations. Anyway, here it is as an animated gif again.
To be clear, this is what would be required of my library, which relies on accessing localStorage via different-domain iframes.
puzrin
commented
8 years ago The polling is definitely not great
You can initiate polling by both timer and "update" event (when available). Problem is not with polling, but with lack of effective oplog to replay events.
bemson
commented
8 years ago Last note in this, before my early weekend starts: IE Edge only fires storage events to iframes within the same domain and document. While no storage events are fired to other windows. Other windows seem to begin with a value, after load (and reload), neither re-reading localStorage or polling seem to work - iframe or not.
tejacques
commented
8 years ago You should submit it as a bug to Microsoft. They are much more likely to fix it in Edge than IE11, and if it's the same code fix they may backport it.
tejacques
commented
8 years ago @bemson just to be clear and so I am sure I understand correctly, what do you mean by "iframe or not" here
bemson:
neither re-reading localStorage or polling seem to work - iframe or not.
Does that mean if you have a tab with Domain A inside of which is an iframe to Domain B, that if you open a second tab to Domain B it will not receive Storage events or updates to localStorage from the iframe in the first tab?
bemson
commented
8 years ago @tejacques, in Edge, my module couldn't read the latest value of a storage item, whether read from a top-level or framed window. Reading from the iframe only worked if it was hosted by a window that made the change.
The oddest thing is that even new windows would not see the latest value. This sequence of window storage actions, should illustrate what I've observed so far. (Windows A, B, and C have the same origin.)
| Window | Key | Action | Result |
|---|---|---|---|
| A | foo |
write 'bar' | |
| B | foo |
read | 'bar' |
| A | foo |
write 'zee' | |
| A | foo |
read | 'zee' |
| B | foo |
read | 'bar' |
| B | random |
write 'kick key' | |
| B | foo |
read | 'bar' |
| C | foo |
read | 'bar' |
Basically, even with newly created windows, if the key is set by another window, it would only ever receive the original value. It's as if changes to a storage are not flushed to disk - only captured when first created.
Unless someone beats me to it, I do intend to verify and file a bug for this behavior - over the weekend.
tejacques
commented
8 years ago Ok wow, so it's just that localStorage is completely broken in Edge. Here are some issues open, but none give a great description:
https://connect.microsoft.com/IE/feedback/details/1694027/storage-event-not-firing-after-a-change-of-local-storate-windows-10-edge-browser https://connect.microsoft.com/IE/feedbackdetail/view/1798743/localstorage-bug
Edit Update: This stackoverflow post has some more details: http://stackoverflow.com/questions/24077117/localstorage-in-win8-1-ie11-does-not-synchronize
The stackoverflow post is pretty revealing -- it shows that on Windows 7 IE11 works just as expected, however on Windows 8.1 IE11, localStorage does not receive updates on other tabs, and on Windows 10 Edge the same is true (consistent with your findings). Since Microsoft's browsers can rely on the OS for some features (example: WebSockets) it stands to reason they are relying on some OS feature for localStorage which is not working correctly in Windows 8.1 and Windows 10, causing Windows 8.1 IE11 and Windows 10 Edge to break.
This makes the IndexDB approach you came up with the only viable strategy currently for Windows 8.1 and Windows 10.
bemson
commented
8 years ago Yes @tejacques, that's pretty bad and neigh impossible to fix - considering how the OS is a likely culprit. (Plus, it seems the IE 11/Edge quirks are throwbacks from IE8.)
Unfortunately, my Bridge module does not use IndexedDB for networking windows, nor the "peers" they create. Instead, I only use IDB for relaying messages. The registry of "bridges" and "peers" is wholly maintained in localStorage.
I did try to use IDB only, but the async logic and numerous failure-paths proved very difficult to code (at least, without Promises). As well, there were other goals for my API and protocol's design which felt best served by simple(r), local, synchronous logic.
Now, in hindsight, I may have to return to a network that uses storage events as a signal to read IDB. (I'd poll localStorage in IE11, and cookies in IE Edge.) I wanted to place my network map in one browser feature, rather than fork it for localStorage and IDB... A silver lining may be to depend on a persistence normalization library, like Storage.js)... It's all bad, so far. :-(
puzrin
commented
8 years ago @bemson are there any requirements for msg delivery latency between local tabs? Need some info from the real world for investigations. My projects have no special requirements, and even 1000ms is ok.
bemson
commented
8 years ago My project has no latency requirements either @puzrin . My goal in addressing IE is message integrity - ensuring that what is sent by one client can be delivered to it's target(s) - also, of course, in the intended order.
I'm not exactly the most disciplined coder, so I have used localStorage (instead of IDB) to avoid the many negative async paths that are common to networking logic, like canceling a request to join a network.
I had originally planned to use IDB as a network registry, but it proved challenging without using a third-party library (which I loath to add due to footprint and performance concerns). As it stands now, I've reduced the number of IDB stores from 3 to 1, which has been a good thing so far...
For now, I'm trying to wrangle localStorage in IE10 & IE11. Despite @tejacques timely reminder, IE10 has some edge case behavior that shows things aren't working quite as expected still. I hope to update this thread more often with test links, etc, to demonstrate my findings.
tejacques
commented
8 years ago I put a large amount of effort into testing this last night. Still need to go over some edge cases in IE10/11, but I will update with my findings later.
One important thing however is that the latency for Edge (which I found was synchronizing) was extremely poor. It took 3s for changes in one window to reflect in another, meaning a 6s latency.
bemson
commented
8 years ago I haven't had much time to put towards this, and it seems I'm back in "survey mode" for IE. But, I completed some test pages that assess how well localStorage reads and writes, in various contexts.
I'm not sure if the dabblet link expires - I signed up, hoping it would not.
It's tedious work, but I hope to provide a table of results for each test setup, per IE browser (10, 11, then Edge, across Windows 7 thru 10). Each page would be tested against itself, in another window. Then, the page would be tested alongside two windows for the other urls.
The test permutations are mind-numbing. But it feels like the only way to get a handle on this beast and provide the best depth of insight for developers and the IE browser team. I hope to update this thread with some results sooner, rather than later.
bemson
commented
8 years ago Ok, I didn't document my findings, but it looks like all IE browsers and platform combinations will see changes to localStorage in other windows - be it a same/different-origin iframe. Thats the good news... The bad/expected news is that storage events won't always work, based on the combination of browser and platform.
In short, without events, you'll have to do a few things to get localStorage to work:
There is still worst news that I have yet to corroborate, regarding performance thresholds where none of these techniques seems to work any more. Another big hassle will be feature-testing the browser, to determine what approach to take. (I've been comfortable with UA checks, so far.)
I'll work to produce that table of results during this week, to corroborate my observations.
bemson
commented
8 years ago I finally have some data to share.
I've compiled two tables, representing two browser windows, in various IE and Windows OS versions, for three scenarios: a frameless page, a same-origin iframe, and a different-origin iframe.
One table is for the window that is observing localStorage changes (the "observer"). The other is for the window manipulating localStorage (the "writer"). Here is an explanation of methods, used to observe/retrieve localStorage.
Below lists how an observing window fared while tracking localStorage changes (by another window).
| Browser | Platform | Frameless | Same-origin | Diff-origin |
|---|---|---|---|---|
| IE 9 | Win 7 | event | event | event |
| IE 10 | Win 7 | event | event | event |
| IE 10 | Win 8 | event | event | event |
| IE 11 | Win 7 | event | read | kick |
| IE 11 | Win 8 | event | read | kick |
| IE 11 | Win 10 | event | read | kick |
| IE Edge | Win 10 | kick | kick | kick |
I corrected IE 11 + Win 7 (again): the "diff-origin" result is "kick", not "read" (sheesh)
Below captures how the writing window fared in tracking it's own localStorage changes. Note, because the window writing to localStorage should not receive a storage event, this isn't terribly useful information - just intriguing enough.
| Browser | Platform | Frameless | Same-origin | Diff-origin |
|---|---|---|---|---|
| IE 9 | Win 7 | event | event | event |
| IE 10 | Win 7 | event | event | event |
| IE 10 | Win 8 | event | event | event |
| IE 11 | Win 7 | event | event | event |
| IE 11 | Win 8 | event | event | event |
| IE 11 | Win 10 | event | event | event |
| IE Edge | Win 10 | read | event | event |
For comparison, a compliant browser would state "read" in all scenarios. (Sigh)
Surprisingly, under the right conditions, IE 11 and Edge will support storage events. -- Ok, scratch that. I had made a clerical error and falsely stated that the observing window received events in IE Edge. (I've since updated the observing window's table.)
Either way, it's mostly true that IE can support events - it's just that the event is in the window writing the change - which is partly useless for cross-window communication. IE Edge only supports accessing writes by another window, after kicking localStorage into gear and reading the stored items.
It's also still true that forking approaches relies on more than one data point (e.g., the browser version alone is not enough). In my mind, this means two things:
To date, I've been reticent to fork my own storage-shim using feature-detection, but the usefulness of native storage events is too compelling. As I feared - because I'm lazy - I will have to create a separate storage-events shim for localStorage.
Everyone is welcome to review my localStorage test pages. If I'm doing anything wrong or unadvisable, do let me know so I can capture and share more accurate information.
bemson
commented
8 years ago So, I tried my hand at feature detecting storage events. This page is supposed to conclude the same event/read/kick result as my manual research demonstrated.
In short, it seems that a lone window & script can not determine a browser's storage-event capabilities. The issue is that IE's localStorage behaves differently between iframes, than it does between windows. This insurmountable fact means a single script won't be enough.
That said... I will try one more test, which attempts to see if sibling iframes behave like sibling windows. My theory is that the storage event bubbles up (from child to parent frames), to the root window. Perhaps then, a single script can monitor two iframes, the way I monitored two windows.
If all that fails - which it likely will - I'm going to forgo a separate localStorage fix/module/repo. I will then add to the browser sniffing logic of my Bridge module, so it at least uses the best storage-event approach for a given IE + platform + framed/origin combination. It won't be an elegant solution, but it will hopefully solve this issue.
bemson
commented
8 years ago Oh well, having one iframe listen to another (in the same page) doesn't match my results either. It looks like a browser's storage event behavior can't be feature-detected via a single page.
I'll double check my findings earlier, then use the best localStorage strategy based on IE version, origin, and platform... UGH!
After a slow bout of not-well documented testing, I've answered the following questions involving iframes, different domains, and localStorage. The questions address functionality that a browser platform must provide, in order to support these SubEtha features:
Can the iframe access data in same origin?
Safari turned out to be the odd-browser out. Safari's default settings produce a kind of sandboxed storage that isolates iframed pages, even when pointing to the same origin. This behavior is due to user settings, and has no apparent workaround.
Can the iframe receive storage events?
Safari results are again blamed on the default user setting. All versions of IE require a workaround.
IE 8-10 provide empty storage events with zero relevant details. However, their occurrence is enough to manually track of changes, vs. receiving them in the event.
IE 11 does not provide storage events to different domain iframes. Any solution will require polling.
Does the iframe receive an unload event when the parent navigates?
Mercifully, all browsers trigger the unload event when the parent page navigates.
Does the iframe receive an unload event when the node is removed from the parent DOM?
IE 8 simply does not provide this event, when the parent DOM removes the bridge's iframe. Having the parent monitor the DOM is not a solution, since it is the bridge (i.e., iframed document) that must observe the unload event, in order to perform it's clean up routine. This finding effectively eliminates IE 8 as a SubEtha compatible browser.
Next steps?
Given my tests, it's clear that Safari and IE 8 can not support SubEtha. As well, workarounds must be coded for IE support. This bug will track progress in developing the IE workarounds.