Meta Issue: Chrome Export Issues Update

[cycomachead]

Ok, I think we should have a new discussion about fixing the exporting issues on Chrome -- that is, clicking "Export Project..." with > 2MB of data leads to the "Aw, Snap!" view, causing lost work.

This currently is also the case with the "Export Summary..." feature.

• We should do something about the crash on existing browsers. (We really should have done something long ago...) See #537
• Chrome 47 - current dev channel (Chrome Canary version doesn't crash!) Woohoo!! However, now there's a bigger issue: When using long data URIs in Chrome 47 there is no "Save As..." function on even though the page loads Some things that generate massive URIs, like the all script pics feature, no longer crash Chrome, but end up displaying only a blank page.

The other good news is that since summer I've been messing around with the library FileSaver.js which gives a good fallback using Blob URIs and gracefully reverts back to existing methods when they don't work. While I know using a library generally isn't acceptable, FileSaver implements 1 function saveAs which handles downloading files. So far I don't see much point in pulling code out of FileSaver.js because they've gone through the hard work of identifying and working around browser bugs -- so redoing things ourselves seems to be a waste of effort IMO. In my hacky test, I overwrote window.open() with a call to saveAs and things worked just fine.

FileSaver: https://github.com/eligrey/FileSaver.js

For Reference:

433 #726 #749 #883 #433 #964 (would likely be fixed by this, but haven't confirmed) #123

[cycomachead]

@jmoenig @brianharvey

Thoughts?

It has occurred to me that having a consistent saveAs() method could also benefit other uses of Snap! such as the iPad app, making it easier to customize grabbing files, or doing anything special.

[jmoenig]

My hunch is just taking the block uri part would solve this issue, and we should just do that. This - like many other related issues - is particular to Chrome. I don't want to change the design of Snap because Google decides to suck :-)

[cycomachead]

The thing is though, that's basically what the library does... Just that there's a lot of bugs still, even in other browsers, of course.

I mean, I can totally just cut out parts of the library, but all I'm doing is making something that's harder to maintain and doesn't handle the edge cases...

[jmoenig]

Ah, I'm not sure about the "hard to maintain" part. Introducing dependencies on 3rd party software has huge downsides in the long run. Personally I prefer to roll our own...

[cycomachead]

Hard to maintain in the sense that we are reduplicating effort, and a library like FileSaver has a bunch of checks for known bugs... In this case the library literally only does 1 thing. The library is < 300 lines, and many of those are just comments formatting.

The only real issue with the FileSaver library is that it doesn't support browsers which don't have the Blob API, but that's an easy thing to account for, where we just use our current solution.

[jmoenig]

(rant ahead)

Let's take a step back and look at the bigger picture here. What we're really trying to accomplish is to hack ourselves out of the browser sandbox. We want Snap to store programmatically generated data on users' local computers so they can use it outside of the browser. The whole point of the JS sandbox is to prevent exactly this, that's why we're having such a hard time. We're fighting the very design of browsers. It's clear that we can't win, no matter which JS library we use, because JS is the problem. It's also foreseeable that any loophole we might exploit can be disabled at any time by any "security update". Blob URIs are such a loophole.

I don't want to spend effort to find and maintain workarounds to workarounds for designed browser limitations, because that's precisely what'll lead us into limbo maintenance-wise.

Currently it seems to me that the only way to really address the issue of getting programmatically generated data out of the browser is to upload it to a server (using XHR2) and then to download it from there using another webpage that has a "user-clickable" download link.

The good news is: We have such a server, it's the SnapCloud! We can totally make a webpage that lets users download their projects from the cloud directly to their computers without having to open them first in Snap. I'm slightly (that's an understatement) annoyed that nobody so far uses our SnapCloud API in other web pages, e.g. to let users manage their accounts, including their passwords, shared projects etc. etc. Thing is, let me stress this: We have an API for that! it's all just REST. I'm actually having a hard time to restrain myself from ranting about this even more. We're supposed to be programmers, computer scientists even, not just web-surfers shopping for scraps of code.

The downside, of course, is the "airplane" scenario. There are solutions for this, too. One would be a "companion app" in, say, Python that either simulates the cloud API or offers its own one. Another is to simply use localstorage until you're back online. That's a very sensible use of localstorage, if you ask me.

Rather than surfing the web for "loophole" workarounds to designed browser limitations let's solve the problem! Let's start by making a webpage that lets users manage (share, unshare, delete, rename) and download their own projects from the SnapCloud! We can do this right now, we have been in a position to do this for more than three years, we have all the tools we need for this.

And let's not forget, that in the meantime our current "export" kludge continues to work in every browser except Chrome. Wait! that's not even right, it does work in Chrome, just not for very big projects.

I think we can totally solve this problem in a good, sustainable way that doesn't fight the browser sandbox but embraces it.

[cycomachead]

Rant as well...

We're doing a lot of things we aren't supposed to be doing. We're abusing HTML5 Canvas to make Snap! work. There are so many student issues from confusion / frustration around the fact that even things like text boxes don't work normally. Fallbacks and work arounds are a life of the web. Things like FileSave.js exist because there are incompatibilities and some stupidly and lack of caring by browser vendors, but mainly we are just dealing with a bunch of versions of browsers where it's impossible to expect complete consistency.

The good news is: We have such a server, it's the SnapCloud! We can totally make a webpage that lets users download their projects from the cloud directly to their computers without having to open them first in Snap. I'm slightly (that's an understatement) annoyed that nobody so far uses our SnapCloud API in other web pages, e.g. to let users manage their accounts, including their passwords, shared projects etc. etc. Thing is, let me stress this: We have an API for that! it's all just REST. I'm actually having a hard time to restrain myself from ranting about this even more. We're supposed to be programmers, computer scientists even, not just web-surfers shopping for scraps of code.

Nobody but a handful of us know that an API exists, and nobody outside of Miosoft really knows how it works. Because it's really freaking confusing...there's no documentation, no source for help...I have built a page to get public projects from the Snap!Cloud (https://alonzo.herokuapp.com/snap-proj.html)

Meanwhile what I've learned about the Snap!Cloud is very confusing...it's sort of rest, but there are no routes like a traditional app, no clear list of what functions are available. Miosoft doesn't have documentation on...anything...online that I can find. Until just a moth ago, the cloud has CORS restrictions to all but a handful of domains making it very impractical to test something. It's all reverse engineering cloud.js and inspecting headers and things...some of it is actually cool to do, but not if I'm trying to build something reliable. Everyone is annoyed that this doesn't exist, but no one has ever setup a position to support this work. It's not just a simple project, it's a cool project, but it takes time and dedication and a lot of help or documentation. It can't just be a side project.. Besides that fact, there's no clear definition so far of what priorities there are for a site, or even a discussion of how we go from miosoft apis to comments, and forums, all that.

The downside, of course, is the "airplane" scenario. There are solutions for this, too. One would be a "companion app" in, say, Python that either simulates the cloud API or offers its own one. Another is to simply use localstorage until you're back online. That's a very sensible use of localstorage, if you ask me.

Yeah, desktop apps should be a thing, but that's besides the point. And yes, that's a good use of local storage except that it needs to be managed in a way that's clear to users, like letting them easily move projects, or making it clear what happens when there is a size limitation.

Currently it seems to me that the only way to really address the issue of getting programmatically generated data out of the browser is to upload it to a server (using XHR2) and then to download it from there using another webpage that has a "user-clickable" download link.

And this is such an annoying solution for when you just want to download something that's already on your computer....It requires TWO transfers of your project, which is slow when you're in a lab with a terrible connection or you have a big project. Still, I'm willing to concede to using this if there's a better Snap! cloud site.

And let's not forget, that in the meantime our current "export" kludge continues to work in every browser except Chrome. Wait! that's not even right, it does work in Chrome, just not for very big projects.

Ugh, it barely works at that! I have lost count of the number of instances where we have lost student code due to a bug which we can entirely prevent! I've lost count of how many times I've hand fixed student files in a variety of situations that demonstrate exporting and saving is confusing, or how many Piazza posts and other notes I've read or edited about how to save your files without losing work, or just to even do it correctly.

I think we can totally solve this problem in a good, sustainable way that doesn't fight the browser sandbox but embraces it.

In this case we are using the native APIs given to browsers, just every browser has a different set of APIs available.

Blob URIs are such a loophole.

I don't want to spend effort to find and maintain workarounds to workarounds for designed browser limitations, because that's precisely what'll lead us into limbo maintenance-wise.

These are not loopholes. Blob URLs are a defined W3C spec. But like everything on the web or even using computers in general...there are cases for bugs and using work arounds. No user cases why something doesn't work, no user cares what browser they're using. No one cares about our code. These work arounds don't lead to limbo is they're documented, and if there are test cases.

We're supposed to be programmers, computer scientists even, not just web-surfers shopping for scraps of code.

Ah, come off it! I suggested this library after having done many hours of research about the problem, and reading through the code (multiple) times. My issue is not that I am incapable of re-writing it -- I am...it's not terribly complex code, but some other groups of people sharing the effort to keep things working. We could do everything ourselves...but I feel fairly confident that Miosoft has not written it's own TLS/SSL library, or is using 3rd party code for a number of things. I'm not saying we go an replace morphic with something else, but you are always at some point relying on someone else's code.

But really -- we have the ability to fix significant user-facing issues today long ago really, and to prevent some potentially more confusing ones from cropping up in the future...

[jmoenig]

We're supposed to be programmers, computer scientists even, not just web-surfers shopping for scraps of code.

Ah, come off it! I suggested this library after having done many hours of research

Oh. I'm sorry, Michael, I really didn't mean to address you personally here, I did mean "us". I appreciate your research into blob uri's and didn't want to discourage it.

I'm well aware that my policy of not allowing 3rd party libraries and trying to do everything in my own way mostly from scratch is contrary to what is considered right or even desirable in industry. In developing Snap there are tensions between wanting to meet users' expections, keeping the code maintainable and wanting to enable new experimental user interface ideas. On one hand Snap! has become production software, on the other hand Snap! continues to be something of an experiment. This causes conflicts, I know.

To support something like Snap in production you'd typically need a much bigger team. Not just developers, but also help desk people, a manager, front-end coders, back-end architects, graphic designers, UI specialists that know about standards etc. etc. I mean, look at how many people are professionally working on Scratch, AppLab etc. Granted, those might be (much) bigger projects, but scaling the user base is only a part of the effort. Since there always is fluctuation you'd also need more and more rigorous documentation on everything, beginning with the architecture, all the way to the cloud API. Plus, it's generally considered good practice to use standard tools and libraries, so new team members can hit the ground running and find help online. Also, projects of this size typically won't be handled as "side projects", but require a dedicated team of maintainers.

However, with such good practices also comes the need to keep up. Someone needs to monitor the issues of the stack of re-used software components, there has to be a testing suite and and update stream, a QA team, compliance efforts, documentation maintenance etc.

Snap, on the other hand, is somewhat of an antithesis to all of this. It's a project much like I imagine projects used to be in the 1950's, built from the ground up, featuring some cool innovations in exchange for a fair share of quirks. Snap is an island, its architecture of self-sustained seclusion being reminescent of Smalltalk. And that's not a coincidence, because above all, Dan Ingalls is my personal all-time über programming hero. But I know that many from the industry disagree. My point is, that I love being able to get my head around all the code in Snap, and I'm a pathological control freak about the Snap code. I could go on to rationalize why this is actually a good thing, how this enables single programmers to single-handedly maintain all of Snap for a decade and hopefully longer, but it really boils down to my purely personal taste of programming. I've myself worked in the industry, read through and written my part of enterprise-scale apps, and decided that I don't want any of that for Snap. I'm not saying it's wrong to use third-party libraries, DOM elements or platform standards, it's just my personal minimalistic taste to make sure Snap as as few outside dependencies as possible, even if the price for this autonomy is making users deal with certain awkwardnesses, such as that text editing in Morphic is its own thing, not the platform's. I - perhaps irrationally - believe that I can better support Snap in the long run this way. And I'm somewhat encouraged in that opinion by the many forks of Snap by very different people on different continents, who all seem to be able to do the same thing for their own version of it.

I disagree about your contention that nobody cares about our code. I know many who are working with our code, and many more who have been looking at it at times, searching for particular details, wanting to find out how things like blocks can be made. I believe having it all in a single capsule (a Smalltalk "image", if you will), its own explorable world even (hence dev mode), is something beautiful in its own. But I really can't support any of this with hard arguments. To me, Snap is more like a novel, a painting or a symphony than a software product. :-)

So, I get that the SnapCloud API is confusing, undocumented and hard to understand, because it needs to be reverse-engineered, which can't be done as a side project. See, that's exactly what I'm talking about!

We can also just use some other server to make single projects downloadable. How about interfacing to Dropbox or some other web-hosting service?

We can, of course, also get help from either myself or MioSoft. That shouldn't be any problem. One problem I see is that even starting to write such a (Cloud account self-management) website is hampered by towering expectations of the "but-does-it-solve-the-simultaneous-remote-collaborative-project-editing-gimmick" kind. I'm afraid that such "bold" and "radical" proposals are discouraging us from actually taking the first of many steps to take us there.

[brianharvey]

I hate this argument every time it comes up, which it does on a regular basis.

Here's my view: At a conservative estimate, we have 150 teachers teaching BJC classes right now. That means we have about four thousand kids frustrated as hell because periodically their hard work goes up in smoke. And, because it's large projects that are the problem, when they lose work they lose a lot of work. This doesn't count the thousand-odd edX students.

(It also doesn't count the even larger number of students in the TEALS version of BJC, because TEALS--rightly, it seems--doesn't believe Snap! is reliable, so they're still using BYOB3.)

Given all that, I don't want to hear about programming philosophy. I want Snap! to work, 100% reliably, so nobody ever loses anything no matter what, no matter whose fault it is. If this requires kludges, then I'm for kludges; they can come back out when there's a better solution. If it can be fixed today without kludges, so much the better. But "today" is what's important.

One problem I see is that even starting to write such a (Cloud account self-management) website is hampered by towering expectations of the "but-does-it-solve-the-simultaneous-remote-collaborative-project-editing-gimmick" kind. I'm afraid that such "bold" and "radical" proposals are discouraging us from actually taking the first of many steps to take us there.

You keep saying this, Jens, but it's not true. Nobody is spinning their wheels waiting to design a simultaneous editing system. They're spinning their wheels, I'm afraid, out of plain incompetence. (Not you, Michael; the students who are supposed to be working on a new web site and who keep designing mockups instead of writing code.) And, even more, the incompetence of their manager, namely me. But in any case, the web site seems to me orthogonal to the issue of not losing data.

P.S. I do agree that Michael is probably overstating the difficulty of getting help from Miosoft.

[jmoenig]

Needing a solution today is a sure way to shoot oneself in the foot. It's not an argument either way, except to keep looking.

Come on, students "losing a lot of work" happens often? Aren't they saving to the cloud? Honestly, have any of you ever "lost" work? I don't want to come across wrong but "losing homework" and blaming "the computer", "Word Perfect", "my dog" etc. isn't particular to Snap! in any way.

Is that really the reason why some TEALS courses still use BYOB3? (I seem to remember quite a lot of photographs on facebook and twitter showing Snap, not BYOB). Or is it internet bandwith? I mean, honestly, BYOB is buggy as hell, and a lot less stable than Snap, especially saving projects and getting them back in the same state.

So, you know, I'm all for a fix, even a quick one. But not for introducing a third-party library dependency. Not gonna happen with me. Either we understand a problem and the solution for it, or we keep looking for one. Apparently blob URIs are the "proposed workaround" for Chrome's "long URL" issue, so I'm fine with using them. Can somebody explain to me what's wrong with this approach?

Do blob URI's solve the problem? Can we add a function to Morphic.js that takes a URL and converts it to a blob URI? Does that solve the problem? Isn't that a fair question? Which "other browser quirks" does that library solve? Ones that we're not aware of? How does it address those problems (e.g. some libraries rely on some plug-in or on Flash)? Does it make sense to follow the approach they're suggesting, or might there be ways better suited for Snap's Morphic environment? Aren't those fair questions?

[jmoenig]

Apparently canvas.toBlob() already is in Chrome Canary... (?)

https://code.google.com/p/chromium/issues/detail?id=67587

[brianharvey]

If a third-party library would solve the problem today, then I'm for using it today, and then, by all means, taking your time about working out the eventual correct long-term solution. But losing student work isn't, you know, "this third party library would give us a really kewl new feature." You'd be right to reject that.

We've had this problem for a year! That makes us look really irresponsible.

[jmoenig]

Obviously this library - which turns out to really be two libraries - doesn't solve any problem today, because it needs to be integrated first. And we also need to test it. Reading the inline comments it seems like there might be other issues - especially in Chrome - waiting for us. Also, some use of interval suggest there might be issues with Snap's threading model.

Let's do some tests! I'd like to find out whether it really solves all of our problems. The way I see it we need both the filesaver and the canvas-to-block libraries, right? If I understand those libraries right we can throw them out again once every browser - especially Chrome - supports the full HTML5 FileSaver and toBlob APIs. So, if it's just giving us something HTML5 is supposed to give us anyway, just a little sooner, it might be worth a peek

[brianharvey]

Obviously this library - which turns out to really be two libraries - doesn't solve any problem today, because it needs to be integrated first.

Okay, Michael, prove him wrong! :-)

[cycomachead]

Personal Background and Philosophical Arguments: My general philosophy as a programmer and a teacher is that I should make the sacrifices for the students or the users. They don't care how the sausage is made, and they just expect things to work. As an instructor, the biggest problem with Snap! is when we spend time teaching and working around idiosyncrasies that detract from the real learning. I want Snap! to be something that people enjoy using because it's awesome, and fun, not because it's the tool we picked for the class. Increasingly I'm seeing more and more TAs that are frustrated by things...

Here's my view: At a conservative estimate, we have 150 teachers teaching BJC classes right now. That means we have about four thousand kids frustrated as hell because periodically their hard work goes up in smoke. And, because it's large projects that are the problem, when they lose work they lose a lot of work. This doesn't count the thousand-odd edX students.

Come on, students "losing a lot of work" happens often? Aren't they saving to the cloud? Honestly, have any of you ever "lost" work? I don't want to come across wrong but "losing homework" and blaming "the computer", "Word Perfect", "my dog" etc. isn't particular to Snap! in any way.

Me, personally? Yes, plenty of times! (Most of my losing work comes from that damned escape key accidentally closing a block editor! Another easy-to-fix frustration.)

I have only lost minor work to the URL issue once or twice, because I know, deeply exactly what is causing the problem and I usually can guesstimate the size of a project. Also, I use Safari mostly.

However, I have seen multiple students in tears (literal tears!) over losing work. I've seen dozens of Piazza posts in various classes / settings about others who are confused. Earlier this summer I was triaging issues in Jeff Gray's CSP course. (Which is a cool course, but they shouldn't be hampered by the fact that they don't have someone who internally knows Snap!.)

I have hand edited and inspected hundreds (over a couple years), of student submissions (only CS10! Not even HS students or edX people or anyone else!) of cases where saving an XML file is hard and confusing. And you wouldn't believe the variety of ways that people try to save XML files. Sure, some portion is user error. But does it really matter? We can make this process much much easier for everyone.

In the last month alone, s.b.e/run/ has 150,000 loads! This doesn't count visits to the direct source, the 10K + loads in edX, or the fact that as you say, this code is used in many forks!

However, with such good practices also comes the need to keep up. Someone needs to monitor the issues of the stack of re-used software components, there has to be a testing suite and and update stream, a QA team, compliance efforts, documentation maintenance etc.

(To all the stuff about production software.) Yeah, I know...and for a few years I've actually tried to tell @ddgarcia and Brian that all of BJC could use more support. (Hey, guys, I don't have a real job yet.) As CSP scales up the efforts and use of Snap! (and BJC!) only continue to grow! I'm not saying we go whole hog crazy on everything, but there are a lot of people relying on us. There are a lot of people who are too scared to rely on us, and they will be a lot more relying on us soon.

Technical Stuff:

We can also just use some other server to make single projects downloadable. How about interfacing to Dropbox or some other web-hosting service?

I thought at some point we discussed the privacy implications of using another service...

I'm afraid that such "bold" and "radical" proposals are discouraging us from actually taking the first of many steps to take us there.

Hey, I agree with Brian...that's not the problem with the website. (That is the problem with implementing colors and other things, though... ;-))

P.S. I do agree that Michael is probably overstating the difficulty of getting help from Miosoft.

My issue is not so much that it isn't possible, but that I don't want to be emailing someone at 1am 4 days a week...I'd feel terrible. Also, with 6 students and a few hours a week we were able to replicate the Snap! Cloud pretty easily in 3-4 months of work...really it wasn't that much and it's 90% of the way there. ( https://ucbsnap.herokuapp.com/)

Obviously this library - which turns out to really be two libraries - doesn't solve any problem today, because it needs to be integrated first. And we also need to test it. Reading the inline comments it seems like there might be other issues - especially in Chrome - waiting for us. Also, some use of interval suggest there might be issues with Snap's threading model.

I'd only use 1 library. The blob.js fallback is only needed for older browsers, where blob API isn't available. In that case we can just use our current method. Most Snap! users are currently running fairly new browsers, so we make huge wins without worrying too much. For other users we incorporate #537 to prevent losing work.

Apparently canvas.toBlob() already is in Chrome Canary... (?)

This also only helps for images, not for serializing the XML. Though I am aware and would check for its existence. That would vastly improve script pics, but not much else about files.

Okay, Michael, prove him wrong! :-)

Yes, I will! I had a 10 line beta test fix! I lost the file somewhere but I can recreate I think. Once I post it here, you can paste in into snap.html and test later.

I didn't want to do too much work, because a full solution does require a bit more care.

However, in terms of "today"... #537 solves losing work.

[jmoenig]

(Most of my losing work comes from that damned escape key accidentally closing a block editor! Another easy-to-fix frustration.)

I've fixed this more than two months ago https://github.com/jmoenig/Snap--Build-Your-Own-Blocks/commit/4e0fe2458cf515f3e7263c97eff9b33d605b627e

Okay, so let's use FileSaver.js just for "Save to disk" at first, and also un-hide it again in the project menu. In exchange we can hide "Export project" (I don't want to completely take it out because I use it a lot myself to inspect project xml data for debugging Snap).

If it works with "Save to disk" we can then also use it for "Export summary..." and rename that to "Save summary...", because that also suffers from the Chrome long-uri bug. But, see, for "Export summary" it's actually very nice not just to save it to disc but to show it in a browser window (as is the case for exporting costumes and screenshots). So, I'm just saying, it's solving one problem but not others....

[cycomachead]

I've fixed this more than two months ago 4e0fe24

Yay! :+1: I forgot about that I think. I just remember being burned pretty badly a couple times.

Okay, so let's use FileSaver.js just for "Save to disk" at first, and also un-hide it again in the project menu. In exchange we can hide "Export project" (I don't want to completely take it out because I use it a lot myself to inspect project xml data for debugging Snap).

If it works with "Save to disk" we can then also use it for "Export summary..." and rename that to "Save summary...", because that also suffers from the Chrome long-uri bug. But, see, for "Export summary" it's actually very nice not just to save it to disc but to show it in a browser window (as is the case for exporting costumes and screenshots). So, I'm just saying, it's solving one problem but not others....

Ok, but can we please then have some wrapper check around window.open() to prevent losing work in Chrome, regardless of method? If Chrome is going to crash, then an error is still a prefferable user experience.

In anycase, I agree about the Export Summary being most useful as a display thing, not sure about the image though since the user knows the output. (though really if we're talking about a website, I feel like the summary type of stuff would be an external tool. There we could easily have a PDF-ify tool.) However, in a few months we may still need to switch to downloading the file, at least in Chrome. Chrome 47 currently does not allow "Save As..." from a data URI, which is completely baffling. :(

[jmoenig]

don't care how the sausage is made

I'm in a sausage making country, don't get me started... :-)

[brianharvey]

I think that (at least at first) we shouldn't remove the Export options, because (1) maybe the new way will have a bug, and (2) students are already used to using Export. For the summary, Save to Disk isn't even the right thing, because it'll save HTML and we want PDF. If there's a Print to Disk option, that'd be okay.

Thank you both for moving forward on this!

[brianharvey]

Jens will test Michael's pull request and then pull it.