Safely allow server-side creation of new files in order to overcome origin issues

[humphd]

Imagine I want to put http://some-other-origin.com/image.png into my filesystem. I have two options:

• download it in the browser (XHR), get the raw bytes, fs.writeFile(), and sync to the server
• download it on the server (request), get the raw bytes, fs.writeFile(), send an OUT_OF_DATE message to the client(s), and do downstream syncs to give them the file.

The problem with the first method is that we can't get the raw bytes for a file we XHR unless it's same-origin. In order to make it possible, we'd have to proxy the file from our MakeDrive origin (e.g., XHR https://makedrive.webmaker.org/download?url=http://some-other-origin.com/image.png). This is possible, but also kind of sucks.

Another option is to do what I've outlined above in option 2, specifically, we somehow tell the MakeDrive server to download this file and get its raw bytes (no origin restrictions on server-side). The tricky part is simulating having the server be another client for this user, and safely writing this new file to the user's filesystem without causing a conflict.

This will help with #402, and the case that we want to "download" an arbitrary URL into our filesystem.

[humphd]

One way to do this would be to have the client specify a filename to use:

• client1 and client2 are both connected for user A
• client1 asks the server to download http://some-other-origin.com/image.png and save it to /images/image.png
• server gets this request from client1, downloads the URL, and tries obtain a lock to the user A filesystem.
• when the lock is acquired, it tries to write the data to /images/image.png, and if the file already exists it either a) fails and quits; b) tries /images/image-2.png, etc until it works.
• server finishes, releases the lock, sends an OUT_OF_DATE message.
• client1 and client2 do downstream syncs, and get images/image.png from the server.

[humphd]

Here's how I think this should be done:

• Add a new message type to lib/syncmessage.js so a client can request that the server download a given url and write the data to a given path (i.e., the client should provide a url and path when sending the command). For example, I might want to download http://some.site.com/image.png to /images/new-image.png.
• The server should receive this message and try to acquire a lock for the user's filesystem. NOTE: I don't want to use our normal client code to do this, since it requires a downstream sync, and we don't want that on the server. However, we do need a lock so that we can write a path to the user's filesystem.
• The server may or may not be able to acquire the lock, and may have to wait, try again, etc. Eventually it gets the lock.
• The server starts by making sure the path specified is not already taken (e.g., stat it). If it is, the server stops the process, releases the lock, and sends an error to the client indicating that the path is invalid.
• If the path is not taken, the server tries to download the file from the given url. It there is an error (404, 500, etc), the server releases the lock and sends an error to the client indicating that the url is inaccessible.
• If the url request is successful, the server writes the file to the given path, releases the lock, and broadcasts an out-of-date to all clients, indicating that they need to downstream the new file.

[humphd]

NOTE: I don't think this is a requirement for Everest, since Mobile Webmaker doesn't need it (yet). However, the CreativeCommons people will want this.

[humphd]

I think I've talked myself out of doing this. It's going to be a really weird user experience if you request a URL be downloaded into your drive in the browser, send that request to the server, wait for the server to get a lock, download that file, then for a downstream sync. It's also going to be complicated to deal with exclusive path access for the downloaded file, which becomes easy if we just grab it directly via the client.

I think the answer is to have MakeDrive provide a proxy so you can download arbitrary data into your drive locally. Maybe we can build something on top of https://github.com/nodejitsu/node-http-proxy.

[humphd]

I really want to find time to discuss this approach with @Pomax, since Thimble does something similar for http->https mixed-content proxying for images and the like, see https://github.com/mozilla/thimble.webmaker.org/blob/20f91c43ba80071e461e20ab05a6e1c4ed742fd4/lib/proxy.js. I wonder if we can use something similar here.

[gideonthomas]

@humphd would we want to implement this only to avoid cross origin issues?

[humphd]

Correct. I think it's worth exploring, but it's not a P1 I don't think. Users are going to want to stuff things into their filesystem that they find on the web. We need to support that somehow, given origin policy.