Outfox should cache audio files

[GoogleCodeExporter]

Go to the test page, watch the server log. Every press of play sound will
fetch the sound from the server.

Original issue reported on code.google.com by piisnear...@gmail.com on 28 Jul 2008 at 1:46

[GoogleCodeExporter]

Useful looking links

http://www.xulplanet.com/references/xpcomref/group_Cache.html
http://www.xulplanet.com/references/xpcomref/ifaces/nsICacheSession.html#method_
openCacheEntry
http://developer.mozilla.org/en/docs/nsICacheSession#Parameters

Original comment by piisnear...@gmail.com on 30 Jul 2008 at 1:04

[GoogleCodeExporter]

Original comment by c...@unc.edu on 30 Jul 2008 at 10:37

• Added labels: Milestone-0.1.1

[GoogleCodeExporter]

First pass:

 * Client sends a command with a URL field.
 * Extension watches for URLs on all commands.
 * Extension uses nsICacheSession from nsICacheService.
 * Extension looks for URL in cache.
 * If it's cached:
  * Open the nsICacheEntryDescriptor synchronously (async?) and get its nsIFile from
the file attribute.
  * Set the nativeTarget on the original command as the file attribute and remove the
URL attribute.
 * If it's not cached:
  * Fetch the item via XHR.
  * Open the nsICacheEntryDescriptor synchronously (async?) and get its nsIFile from
the file attribute.
  * Set the nativeTarget on the original command as the file attribute and remove the
URL attribute.

Problems:

 * All of this has to happen synchronously, otherwise commands can get out of order.
 * How long does the extension wait before timing out?
 * Will this model work for all other servers that might want to cache data? Can we
watch for URLs only, or does it need to be more explicit? Are we adding special 
code
to the ext for sound only?

Original comment by c...@unc.edu on 30 Jul 2008 at 11:38

[GoogleCodeExporter]

    getFromCache: function(cmd) {
    var cs = Components.classes["@mozilla.org/network/cache-service;1"].
        getService(Components.interfaces.nsICacheService);
    var nsic = Components.interfaces.nsICache;
    var sess = cs.createSession('HTTP', nsic.STORE_ANYWHERE,
        nsic.STREAM_BASED);
    try {
        var entry = sess.openCacheEntry(cmd.url, nsic.ACCESS_READ, 
        nsic.BLOCKING);
    } catch(e) {
        entry.close()
        return null;
    }
    logit(entry.file.target);
    entry.close();
    }

This method successfully fetches the full path of an mp3 file from the cache on 
osx.
The mp3 file was accessed directly from the url bar for testing purposes before
running the cache test.

Some concerns:

* I had to specify STORE_ANYWHERE instead of STORE_ON_DISK_AS_FILE for this to 
work.
But what if the cached data ends up being in memory on some platform or config 
instead?
* I had to use STREAM_BASED instead of NOT_STREAM_BASED for the mp3 file. Will 
this
be true of all sounds? Of all urls going to extensions?

Next task is to figure out how to best get items into the cache. Doing XHR is 
going
to give the data back to the caller, which isn't necessary. Just want it in the
cache. The prefetch service might be the answer
(http://www.xulplanet.com/references/xpcomref/comps/c_prefetchservice1.html).
Problems to solve:

* How can we queue all other commands until the fetch is done to preserve 
ordering?
* What happens when an action like "stop" or "reset" is received? These actions 
are
service specific, and ideally the extension knows nothing about them. But on a 
stop
or reset for sound, ideally the extension would abort any prefetch of a url
immediately to avoid delay. What's the proper balance?

We might also want a way for the JS to pre-cache data in urls in batches, not 
only
incrementally when individual sounds are played (for example). If we went with a
batch only precaching approach, then all of the ordering / stopping / timing 
issues
practically disappear. The trade-off is the app writer has to request 
pre-caching
ahead of time, else sounds are slowly fetched off the network and never cached. 
A
hybrid approach is best, but might be very difficult to get right.

Original comment by c...@unc.edu on 2 Aug 2008 at 3:21

[GoogleCodeExporter]

Original comment by c...@unc.edu on 2 Aug 2008 at 3:21

• Changed state: Started

[GoogleCodeExporter]

Yow! That is getting really complicated. Perhaps explicit is better than 
implicit.
Make it easy with a manifest of some sort. Then the implementation in the server
would be pretty simple. 

Original comment by piisnear...@gmail.com on 2 Aug 2008 at 10:49

[GoogleCodeExporter]

If we go with explicit, then we don't want to tie to a manifest specifically. 
The JS
API would be prefetch(array of urls). How to get that array is up to the 
developer.

I don't think it's time to give up on implicit caching yet. I believe the 
toughest
problems of how to abort prefetches and how to keep things in order can be 
solved
with message-based deferreds:

* Extension sees a url in a command.
* Extension starts asynchronous prefetch.
* Extension forwards command to the external server with some tag saying it's 
in the
process of being prefetched.
* External handler queues the command like always..
* External handler eventually processes the command.
* If command has prefetch tag, check a dictionary to see if another immediate
(non-queued) command has arrived in the meantime saying the url data is 
available
locally.
 * If yes, continue processing the command and use the local data.
 * If no, stall the queue processing.

Meanwhile ...

* Extension observes prefetch has finished.
* Extension sends a private command (e.g., {"action" : "prefetch-done", 
"filename" :
whatever, "url" : original url}) to the server.
* The private command eventually reaches the handler where it is processed
immediately (not queued).
* If queue processing stalled on a command with a matching url, continue 
processing
using the new filename as the data source.
* If the queue stalled on a command with a different url or is not stalled at 
all,
toss the command in a dictionary with key cmd.url and value cmd.

The extension can inform the server handler of errors in prefetching the same 
manner.

Benefits:

* Order is preserved by the stall.
* Extension doesn't have to be aware of how commands will be processed in an 
extension.
* The handler in the server gets to dictate when to clear the cache, what to do 
for a
command that is prefetching, etc.

Original comment by c...@unc.edu on 2 Aug 2008 at 5:37

[GoogleCodeExporter]

Got prefetch working. An item not previously in the cache is added to the cache 
after
invoking the service with the proper nsIURI.

The next problem is that the prefetch service for JS doesn't appear to have any
interface for observing notifications when a certain prefetch has completed. It
implements nsIWebProgressListener itself, but doesn't appear to give the caller 
any
way of listening too. It does have an enumeration method for visiting all queued
pre-fetch requests, but I'm not sure that helps produce a sane solution.

Google searches don't turn up much. Probably going to have to poke around on 
IRC.

Original comment by c...@unc.edu on 3 Aug 2008 at 6:48

[GoogleCodeExporter]

Rev 75 includes code that checks the cache for all commands including a url 
parameter
for local filename information. If the url is cached, the local filename is 
attached
to the command under the "filename" attribute. If the url is not cached, an XHR 
get
of the data for the cache is initiated. Meanwhile, the original command is 
forwarded
to the outfox server with attribute "deferred" set to a unique integer. The 
server
can decide whether to wait for the deferred result or process the command
immediately, ignoring the cache.

Later, when an XHR GET completes and the item is in the cache, a matching 
command
with "action" value of "deferred-result" is emitted to the outfox server. The
"deferred" field on this new command matches the unique ID on the original. The 
new
command also includes the local cache "filename" info.

Right now, all of the above takes place. However, the server modules for win32, 
osx,
and nix, are not currently making use of the extra information yet. They pay no
attention to the filenames or deferred results, and just fetch the url data and 
play
it as before. Next step is to implement their support for the deferreds.

Original comment by c...@unc.edu on 7 Aug 2008 at 2:28

[GoogleCodeExporter]

Added support for deferreds to osx/channel.py. Tested with new caching example 
on
demo page and some print statements. Works fine. Need to check if error cases 
are
handled properly so the queue never stalls permanently.

Will add support to win32 and nix next. Going to open a new bug about 
refactoring all
of this shared code into a common base class.

Original comment by c...@unc.edu on 10 Aug 2008 at 7:45

[GoogleCodeExporter]

Added support for deferreds to win32/channel.py and nix/channel.py (rev 77). 
Tested
with caching example on demo page and some print statements. Works. Still need 
to
check error cases.

win32 and nix differ from osx implementation in that they must distinguish 
playing of
local files from remote ones opening with urllib2. Remote URL based file 
handles do
not support seek so we have to test for the mp3 file extension to determine how 
to
handle them. Conversely, local cache files do not have proper file extensions 
so we
have to try treating them as mp3s, fail with an exception, seek back to start, 
then
try as all other formats supported by pygame.

Original comment by c...@unc.edu on 10 Aug 2008 at 11:55

[GoogleCodeExporter]

No further problems reported. Marking as fixed for 0.1.1. Future problems 
should be
opened in new bug reports.

Original comment by c...@unc.edu on 31 Aug 2008 at 1:10

• Changed state: Fixed