Implement urlArgs feature for cache busting AMD modules

ryan-blunden commented 11 years ago

Server is telling client to cache files forever (good production practice)
Client AMD module changes
Client needs to alter the request path in order to cache bust
There seems no way to do this currently via Curl

RequireJS provides this - http://requirejs.org/docs/api.html#config-urlArgs

unscriptable commented 11 years ago

Hey @ryan-blunden,

We've typically solved this problem by putting the version information in the baseUrl. This allows you to have multiple versions deployed without server-side trickery. I have some vague memory of having trouble with url params on GETs with some proxies once, too (????).

examples:

baseUrl: "company/version-1.3/app" // semantic versioning

baseUrl: "../4f6da329/app" // commit hash (partial)

Would this work for your project?

-- John

ryan-blunden commented 11 years ago

Yes it would work and is exactly what I've implemented in the meantime.

I would ask you to consider this though for people in environments that do not have any control over the build system or server setup. This feature would allow them to cache bust without any configuration of an external system.

sveisvei commented 11 years ago

+1, Static fileservers wont allow me to change paths.

dianasotirova commented 11 years ago

+1

unscriptable commented 11 years ago

Cool. Love the feedback! Quick question for each of you:

Are you experiencing this problem during development, [deployment to] production, or both?

ryan-blunden commented 11 years ago

Development is not a problem as I disable caching via settings in Chrome development tools. This is only for running in production.

ryan-blunden commented 11 years ago

I should also point out that I absolutely love everything about Curl.js and this is the only issue I've had with it so far.

unscriptable commented 11 years ago

Hey thanks @ryan-blunden :)

The reason I asked about development is that I'm thinking the best use case for this type of transform (we consider this feature to be a url transform) is to invalidate the cache each time a new build of the entire application is created. (If you've got another use case, please let me know!)

Considering this, my proposal is to add two new features to curl's config: "buildId" and "buildTransform".

buildId: {String} (or anything that can be coerced to a string) buildTransform: {String|RegExp} a regular expression used to replace part of the url with "buildId"

If buildTransform is a string, it will be converted to a RegExp. You don't have to specify buildTransform if you want it to append onto the url. By default it would do something similar to RequireJS's urlArgs, which appends a query param.

So something like this:

if (buildId && buildTransform) url = url.replace(buildTransform, buildId);

Thoughts?

-- John

asilvas commented 11 years ago

While I get the functionality, what is the idea around "build"? Does it different from a "pathTransform"? Perhaps this can be achieved in a more generic fashion to avoid additional identifiers or transforms in the future? This might also be less accessible (developer friendly) than RequireJS's solution (but also less powerful). I don't have any suggestions at this point, but wanted to note my concerns to avoid unnecessary feature bloat.

sveisvei commented 11 years ago

For me its mainly for production - appending application version, but with requirejs i have also used it as a easy cache-buster (appending Date).

asilvas commented 11 years ago

@sveisvei Unrelated to the topic, but you might want to consider some form of "version" tag in place of Date, as using the Date can result in invalid cache same-day as deployments. Most people get away with this, but you can just as easily avoid the scenario with a configuration-based "version" to guarantee that when you deploy a new version, all clients will immediately get the new data regardless if how long the expiration is. Additionally, this technique will allow clients to cache your resources for greater than 1 day.

unscriptable commented 11 years ago

Ah, yes, @asilvas! Thanks for reminding me. IIRC, your team has a requirement that lends itself to a generic url transform function (yes, path transform == url transform, effectively*).

Does this proposed feature satisfy your requirements? Is it just the term "build" that is too specific?

To be ultra-generic, we'd have to allow a transform function to be specified in the config. I believe you suggested this on an issue a few months back.

urlTransform: function (url, id) { return addVersion(url); }

This is a great idea, but we also need a JSON (or JSON-ish) config option for build tools that use JSON. This is all I've got:

urlTransform: { replaceRx: "myapp/", replaceWith: "myapp/v1.2.3/" }

-- John

Within the cujojs team, we started calling path transforms "url transforms" to better distinguish from id transforms like the expansion of package main modules and shortcut plugin ids into their full ids (text --> curl/plugin/text). These happen at different stages of the "resolution pipeline".

asilvas commented 11 years ago

Your syntax looks good. Optionally support urlTransform as an array? The minor change could be a nice boost in scenerios -- rather than using some obscenely complex expressions, use a series of unrelated expressions.

Additionally, could even provide an "urlArgs" option that would auto-append a transform to the urlTransform array since that'd likely be one of the more common use cases. Or just provide a reference example of urlTransform in the documentation if it's not worth its weight.

Food for thought.

FYI, we're currently using a shim to do all of our pathing magic. This is working quite well, so I currently have no immediate need for transforms, but that doesn't make it any less of a welcome feature (and our dev teams could take advantage of it if needed without worry of it impacting our shim).

tmaslen commented 11 years ago

The ability to add cachebusting onto the end of the file path would be useful for me too.

I've moved teams and am having to load modules in via both require AND curl :-)

This team has a more adhoc release process (involving FTP), and can directly edit each page as opposed to editing templates. Sometimes we make tiny edits to JS files and then need to break the cache after FTPing them. These aren't strictly new releases of products (unlike my old team) but just tiny, random changes requested by editorial.

A BuildId value is a nice concept, especially if it was something we stated once in the config and then didn't have to add onto every file. This would work for us as long as it didn't affect the directory structure.

Thanks, /t

ryan-blunden commented 11 years ago

This would make a most excellent holidays gift :)