chapel-lang / chapel

a Productive Parallel Programming Language
https://chapel-lang.org
Other
1.8k stars 422 forks source link

[Feature Request] In-Memory Cache/Database (similar to 'memcached') #11616

Open LouisJenkinsCS opened 6 years ago

LouisJenkinsCS commented 6 years ago

One thing that I desire to do in my HPC applications is to create a caching system that can cache any key to any value. For example, in my application I have to compute data that is computationally expensive, as in taking hours and possibly even days to compute. The ability to map arbitrary keys to arbitrary data would be nice feature, as well as being able to set a time for when the cached data can be reclaimed if unused for such a period of time. I would, as an application writer, like to have a third-party or optimally standard module to allow this.

For a proof-of-concept, it may be possible to implement something where an associative domain that allows arbitrary keys, domain(opaque), can be used to map to buffers of arbitrary bytes?

class Cache {
   var dom : domain(opaque);
   var arr : [dom] ByteBuffer;
}

proc Cache.put(key, value) {
   var buf = new ByteBuffer(value);
   dom += key;
   buf <=> arr[dom];
   delete buf; 
}

proc Cache.get(key, type t) {
    assert(dom.contains(key));
    return arr[key] : t;
}
LouisJenkinsCS commented 6 years ago

One interesting application would be for it to serve as the privatization table. Cache.get(pid, dataType) is very similar to chpl_getPrivatizedCopy. If not for the runtime and internal modules then for user-level privatization.