emews / EQ-SQL

1 stars 0 forks source link

Update Proxystore code to use keys #35

Open ncollier opened 1 year ago

ncollier commented 1 year ago

Rather than storing a representation of the proxy object in the DB use its key.

Create a simple proxy store example that exercises this and put in emews_examples repo.

ncollier commented 1 year ago

Relevant points:

  1. keys are used as keys when using the object store as a key / value store. The key doesn't give you the proxy but rather the object itself.
  2. keys themselves are actually objects -- NamedTuples. See for example, https://docs.proxystore.dev/main/api/connectors/file/#proxystore.connectors.file.FileKey

So, if we use keys, we are by-passing the proxying -- which may be fine given that we most likely resolve an object as soon as we use it. W/r to 2., given that a key is an object we would need to pickle it and go through the encoding / decoding that we do now when pickling the proxies themselves.

ncollier commented 1 year ago

The proxystore API has some changes since the proxies.py code was written. Stores are now created with

# Store(name, Connector)
store = Store('my-store', FileConnector('/tmp/proxystore'))

When we sort out what to do in light of https://github.com/emews/EQ-SQL/issues/35#issuecomment-1601550630 we also need to update to the new API.

ncollier commented 1 year ago

Options:

For context, proxy payload looks like:

func = proxies.dump_proxies(f=test_proxy_wf.task_func)['f']
proxy_map = proxies.dump_proxies(c=1.0)
params = pop_to_dict(pop, ('x', 'y'))
# use the required payload dict names
payload = json.dumps({'func': func, 'proxies': proxy_map, 'parameters': params})

And swift-t loads the proxies with:

f_str = r'%s'
// f_str in the proxied function
f = proxies.load_proxies({'f': json.loads(f_str)})['f']
proxy_str = r'%s'
params_str = r'%s'
r = f(proxy_str, params_str)

The function uses our decorator:

# this function is proxied in the ME, and proxies.app decorator
# unpacks the arguments.
@proxies.app
def task_func(c, x, y):
    result = c * sin(4 * x) + sin(4 * y) + -2 * x + x**2 - 2 * y + y**2
    return result