Constants and Wallaroo - How should constants be handled?

[nisanharamati]

Issue:

• [x] Decision: do not create a special mechanism for this in Wallaroo, but instead use native language functionality (like global variables in Python and Go).
• [ ] Establish a pattern for how to use globally scoped constants in Go and Python applications
• [ ] Update documentation to describe that pattern and provide some examples (maybe also a blog post?)

The discussion section below is kept for context This issue is intended to discuss how state that is static should be handled in Wallaroo.

A canonical example is a constant that you want to be able to access in a stateless computation. e.g. a lookup table. On the one hand, it is definitely a "State", because it is data that is part of the application specification that the event stream interacts with. But it is also not like any current Wallaroo state, because it is

• read-only
• you might need to access it in parallel (to the same copy, or to a different copy of the same static state)
• you need a way to copy it to new workers (whether by loading it at worker startup, or by propagating it from existing workers).

You can currently do this in Python by using a globally scoped object:

from job import read_db
# globally visible db
DB = {}

def application_setup(args):
    # macher state setup
    global DB
    DB = read_db('my_db_file')
    ...

@wallaroo.computation(name="computation")
def computation(data):
    if data.member in DB:
        return data
    return None

but this can lead to users starting to treat this globally scoped object as mutable, which will lead to consistency issues (this object isn't managed by Wallaroo, isn't necessarily consistent across the application workers, etc.) and data races.

So this raises the question: How do we support a pattern of allowing user defined computations to interact with state that is static, that is perhaps similar in behaviour to a global constant in a regular language environment?

Are there specific properties it should definitely have, and not have?

• should it be concurrently accessible?
• should its propagation to new workers be Wallaroo, or the user's responsibility?
• should it support partitions and routing?
• how does the user access it?
  • via method parameters, like my_computation(data, state, constants)
  • via implicit parameters (e.g. place a CONST namespace in the computation's scope so they can use it to access static variables that are in CONST)
  • via the regular variable lookup mechanism of the language in question
  • ???

[aturley]

For right now my preference would be to provide the user with a good model of when it would be safe to initialize a value like this and then let the user handle things from there using whatever mechanism they think is best. This should work fine for Python and Go. It starts to fall apart when you get to a language like Pony that doesn't have a concept of a global variable, but we don't officially support the Pony API for now so I think we can wait on this.

[jtfmumm]

I agree with Andy here since as far as I understand the Python API, even if we exposed a mechanism for using global constants, the user can still use global variables however they like. For this reason, it seems more important that we explicitly describe when this could work and when it could cause problems, even if we discourage it in general.

[nisanharamati]

Discussion result:

• For now, we are choosing to recommend a pattern of using global scoped objects in Python and Go, and update the documentation accordingly.
• When this is no longer an adequate solution (e.g. if we start supporting the Pony API publicly, or a language with similar constraints), we can revisit this issue.