search

Hydrospheredata / mist

Serverless proxy for Spark cluster
http://hydrosphere.io/mist/
Apache License 2.0
326 stars 68 forks source link

Python - mistpy: BadParameterException #535

Open pybala opened 5 years ago

pybala commented 5 years ago

I'm getting error while trying to get the Spark Context.

from mistpy.decorators import *

@on_spark_context
def testContext(sc):
    print(sc)

testContext()

Am I missing something? why the wrapper is not passing the argument ?

Thanks, Bala

dos65 commented 5 years ago

These decorators just provide information that mist uses to know what type of context function takes. So, you want to call the function directly from python, you need to create and pass sparkContext into it.

pybala commented 5 years ago

Oh ok thanks @dos65 , then how do we share the same context across the applications.

My requirment is to share Dataframe object across applications for interactive requests. Is it possible to do this using Mist?

Thanks, Bala

dos65 commented 5 years ago

Unfortunately, there aren't any built-in feature to share anything except context. Technically, it is possible to cache data on the worker process, but you need to do it on your own.

A context can be shared between several function invocation - there is shared worker-mode option for it (details: 1, 2) Actually, it means that Mist tries to reuse worker processes to not to start a new one per every function invocation.