Error with Python3 runtime

[chachra]

Prerequisites

• [x] I am running the latest version. (up upgrade)
• [x] I searched to see if the issue already exists.
• [x] I inspected the verbose debug output with the -v, --verbose flag.
• [ ] Are you an Up Pro subscriber?

Description

Getting Unable to import module '_proxy': No module named '_proxy' in logs when deployed to staging/AWS.

Steps to Reproduce

Using Python3.6 runtime and proxy override of "python3 app.py". Runs perfectly locally, but on deploy does not work and checking up logs gives Unable to import module '_proxy': No module named '_proxy'

Love Up?

Please consider signing up for Up Pro (https://up.docs.apex.sh/#guides.subscribing_to_up_pro) or donating via https://opencollective.com/apex-up/donate.

Slack

Join us on Slack https://chat.apex.sh/

[tj]

Currently it's only possible to use the Node runtimes, just due to how an internal shim works, but I was planning on adding a Python one (issue #671).

It passes events to Up's Go binary, basically just reading some JSON over stdio like https://github.com/apex/up/blob/master/internal/shim/index.js, but I haven't tried writing one in Python yet

[chachra]

Thanks @tj - Is there a workaround if someone wants to use python3 with up then?

[tj]

Currently stuck with python3.4 I believe, if you define proxy.command in up.json to python3 app.py I think that should do the trick. Thought about making that the default but I'm not sure if that's confusing in the python ecosystem or not

There just happens to be python3.4 in the Node.js Lambda container haha..

[chachra]

Hmm.. I did do that only but still got an error. Does it work that way for others?

On Thu, Jun 7, 2018 at 6:12 PM TJ Holowaychuk notifications@github.com wrote:

Currently stuck with python3.4 I believe, if you define proxy.command in up.json to python3 app.py I think that should do the trick. Thought about making that the default but I'm not sure if that's confusing in the python ecosystem or not

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/apex/up/issues/676#issuecomment-395614459, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEdiA-SMMBO34JhhV_zcEHw6oLGTz5Hks5t6c-MgaJpZM4UfVQS .

[chachra]

gotcha... changing proxy but not changing runtime works. I will stick to 2.7 runtime for now I guess then. Tx.

[tj]

Oh I missed that, try removing the runtime in up.json. I'll clarify in the docs but it has to be one of the node* runtimes currently (hard to explain why without getting into implementation details)

[tj]

Yeah it's a little confusing, I'll try to get the python3.6 runtime working tomorrow

[chachra]

That would be awesome. Thanks!

On Thu, Jun 7, 2018 at 6:31 PM TJ Holowaychuk notifications@github.com wrote:

Yeah it's a little confusing, I'll try to get the python3.6 runtime working tomorrow

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/apex/up/issues/676#issuecomment-395617352, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEdiJETm_T6_n6q3f8LBsrK03tYXHMQks5t6dP-gaJpZM4UfVQS .

[chachra]

hi, any update on this ?

[tj]

For now it's probably best to use proxy.command of python3 app.py which is python3.4 I believe. I couldn't get a python shim to work but I know nothing about Python — basically all it needs to do is write/read JSON events which are passed to and from Up.

Something like this but it seems to block on the readline() indefinitely

import subprocess
import json

proc = subprocess.Popen(['./main'],
  stdin=subprocess.PIPE,
  stdout=subprocess.PIPE,
  universal_newlines=True,
  bufsize=1)

def id_generator():
    n = 0
    while True:
        yield n
        n += 1

ids = id_generator()

def handle(event, context):
  # write event
  # TODO: context is not JSON serializable
  proc.stdin.write(json.dumps({
      'id': ids.send(None),
      'event': event,
      'context': {}
  }))

  # read event
  line = proc.stdout.readline()
  event = json.loads(line)

  # respond
  # TODO: raise exception on error
  return event['value']

[chachra]

Mind putting this on a branch or telling me where this shim needs to go and I can try and help! Thanks.

[tj]

Should have time to get back into this tomorrow, I could do with learning some Python anyway haha it's about time :D

[chachra]

Cool. Thanks!

On Wed, Jun 27, 2018 at 8:42 PM TJ Holowaychuk notifications@github.com wrote:

Should have time to get back into this tomorrow, I could do with learning some Python anyway haha it's about time :D

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/apex/up/issues/676#issuecomment-400901256, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEdiAVx45mUUFgC8jo_000GHuI3trt8ks5uBFCIgaJpZM4UfVQS .

[tj]

Hmm yeah it almost works but my python app itself seems to never receive the request, I'll have to dig around some more, but at least the existing Node runtime has Python 3.4.3

[aevumnathan]

I'm strongly considering proposing Up for a new work project, but the app I'd like to deploy is written assuming Python 3.6, so here's another vote for this issue!

FWIW, if I do end up using it, I'd definitely become a Pro subscriber :)

[chachra]

Any updates?

[tj]

@chachra I don't think I have the python skills to do it properly right now, it gets a bit more complex too because the shim has to work for all versions of python. Scripting languages are a bit of a pain for Lambda as-is at least when it comes to "wrapping" like Up does

[komuw]

@tj hi

Something like this but it seems to block on the readline() indefinitely

It is because there's a deadlock(I think). ./main is blocking waiting for input This should work;

def handle(event, context):
    # add newline and flush so that binary program(./main) doesnt deadlock waiting for input
    data = json.dumps(
        {
            "id": ids.send(None), 
            "event": event, 
            "context": {}
        }
        ) + "\n"
    proc.stdin.write(data)
    proc.stdin.flush()

    # read event
    line = proc.stdout.readline()
    proc.stdin.close()
    proc.wait(timeout=2.5)

    event = json.loads(line)
    return event

1. https://eli.thegreenplace.net/2017/interacting-with-a-long-running-child-process-in-python/

[komuw]

https://www.komu.engineer/blogs/lambda-shim/lambda-shim

[tj]

@komuw thanks man! I'll give that a try soon

[komuw]

you are welcome, let me know if I can help(I write python for my day-job and Go for my personal projects). Although I'm not a big fan of a shim per-language (aka https://github.com/apex/up/issues/671) I do not think that would scale very well.