Environment Variable not seen by brenda, but there when I ssh into the instance

[aliasguru]

This is driving me nuts. I've created a custom AMI on AWS to incorporate my own tools / add-ons used with Blender. While configuring the image on AWS, everything works as expected, both as user ubuntu as well as user root. I need to set an environment variable to locate the tools, and I do it like this:

in /etc/environment: BLENDER_USER_CONFIG=/mypath/dingdong/ BLENDER_USER_SCRIPTS=/mypath/dongding/

I then created the AMI and configured Brenda to use it, which works just fine. With one exception: When Brenda fires off the job, it does not see my custom vars. I ssh into the running instance again, and as before, no matter if I am user ubuntu or root, or even use sudo printenv, all the vars are listed and correct. I can run Blender from the command line in all circumstances, the tools will be loaded. It is only when submitting the job using brenda that the custom dirs are never found.

What can I do? Which key concept do I miss here?

[aliasguru]

The AMI is based on Ubuntu 16.04.2, Blender is version 2.79.

[tuttlen]

I think I may have run into this. I have to look at the script again. But I believe it starts the instance and then runs the script. There may be a sudo command inside the script that changes the system somehow. Can you insert a couple pipe out commands at the beginning and end of the blender script to see the if the variables change?

On Oct 8, 2017 1:46 PM, "aliasguru" notifications@github.com wrote:

The AMI is based on Ubuntu 16.04.2, Blender is version 2.79.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jamesyonan/brenda/issues/32#issuecomment-335028962, or mute the thread https://github.com/notifications/unsubscribe-auth/ABIPf9PPmK2erlSjkTYgV5Vsb0bPDdfYks5sqRhrgaJpZM4PxzkS .

[aliasguru]

I've modified the brenda source code to inject a few echos and system commands. The script sent to the server now looks like this:

----------------------------
AMI ID: ami-xxxxxxxxx
Max bid price 0.06
Request type: one-time
Instance type: c4.xlarge
Instance count: 1
Instance store device: /dev/sdb
SSH key name: xxxxxxxxxxxxxxxx
Security groups: ('brenda-web',)
Startup Script:
   #!/bin/bash
   echo user
   echo ----
   whoami
   echo current location
   echo ----------------
   pwd
   echo environment variables:
   echo ----------------------
   printenv
   echo ----------------------
   # run Brenda on the EC2 instance store volume
   B="/mnt/brenda"
   if ! [ -d "$B" ]; then
     for f in brenda.pid log task_count task_last DONE ; do
       ln -s "$B/$f" "/home/ubuntu/$f"
     done
   fi
   export BRENDA_WORK_DIR="."
   # export BLENDER_USER_CONFIG=/CustomSoftware/xxx/
   # export BLENDER_USER_SCRIPTS=/CustomSoftware/yyy/
   # export OCIO=/CustomSoftware/filmic-blender/config.ocio
   mkdir -p "$B"
   cd "$B"
   echo user
   echo ----
   whoami
   echo current location
   echo ----------------
   pwd
   echo environment variables:
   echo ----------------------
   printenv
   echo ----------------------
   /usr/local/bin/brenda-node --daemon <<EOF
   AWS_ACCESS_KEY=[redacted]
   AWS_SECRET_KEY=[redacted]
   BLENDER_PROJECT=s3://bucketName/renderme.zip
   WORK_QUEUE=sqs://Kiska-Test
   RENDER_OUTPUT=s3://outBucketName
   DONE=shutdown
   echo user
   echo ----
   whoami
   echo current location
   echo ----------------
   pwd
   echo environment variables:
   echo ----------------------
   printenv
   echo ----------------------
   EOF

As you can see, I've commented out my own export commands, which allowed me to finally set those vars as I really need them. When I now retrieve the system log from that instance using the AWS console, I see that the commands are executed:

[   20.783797] cloud-init[1331]: user
[   20.786265] cloud-init[1331]: ----
[   20.796147] cloud-init[1331]: root
[   20.798499] cloud-init[1331]: current location
[   20.801062] cloud-init[1331]: ----------------
[   20.803646] cloud-init[1331]: /
[   20.805912] cloud-init[1331]: environment variables:
[   20.808564] cloud-init[1331]: ----------------------
[   20.811132] cloud-init[1331]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[   20.814566] cloud-init[1331]: PWD=/
[   20.816847] cloud-init[1331]: LANG=en_US.UTF-8
[   20.819416] cloud-init[1331]: SHLVL=1
[   20.821770] cloud-init[1331]: _=/usr/bin/printenv
[   20.824306] cloud-init[1331]: ----------------------
[   20.827020] cloud-init[1331]: user
[   20.829361] cloud-init[1331]: ----
[   20.831664] cloud-init[1331]: root
[   20.833954] cloud-init[1331]: current location
[   20.836465] cloud-init[1331]: ----------------
[   20.838914] cloud-init[1331]: /
[   20.841150] cloud-init[1331]: environment variables:
[   20.843732] cloud-init[1331]: ----------------------
[   20.846318] cloud-init[1331]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[   20.849641] cloud-init[1331]: PWD=/
[   20.851882] cloud-init[1331]: LANG=en_US.UTF-8
[   20.854362] cloud-init[1331]: SHLVL=1
[   20.856656] cloud-init[1331]: _=/usr/bin/printenv
[   20.859302] cloud-init[1331]: ----------------------
[   20.861906] cloud-init[1331]: ln: failed to create symbolic link '/home/ubuntu/log': File exists
[   20.865175] cloud-init[1331]: user
[   20.867372] cloud-init[1331]: ----
[   20.869637] cloud-init[1331]: root
[   20.871885] cloud-init[1331]: current location
[   20.874330] cloud-init[1331]: ----------------
[   20.876976] cloud-init[1331]: /mnt/brenda
[   20.879422] cloud-init[1331]: environment variables:
[   20.882121] cloud-init[1331]: ----------------------
[   20.884818] cloud-init[1331]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[   20.888208] cloud-init[1331]: PWD=/mnt/brenda
[   20.890816] cloud-init[1331]: LANG=en_US.UTF-8
[   20.893339] cloud-init[1331]: SHLVL=1
[   20.895659] cloud-init[1331]: BRENDA_WORK_DIR=.
[   20.898248] cloud-init[1331]: _=/usr/bin/printenv
[   20.900814] cloud-init[1331]: OLDPWD=/
[   20.903237] cloud-init[1331]: ----------------------

So, at all times, the user which runs the brenda commands is root, and the current location is / (which is odd, I'd have expected it to be /root). There is a symlink error, but that one is caused by myself, I had already created that one on the AMI. The error is harmless. The environment vars however are clearly the minimalistic ones, not the ones ssh gives me.

One hope I can see: While digging around, I noticed that the startup script actually looks for a key called STARTUP_PRERUN. I interpret this as someone (presumably James Yonan) giving us a possibility to inject own commands to the script, by placing them into .brenda.conf. That's what I'll try next. If it works, users are not obliged to change source code for defining own AMI vars.

[aliasguru]

Tested it right now, works perfectly!

Open your .brenda.conf, and add a section like this at the end:

# CUSTOM VAR section
# add commands to be executed before rendering here
STARTUP_PRERUN=export BLENDER_USER_CONFIG=/path/you/want/;export BLENDER_USER_SCRIPTS=/path/you/need/;export OCIO=/guess/what/config.ocio

Separate all vars and commands with a semicolon as indicated above. I guess things like this would work as well (but have not tested it):

# CUSTOM VAR section
# add commands to be executed before rendering here
STARTUP_PRERUN=sudo ln -s /mnt/brenda/log /home/ubuntu/log

This would for example re-enable the ability to simply execute brenda-tool ssh tail log again (which atm doesn't work)