search

dignoe / whenever-elasticbeanstalk

Allows you to run cron jobs easily on one or all AWS Elastic Beanstalk instances.
MIT License
48 stars 35 forks source link

Command cron_01_set_leader failed #18

Open sumit20rai opened 9 years ago

sumit20rai commented 9 years ago

After installing/configuring this gem, I'm seeing the following error in /var/log/cfn-init.log on my EC2 instance after running git aws.push from my local repo.

2014-10-21 08:08:37,602 [DEBUG] Running test for command cron_01_set_leader 2014-10-21 08:08:37,744 [DEBUG] Test command output: 2014-10-21 08:08:37,745 [DEBUG] Test for command cron_01_set_leader passed 2014-10-21 08:08:38,085 [ERROR] Command cron_01_set_leader (su -c "/usr/local/bin/bundle exec create_cron_leader --no-update" $EB_CONFIG_APP_USER) failed 2014-10-21 08:08:38,086 [DEBUG] Command cron_01_set_leader output: bash: /usr/local/bin/bundle: No such file or directory

2014-10-21 08:08:38,086 [ERROR] Error encountered during build of postbuild_0_tubett: Command cron_01_set_leader failed Traceback (most recent call last):

I have added the whenever-elasticbeanstalk Below is my cron.config file content.. Any idea ...what am i doing wrong?

files:

Reload the on deployment

/opt/elasticbeanstalk/hooks/appdeploy/post/10_reload_cron.sh: mode: "00700" owner: root group: root content: |

!/usr/bin/env bash

  . /opt/elasticbeanstalk/containerfiles/envvars
  cd $EB_CONFIG_APP_CURRENT
  su -c "/usr/local/bin/bundle exec setup_cron" $EB_CONFIG_APP_USER

Add Bundle to the PATH

"/etc/profile.d/bundle.sh": mode: "000755" owner: root group: root content: |

!/usr/bin/env bash

  export PATH=$PATH:/usr/local/bin
encoding: plain

container_commands: cron_01_set_leader: test: test ! -f /opt/elasticbeanstalk/containerfiles/.cron-setup-complete leader_only: true cwd: /var/app/ondeck command: su -c "/usr/local/bin/bundle exec create_cron_leader --no-update" $EB_CONFIG_APP_USER cron_02_write_cron_setup_complete_file: cwd: /opt/elasticbeanstalk/containerfiles command: touch .cron-setup-complete

Thanks Sumit

jufemaiz commented 9 years ago

Cross post @ http://stackoverflow.com/questions/26483509/command-cron-01-set-leader-output-bash-usr-local-bin-bundle-no-such-file-or

Can you please provide further information on the instance you are attempting to use this with? What is the full path to bundle on your instance? If this has changed, we have an issue obviously.

jufemaiz commented 9 years ago

@sumit20rai ping

tboyko commented 9 years ago

Seeing the same issue with bundle not existing where the script expects it. I'm running the "64bit Amazon Linux 2014.03 v1.0.9 running Ruby 2.1 (Puma)" instance and bundle exists at "/opt/rubies/ruby-2.1.2/bin/bundle". Why not incorporate something along the lines of BUNDLE_EXE=which bundle to the script and running off of the result?

Digging further, this creates other issues. I'm now seeing this:

2014-11-03 21:00:05,949 [INFO](13482 MainThread) [directoryHooksExecutor.py-29] [root directoryHooksExecutor info] Executing script: /opt/elasticbeanstalk/hooks/appdeploy/post/10_reload_cron.sh 2014-11-03 21:00:06,443 [INFO](13482 MainThread) [directoryHooksExecutor.py-29] [root directoryHooksExecutor info] Output from script: bundler: command not found: setup_cron Install missing gem executables with bundle install

2014-11-03 21:00:06,443 [ERROR](13482 MainThread) [directoryHooksExecutor.py-33] [root directoryHooksExecutor error] Script /opt/elasticbeanstalk/hooks/appdeploy/post/10_reload_cron.sh failed with returncode 127

jufemaiz commented 9 years ago

@tboyko that looks like a better approach. One of the problems I've found in investigating this on the older instance is the inability to find the gem even if the bundle is correct. Will post some extra info shortly.

tboyko commented 9 years ago

Looks like the path issue is throughout the gem. This is tough for me to test (with a forked repo) because my experience with EB is that it just doesn't play nicely with git repos or local paths in the Gemfile...

jufemaiz commented 9 years ago

@tboyko ah yeah… that's a "fun" part of EB eh. I've found ways and means to cache into /vendor (simlinked into /var/app/containerfiles for persistency across deployments )

jufemaiz commented 9 years ago

@sumit20rai and @tboyko can you please try release v1.1.5 on your codebase. Note: you'll need to remove your .ebextensions/cron.config before running wheneverize-eb again.

tboyko commented 9 years ago

There seems to be an issue. Here's what I got after upgrading, deleting and recreating cron.config, and deploying:

2014-11-07 17:50:08,242 [DEBUG] Running command a_elasticbeanstalk_support_path 2014-11-07 17:50:08,242 [DEBUG] No test for command a_elasticbeanstalk_support_path 2014-11-07 17:50:08,248 [ERROR] Command a_elasticbeanstalk_support_path (. /tmp/update_appenv_include_eb_config_support.sh) failed 2014-11-07 17:50:08,248 [DEBUG] Command a_elasticbeanstalk_support_path output: containerfiles export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/support /tmp/update_appenv_include_eb_config_support.sh: line 8: /opt/elasticbeanstalk/support/envvars.d/appenv: No such file or directory

2014-11-07 17:50:08,248 [ERROR] Error encountered during build of prebuild2[REMOVED]: Command a_elasticbeanstalk_support_path failed Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 511, in run_config CloudFormationCarpenter(config, self._auth_config).build(worklog) File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 247, in build changes['commands'] = CommandTool().apply(self._config.commands) File "/usr/lib/python2.6/site-packages/cfnbootstrap/command_tool.py", line 113, in apply raise ToolError(u"Command %s failed" % name) ToolError: Command a_elasticbeanstalk_support_path failed 2014-11-07 17:50:08,249 [ERROR] Unhandled exception during build: Command a_elasticbeanstalk_support_path failed Traceback (most recent call last): File "/opt/aws/bin/cfn-init", line 122, in worklog.build(detail.metadata, configSets) File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 117, in build Contractor(metadata).build(configSets, self) File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 502, in build self.run_config(config, worklog) File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 511, in run_config CloudFormationCarpenter(config, self._auth_config).build(worklog) File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 247, in build changes['commands'] = CommandTool().apply(self._config.commands) File "/usr/lib/python2.6/site-packages/cfnbootstrap/command_tool.py", line 113, in apply raise ToolError(u"Command %s failed" % name) ToolError: Command a_elasticbeanstalk_support_path failed

jufemaiz commented 9 years ago

Crap. Sorry. Hadn't committed one of the files :(

Change .ebextensions/cron.config to:

    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      # update_appenv_include_eb_config_support.sh
      if test -d /opt/elasticbeanstalk/containerfiles; then
        echo "containerfiles"
        grep -Fwq "export EB_CONFIG_SUPPORT" /opt/elasticbeanstalk/containerfiles/envvars.d/appenv
        if [ $? -eq 1 ]; then
          echo "export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/containerfiles"
          echo "export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/containerfiles" >> /opt/elasticbeanstalk/support/envvars.d/appenv;
        fi
      elif test -d /opt/elasticbeanstalk/support; then
        echo "support"
        grep -Fwq "export EB_CONFIG_SUPPORT" /opt/elasticbeanstalk/support/envvars.d/appenv
        if [ $? -eq 1 ]; then
          echo "export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/support"
          echo "export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/support" >> /opt/elasticbeanstalk/support/envvars.d/appenv;
        fi
      fi
    encoding: plain
tboyko commented 9 years ago 
2014-11-10 17:33:12,126 [DEBUG] Command a_elasticbeanstalk_support_path output: containerfiles
export EB_CONFIG_SUPPORT=/opt/elasticbeanstalk/containerfiles
/tmp/update_appenv_include_eb_config_support.sh: line 8: /opt/elasticbeanstalk/support/envvars.d/appenv: No such file or directory

2014-11-10 17:33:12,126 [ERROR] Error encountered during build of prebuild_2_[removed]: Command a_elasticbeanstalk_support_path failed
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 511, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 247, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/command_tool.py", line 113, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command a_elasticbeanstalk_support_path failed
2014-11-10 17:33:12,127 [ERROR] Unhandled exception during build: Command a_elasticbeanstalk_support_path failed
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 122, in <module>
    worklog.build(detail.metadata, configSets)
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 117, in build
    Contractor(metadata).build(configSets, self)
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 502, in build
    self.run_config(config, worklog)
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 511, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 247, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.6/site-packages/cfnbootstrap/command_tool.py", line 113, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command a_elasticbeanstalk_support_path failed
tboyko commented 9 years ago

I think there may be bigger issues at hand here. TL;DR - Amazon Linux AMIs are changing their structure and breaking the whole $EB_ variable notion, which this gem depends on. Seems like a major gem restructuring may be involved :/

https://forums.aws.amazon.com/thread.jspa?messageID=583440&#583440

jufemaiz commented 9 years ago

Oh crap. WTF is Amazon doing? Surely this is stuff that should be well documented and notified to users :(

tboyko commented 9 years ago

No kidding. Looks like the OpsWorks platform isn’t immune to this sort of thing either:

http://blogs.aws.amazon.com/application-management/post/Tx2G95J53SKI2E0/AWS-OpsWorks-supports-application-environment-variables http://blogs.aws.amazon.com/application-management/post/Tx2G95J53SKI2E0/AWS-OpsWorks-supports-application-environment-variables

Read the first comment…

On Nov 13, 2014, at 2:01 PM, Joel Courtney notifications@github.com wrote:

Oh crap. WTF is Amazon doing? Surely this is stuff that should be well documented and notified to users :(

— Reply to this email directly or view it on GitHub https://github.com/dignoe/whenever-elasticbeanstalk/issues/18#issuecomment-62974408.

jufemaiz commented 9 years ago

Ugh :(

WHY?!?!? BUT WHY?!?!?

jufemaiz commented 9 years ago

Ok @tboyko I've done some digging.

As root, you have access to the following shell script (which in turn calls a ruby script using a gem not available as ec2-user).

/opt/elasticbeanstalk/bin/get-config

With that you can identify which data you want to access with the following options:

Usage: get-config CATEGORY [OPTIONS]

Categories:
    optionsettings                   environment option settings that affect instance
    container                        container specific configurations
    addons                           addon configurations
    environment                      environment variables
    meta                             EB environment meta-data

Options:
    -k, --key KEY                    Key of specific configurations.
        --output OUTPUT_FORMAT       Output format. Can be JSON or YAML. Default is JSON.
    -n, --namespace NAMESPACE        Otion Setting namespace for retrieval. Only applied to Category optionsettings.
    -o, --option-name OPTION_NAME    Option Setting name for retrieval. Only applied to Category optionsettings.
    -a, --add-on ADDON               Add-on name. Only applied to Category addons.
    -h, --help                       Help

This itself is obtained from files on the instance, I believe anyway, at /opt/elasticbeanstalk/deploy/configuration/containerconfiguration

So now the question is, do older instances have this available to them. If so not so bad. If not… :(

benmkramer commented 9 years ago

Has there been any progress or updates to this on Elastic Beanstalk? Are there any alternatives that would be recommended?

rmahnovetsky commented 9 years ago

anyone made any progress on this one?

sanjayginde commented 9 years ago

@rmahnovetsky I hacked this together to get things functional for my own project. I'm hoping to clean it up at some point: https://github.com/contently/whenever-elasticbeanstalk/pull/1/files

rmahnovetsky commented 9 years ago

@sanjayginde thanks. I'll take a look and let you know how I get on

mcbsys commented 9 years ago

Not specifically about this gem, but here are a couple resources I found recently on the environment variable issue:

In summary you probably want to do something like this starting from the 2014.09 AMIs:

# Loading environment data
EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)
EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
EB_APP_CURRENT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
EB_APP_PIDS_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)

Here are some commands to check the AMI version:

cat /etc/system-release
cat /etc/image-id
cat /etc/yum.conf | grep releasever # Beanstalk locks repositories using this line
langtianlang commented 9 years ago

Is there any update on this? I'm looking to use whenever on ebs. Should I be using this project? Any guidance would be appreciated, thanks.

rmahnovetsky commented 9 years ago

I ended up creating a worker environment and using the periodic scheduler. http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html

JaisonBrooks commented 9 years ago

I would love to know if there is a solution in the works or something out there to show how to use this gem and eb.

mkafkas commented 9 years ago

is there any update on this issue?

binodmainali commented 8 years ago

I am also facing something similar and stuck with Unauthorized access error

binodmainali commented 8 years ago

Above is your schedule file converted to cron syntax; your crontab file was not updated. I am still getting this as the output

CyborgMaster commented 8 years ago

I got this completely working with the latest AMI (2015.03). I based my work of the fork from @sanjayginde, but had to make a few more tweaks. I've got a pull request back to his fork open here: contently/whenever-elasticbeanstalk#2. I recommend that he accept my pull request and then open another one from his fork back here to the main repo, because he did 95% of the work.

samhamilton commented 8 years ago

Has anyone got this working with the latest AMI 2015.09? Seems to be broken again.

jufemaiz commented 8 years ago

https://github.com/dignoe/whenever-elasticbeanstalk/pull/26

For those with issues who still would like this to work, please provide me with some information on whether or not this works for you.

dharmdip commented 7 years ago

Problem is EB is not checking stack ruby version and complains bundle with this error

Activity execution failed, because: bundler: command not found Install missing gem executables withbundle install(ElasticBeanstalk::ExternalInvocationError) caused by: bundler: command not found Install missing gem executables withbundle install(Executor::NonZeroExitStatus)

need to tell ruby version where gems are installed by adding this line or manage by environment variables.

. /opt/elasticbeanstalk/support/scripts/use-app-ruby.sh

Hope this solves issue.

bamonde commented 7 years ago

I had the same issue and found this article: http://www.dannemanne.com/posts/post-deployment_script_on_elastic_beanstalk_restart_delayed_job

Maybe that helps to solve somebody problem and/or gem`s context.

feliperaul commented 5 years ago

I was reading about cron jobs with ElasticBeanstalk and the oficial AWS docs now state this:

Note: If you have a periodic task to be run on only one instance, consider using the cron-leaderonly-linux.config file for web environments. If you're using a dedicated worker environment, consider using periodic tasks.

The file they link to is this one, which has a commit date of May 1st 2017, which is after the last comment on this issue is from April 4 2017.

I have no idea if that can help solve the main issue this gem faced after AWS did the breaking changes (which, as far as I could understand, was related to leader ellection), but now AWS links to an official script to detect the leader.

Maybe there's a chance of fixing this gem?

If not, how is everybody handling this problem (making sure that cron jobs run only on one instance at a time in Elastic Beanstalk) ?

jufemaiz commented 5 years ago

@feliperaul sounds like a good option for you to raise as a separate issue?

For the record, we shifted to using containers and thus able to run them via ECS for background tasks.

feliperaul commented 5 years ago

@jufemaiz Any writeup on that path? We really would like to use EBS to deploy (specially because autoscalling due to ephemeral instances), but Sidekiq and CRON are two of our main blocks this path right now.