search

erdl / server

A collection of scripts, documentation and to-dos associated to ERDL Servers.
0 stars 0 forks source link

Need to be able to run cron jobs on the user level #5

Closed ryantanaka closed 6 years ago

ryantanaka commented 7 years ago

Investigate:

carlosparadis commented 7 years ago

Looks good, just remember it is more so how to organize scripts in the user cron job folder or whatever way this is setup at user level.

I am particularly curious if the only way to set a script on a cron is by dragging to special folders or if there are other metaphors we can use.

ryantanaka commented 7 years ago

for personal reference: some best practices

I'm able to run cron jobs on my computer as root and as my user both. Also, we can specify what scripts to use rather than dropping scripts into the cron.hourly folder. Another level of protection would be to disallow "writes" to a script that we are using in cron jobs. I'll be posting more on this soon, and some documentation once I have enough info.

ryantanaka commented 7 years ago

Running cron jobs at the user level

We can run cron jobs at the user level using two different methods:

In our case, the former seems like a good option, however both methods will be covered.

This is a high level overview to get us started, but more info can be found on the official Ubuntu page. Some pointers on best practices can be found here as well.

Specifying the user that executes a cron task in the system crontab file

First, open the contents of the system crontab file using a text editor. It will be located at /etc/crontab.

The contents of the file should look just like this:

# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user  command
17 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
25 6    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6    * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6    1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

# symbols represent comments m : minute(s) h: hour(s) dom: day of month(s) mon: month(s) dow: day(s) of week user: the user that executes the command command: the command to be executed

To better understand how the timings work, see "Crontab Lines" section here.

Let's examine the following line from the crontab file: 17 * * * * root cd / && run-parts --report /etc/cron.hourly

The first 5 columns say that the command located in the last column will execute the files in /etc/cron.hourly on the 17th minute of every hour of every day of every month of every day of the week. This is why scrape-util runs every 17 minutes past the hour. The 6th column specifies the user that the command will be executed as.

We can add more lines following the exact same format. Specify the time(s) when you want the command to execute as a desired user.

In order to run python scripts, the python scripts themselves must contain #!/usr/bin/env python located at the first line and the script must be executable. When specifying paths to files, use absolute paths. The following line is an example of what can be added the system crontab file.

  * *   * * *   ryan    /home/ryan/Programming/test_cron_job/cron.py >> /home/ryan/Programming/test_cron_job/system_cron_job_output.txt

The above line translates to running the file cron.py located at /home/ryan/Programming/test_cron_job/ as user ryan at every minute of every hour of every day of every month of every day of the week. The second half of the command >> /home/ryan/Programming/test_cron_job/system_cron_job_output.txt just takes the output of cron.py and appends it to a text file. This is useful for logging.

Placing scripts into the cron.hourly, cron.daily, and cron.monthly execute commands as root by default. Instead we can place our scripts in an agreed upon folder, and add the appropriate lines to the system crontab file.

Setting up a user specific crontab file

Setting up a specific user crontab consists of the following steps:

  1. logging in to that user
  2. type crontab -e to open the user specific crontab file in a text editor
  3. adding the same type of lines to the crontab as mentioned above with the exception of user, which is omitted because it is implied

Checking to see if your cron job ran

There are several ways to confirm if your cron job has run.

  1. ps -ef | grep "name of script <- if the script shows up at the time when it should've run, then it ran
  2. tail -f /var/log/syslog if you have sudo permissions to view the last 10 lines of the sys log. The process should show up here if you check at the time of execution.

Other Notes

When we have tasks that should be executed by a specific user, it is recommended to create said user, and specify it in the system crontab file.

If a command is too long to execute, it should be placed in a script file instead of cluttering the crontab file.

carlosparadis commented 7 years ago

Did you try running any of the variants in our server as a test run? Could be some 2 line script appending a timestamp of the current time to a .txt file.

ryantanaka commented 7 years ago

@carlosparadis , not yet i will give that a try now

carlosparadis commented 7 years ago

That would be great. It is also a nice test run to see if it works without sudo, given you don't have sudo access yet in sea-vm. Keep an eye on the possibilities on choosing the time resolution to run the scripts (i.e. if we can choose every hour, every minute, etc and if we can freely choose whatever we prefer).

ryantanaka commented 7 years ago

It is also a nice test run to see if it works without sudo, given you don't have sudo access yet in sea-vm.

The user specific crontab method mentioned above works fine on the IT-VM. I created my own crontab using crontab -e then entered this line into my user's crontab file:

# m h  dom mon dow   command
  0 *    *   *   *   /home/ryan/cron_python.py >> /home/ryan/crontab_output_hourly.txt

The python script ran on the hour. (as you suggested, I got the current time and appended the current time to a text file).

And as for the first method (system crontab file), you can try to test it out later since you have write permissions to edit /etc/crontab. Looks like only file owner has write permissions: 

-rw-r--r--   1 root root     722 Apr  5  2016 crontab

Keep an eye on the possibilities on choosing the time resolution to run the scripts (i.e. if we can choose every hour, every minute, etc and if we can freely choose whatever we prefer).

Yes, we can choose when to run the script. All we need to do is specify it in the first five columns. It does not need to be strictly on the minute, hour, day, etc. By putting 0 * * * *, the command runs 0 times a minute, every hour of every day of every month of every day of the week. These values can be adjusted according to our needs. The scripts we need to run also do not need to be any specific folder.

carlosparadis commented 7 years ago

Wouldn't /etc/crontab run on sudo if requires sudo to create? How is this different than the cron.hourly situation?

ryantanaka commented 7 years ago

notes from meeting 10/12/17:

ryantanaka commented 7 years ago

Now that we understand cron jobs, the next step is to have a user that will be:

Step 1: Creating a "non-user" type user

With sudo permissions, run the following commands:

useradd -M <username> : this adds the user without creating a home directory usermod -L <username> : this locks the user password usermod -s /bin/false <username> : this disables any shell use to prevent further interaction with the user

Step 2: Checking that the user was created properly

Enter the following command:

cat /etc/passwd | grep <username> <-- additional information about /etc/passwd can be found by using the man 5 passwd command

The output should resemble this:

<username>:x:1001:1001::/home/<username>:/bin/false

The last field delimited by : says /bin/false which means the user can't access a shell.

A home directory is listed, however typing ls -alh /home should reveal that there actually isn't a home directory for this user.

Next, enter the following command:

cat /etc/shadow | grep <username> <-- additional information about /etc/shadow can be found by using the man 5 shadow command

The output should resemble this: 

<username>:!$6$FgAhWtTM$GEmTvG9zeNF1iZn8.VN7IMwKJBwheno7FN067Y4zYXF1NEyIFv3PG0drF9v8d2eHfAao2.950p3GiBWMLDtzO1:17457:0:99999:7:::

The second field after contains an encrypted password (i had set no up earlier), however this second field needs to begin with ! which signifies that the password (even if there isn't one) is locked.

Finally, try to log into that account you just made with the following command:

sudo su - <username>

The output should look like this:

No directory, logging in with HOME=/

Now enter the command: whoami

It should return YOUR username and not the one we just created. We were not able to effectively log in because 1. the password is locked, and 2. the shell is disabled...

Step 3: Running scripts as this user

Now that the user is set up, we can run scripts as that user by specifying that in the crontab file located at /etc/crontab.

Because the user has no home folder, you MUST give the appropriate permissions to the files and directories which the user will interact with or else the cron job won't run. (The user does not have permissions outside of its home, which we know has no home..)

See this guide for an in depth guide on how to do this but TLDR:

chmod o+x <script_name> : so that the user can execute chmod o+rw <file_or_directory_name> : so that the user can read and write to a file or directory

notes

Reasoning behind this usage pattern comes from this post.

@carlosparadis, I think you mentioned at our previous conference call 10/12 that you and Justin would meet in person at some point to discuss stuff related to user accounts. Maybe double check with him to see if he has any insight into this or suggestions.

Also could you give me access to this repo? I think some of the stuff here and in #3 could be placed in a wiki to be referenced quicker.

ryantanaka commented 7 years ago

look up location of backup scripts one is on postgres user and other on root

postgres backup script located at: /home/postgress/pg_dumpallscript.sh system backup script located at: /root/backup.sh

carlosparadis commented 7 years ago

@ryantanaka

  1. I already gave you access. It is saying it is awaiting your response. I agree this should be moved to the wiki asap.

  2. You should try to create a shell script that add bot users and/or look for one online to validate yours. BASH is not hard at all. I believe I posted a tutorial not so long ago on slack. This will facilitate the task in the long-run by a lot. Make sure to add comments on almost every line, the code can be strange to newbies.

  3. Why don't you link Justin to the wiki once you're done editing and create the shell script? That will be a better way to validate than me trying to explain to him as a middleman.

  4. Anything I missed? Did you mention about seeing all users, the permissions they have and if that is normal or not? I know you referred me to which application runs what (did you split the user for web_display yet?), but did we confirm users like www-data are being used the way they should, for example by survey_display?

ryantanaka commented 7 years ago

@carlosparadis

  1. dang sorry i missed that, i accepted n will transfer my notes over into a nicer format
  2. ok good idea
  3. ok i'll cc you on it

Did you mention about seeing all users, the permissions they have and if that is normal or not?

yeah, its in the comments which will be in the wiki soon

did you split the user for web_display yet

I did that on my computer, but had to update the permissions in my database to accomodate the new user. So, before I do anything to the IT-VM database kiosk table, I'm taking some time to understand Postgres permissions/users etc.

did we confirm users like www-data are being used the way they should, for example by survey_display?

i looked into it. when you get the chance, take a look at my comment in #3