CRON job for indexing

[canadabri]

If I run the following command, the index fails: ruby /path-to-redmine/plugins/redmine_dmsf/extra/xapian_indexer.rb

There is an error: Exception: DatabaseLockError: Unable to get write lock on /path-to-redmine/redmine-5.1.1/dmsf_index/english: Couldn't open lockfile: Permission denied

If I run this, the index works sudo ruby /path-to-redmine/plugins/redmine_dmsf/extra/xapian_indexer.rb

If I put the working command into a cron job, the command doesnt seem to get run :

sudo crontab -e /2 * ruby /path-to-redmine/plugins/redmine_dmsf/extra/xapian_indexer.rb -v

(note there is one task in the line above this one which does work properly - a mail fetch task)

Do I need to change permissions on the dmsf_index folder? I notice all my other folders are set to redmine:redmine. The index folder is root:root

[canadabri]

I am hoping someone can offer some insight as to why the cron job is not properly doing the index job.

I have tried changing the permissions in the '/path-to-redmine/dmsf_index' folder so that when I now run:

ruby /path-to-redmine/plugins/redmine_dmsf/extra/xapian_indexer.rb from the command line, the command works, and the files are indexed. Tested by uploading a new document with a keyword... run the index command, and searching for the keyword finds the file.

If I use the exact same command in a cron job, upload a new document with a new keyword and wait for the next cron cycle (in my test, the index command should run every 2 minutes) */2 * * * * ruby /path-to-redmine/plugins/redmine_dmsf/extra/xapian_indexer.rb -v

The search for the new keyword does not return anything. I have tried both a cron job using sudo, and a cron job for the logged in account.

So, there must be something that is required in the config, or in the index command to get it to work as a scheduled job. My syslog shows that the cron task was run.

Is there anything that is required from this part of the install instructions? I am not sure what to put in that file when the command works when run directly from the command line.

Change the configuration part of redmine_dmsf/extra/xapian_indexer.rb file according to your environment. (The path to the index database set in xapian_indexer.rb must corresponds to the path set in the plugin's settings.)

[picman]

The cron job must be run with the same account as your web server. Is your web run with ruby user? For Apache the default user is www-data on Debian, so I've in cron:

@daily www-data ruby /path-to-redmine/plugins/redmine_dmsf/extra/xapian_indexer.rb

You have to specify a user for the cron job, otherwise it is run with root and the created index is not readable for the web app user.

[canadabri]

Server us Ubuntu (22.04), and web server is Apache. I assume the web is run as www-data (all files and folders are chowned as www-data). Linux is still somewhat new to me so I am not sure how to confirm all this.

I did just create a new crontab (not using sudo) using the same syntax, but still not getting any index results.

*/2 * * * * www-data ruby /opt/www/site-name/plugins/redmine_dmsf/extra/xapian_indexer.rb

admin_user@server:/opt/www/site-name/dmsf_index/english$ ls -l total 148 -rw-rw-r-- 1 www-data www-data 24576 Jan 2 08:37 docdata.glass -rw-rw-r-- 1 www-data www-data 0 Jan 2 08:37 flintlock -rw-rw-r-- 1 www-data www-data 117 Jan 2 08:37 iamglass -rw-rw-r-- 1 www-data www-data 24576 Jan 2 08:37 position.glass -rw-rw-r-- 1 www-data www-data 57344 Jan 2 08:37 postlist.glass -rw-rw-r-- 1 www-data www-data 8192 Dec 19 13:06 spelling.glass -rw-rw-r-- 1 www-data www-data 8192 Dec 19 13:06 synonym.glass -rw-rw-r-- 1 www-data www-data 24576 Jan 2 08:37 termlist.glass

[picman]

1. Be sure that you have unchecked "Search titles only" and checked "DMS Documents" in the search form.
2. Check log/production.log for any errors after a search is triggered.

[canadabri]

Search title only is not checked. DMS Documents is selected. I cannot see any errors in the production log, unless this is useful: Processing by SearchController#index as HTML

Apache envars are default, so web is running as www-data.

[canadabri]

Permissions in the files/dmsf folder

admin_user@server:/opt/www/site-name/files/dmsf/2024/01$ ls -l total 24 -rw-r--r-- 1 www-data www-data 8561 Jan 2 08:35 240102133508_5_test3.odt -rw-r--r-- 1 www-data www-data 8753 Jan 2 08:39 240102133923_5_test3.odt

[picman]

Disable the cronjob temporarily and run the indexation manually with -v to see whether the above listed files are being indexed.

sudo -u www-data ruby /opt/www/site-name/plugins/redmine_dmsf/extra/xapian_indexer.rb -v

[canadabri]

That command works.

admin_user@server:/opt/www/site-name$ sudo -u www-data ruby /opt/www/site-name/plugins/redmine_dmsf/extra/xapian_indexer.rb -v Trying to load Redmine environment <</opt/www/redmine-5.1.1/config/environment.rb>>... Redmine environment [RAILS_ENV=production] correctly loaded ... /usr/bin/omindex -s english --db /opt/www/redmine-5.1.1/dmsf_index/english files/dmsf --url / --depth-limit=0 -v [Entering directory ""] [Entering directory "2024/"] [Entering directory "2024/01/"] Indexing "2024/01/240102133923_5_test3.odt" as application/vnd.oasis.opendocument.text ... added Indexing "2024/01/240102133508_5_test3.odt" as application/vnd.oasis.opendocument.text ... already indexed [Entering directory "2023/"] [Entering directory "2023/12/"] Indexing "2023/12/231219182642_3_test.odt" as application/vnd.oasis.opendocument.text ... already indexed Indexing "2023/12/231219180605_1_testpdf.pdf" as application/pdf ... already indexed Indexing "2023/12/231219182038_2_test.docx" as application/vnd.openxmlformats-officedocument.wordprocessingml.document ... already indexed Indexing "2023/12/231222182732_4_test2.odt" as application/vnd.oasis.opendocument.text ... already indexed Redmine DMS documents indexed

(note: my 'site-name' is a symbolic link to 'redmine-5.1.1' )

[picman]

And there is still no search result? Is the document found if you search it by its title? The production.log still error free?

[canadabri]

oh the document is now found in search once that command was run.

But if the job is run by a cron task, the indexing does not seem to happen or does not happen correctly

[picman]

Redirect the output from the cron job to a log file to see the output. Then check it for any errors.

*/2 * * * * www-data ruby /opt/www/site-name/plugins/redmine_dmsf/extra/xapian_indexer.rb -v >> /tmp/xapian.log 2>&1

[canadabri]

This is all I am seeing in that log file:

cat /tmp/xapian.log /bin/sh: 1: www-data: not found /bin/sh: 1: www-data: not found

[picman]

The format of your cron job seems to be the problem. You should have a file, e.g. myjobs located at /etc/cron.d, so

cat /etc/cron.d/myjob
*/2 * * * * www-data ruby /opt/www/site-name/plugins/redmine_dmsf/extra/xapian_indexer.rb -v >>  /tmp/xapian.log 2>&1 

Is it so?

P.S. Don't forget to delete the previous cron job added by crontab -e.

[canadabri]

All that exists in /etc/cron.d is: e2scrub_all php

I have cleared out the job from sudo crontab -e as well as crontab -e

Now just created the same job again using just crontab -e (using the syntax provided)

will wait a couple minutes and search for my new keyword (recently uploaded new doc version now that previous keyword was indexed by manual run of command)

the cron job just ran. the ouput in the xapian.log remains the same..

cat /tmp/xapian.log /bin/sh: 1: www-data: not found /bin/sh: 1: www-data: not found /bin/sh: 1: www-data: not found /bin/sh: 1: www-data: not found /bin/sh: 1: www-data: not found /bin/sh: 1: www-data: not found /bin/sh: 1: www-data: not found /bin/sh: 1: www-data: not found /bin/sh: 1: www-data: not found /bin/sh: 1: www-data: not found /bin/sh: 1: www-data: not found

[canadabri]

This shows the CRON events in my syslog.

The email fetch command is setup using sudo crontab -e - that command works.

just below you can see the cron task for admin_user (the one being worked on here). It is getting run, but it is just not working as expected.

admin_user@server:/etc$ grep CRON /var/log/syslog | tail -10 Jan 2 11:10:01 server CRON[476393]: (CRON) info (No MTA installed, discarding output) Jan 2 11:10:03 server CRON[476394]: (CRON) info (No MTA installed, discarding output) Jan 2 11:12:01 server CRON[476453]: (admin_user) CMD (www-data ruby /opt/www/site-name/plugins/redmine_dmsf/extra/xapian_indexer.rb -v >> /tmp/xapian.log 2>&1) Jan 2 11:12:01 server CRON[476454]: (root) CMD (rake -f /opt/www/site-name/Rakefile redmine:email:receive_imap RAILS_ENV=production host=smtp-server port=993 ssl=1 move_on_failure=INBOX\Failed username=smtp-user password= allow_override=project,assigned_to,tracker,status,priority unknown_user=accept no_permission_check=1) Jan 2 11:12:01 server CRON[476451]: (CRON) info (No MTA installed, discarding output) Jan 2 11:12:03 server CRON[476452]: (CRON) info (No MTA installed, discarding output) Jan 2 11:14:01 server CRON[476513]: (root) CMD (rake -f /opt/www/site-name/Rakefile redmine:email:receive_imap RAILS_ENV=production host=smtp-server port=993 ssl=1 move_on_failure=INBOX\Failed username=smtp-user password= allow_override=project,assigned_to,tracker,status,priority unknown_user=accept no_permission_check=1) Jan 2 11:14:01 server CRON[476514]: (admin_user) CMD (www-data ruby /opt/www/site-name/plugins/redmine_dmsf/extra/xapian_indexer.rb -v >> /tmp/xapian.log 2>&1) Jan 2 11:14:01 server CRON[476511]: (CRON) info (No MTA installed, discarding output) Jan 2 11:14:02 server CRON[476512]: (CRON) info (No MTA installed, discarding output)

[picman]

Add the cron job as a script into /etc/cron.d. Don't use crontab -e.

[canadabri]

Ok - I think I got the cron added in there...

I had to use sudo nano siteindexer and added this as the content: */2 * * * * www-data /usr/bin/ruby /opt/www/redmine-5.1.1/plugins/redmine_dmsf/extra/xapian_indexer.rb -v >> /tmp/xapian.log 2>&1

Then waited a few minutes. The site is still not updating the indexing as expected, however there is much more detail in the xapian.log now.

Trying to load Redmine environment <</opt/www/redmine-5.1.1/config/environment.rb>>... /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:63:in rescue in block (2 levels) in require': There was an error while trying to load the gem 'redmine_crm'. (Bundler::GemRequireError) Gem Load Error is: Could not create directory /var/www/public/plugin_assets/redmine_crm/images: Permission denied @ dir_s_mkdir - /var/www/public Backtrace for gem load error is: /var/lib/gems/3.0.0/gems/redmine_crm-0.0.62/lib/redmine_crm/assets_manager.rb:18:inrescue in install_assets' /var/lib/gems/3.0.0/gems/redmine_crm-0.0.62/lib/redmine_crm/assets_manager.rb:15:in install_assets' /var/lib/gems/3.0.0/gems/redmine_crm-0.0.62/lib/redmine_crm.rb:60:in<top (required)>' /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:60:in require' /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:60:inblock (2 levels) in require' /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:55:in each' /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:55:inblock in require' /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:44:in each' /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:44:inrequire' /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler.rb:187:in require' /opt/www/redmine-5.1.1/config/application.rb:18:in<top (required)>' <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in require' <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:inrequire' /opt/www/redmine-5.1.1/config/environment.rb:4:in <top (required)>' <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:inrequire' <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in require' /opt/www/redmine-5.1.1/plugins/redmine_dmsf/extra/xapian_indexer.rb:119:in

' Bundler Error Backtrace: from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:59:in block (2 levels) in require' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:55:ineach' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:55:in block in require' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:44:ineach' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:44:in require' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler.rb:187:inrequire' from /opt/www/redmine-5.1.1/config/application.rb:18:in <top (required)>' from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:inrequire' from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in require' from /opt/www/redmine-5.1.1/config/environment.rb:4:in<top (required)>' from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in require' from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:inrequire' from /opt/www/redmine-5.1.1/plugins/redmine_dmsf/extra/xapian_indexer.rb:119:in <main>' /var/lib/gems/3.0.0/gems/redmine_crm-0.0.62/lib/redmine_crm/assets_manager.rb:18:inrescue in install_assets': Could not create directory /var/www/public/plugin_assets/redmine_crm/images: Permission denied @ dir_s_mkdir - /var/www/public (RuntimeError) from /var/lib/gems/3.0.0/gems/redmine_crm-0.0.62/lib/redmine_crm/assets_manager.rb:15:in install_assets' from /var/lib/gems/3.0.0/gems/redmine_crm-0.0.62/lib/redmine_crm.rb:60:in<top (required)>' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:60:in require' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:60:inblock (2 levels) in require' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:55:in each' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:55:inblock in require' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:44:in each' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:44:inrequire' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler.rb:187:in require' from /opt/www/redmine-5.1.1/config/application.rb:18:in<top (required)>' from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in require' from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:inrequire' from /opt/www/redmine-5.1.1/config/environment.rb:4:in <top (required)>' from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:inrequire' from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in require' from /opt/www/redmine-5.1.1/plugins/redmine_dmsf/extra/xapian_indexer.rb:119:in

' /usr/lib/ruby/3.0.0/fileutils.rb:253:in mkdir': Permission denied @ dir_s_mkdir - /var/www/public (Errno::EACCES) from /usr/lib/ruby/3.0.0/fileutils.rb:253:infu_mkdir' from /usr/lib/ruby/3.0.0/fileutils.rb:231:in block (2 levels) in mkdir_p' from /usr/lib/ruby/3.0.0/fileutils.rb:229:inreverse_each' from /usr/lib/ruby/3.0.0/fileutils.rb:229:in block in mkdir_p' from /usr/lib/ruby/3.0.0/fileutils.rb:211:ineach' from /usr/lib/ruby/3.0.0/fileutils.rb:211:in mkdir_p' from /var/lib/gems/3.0.0/gems/redmine_crm-0.0.62/lib/redmine_crm/assets_manager.rb:16:ininstall_assets' from /var/lib/gems/3.0.0/gems/redmine_crm-0.0.62/lib/redmine_crm.rb:60:in <top (required)>' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:60:inrequire' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:60:in block (2 levels) in require' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:55:ineach' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:55:in block in require' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:44:ineach' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler/runtime.rb:44:in require' from /var/lib/gems/3.0.0/gems/bundler-2.4.22/lib/bundler.rb:187:inrequire' from /opt/www/redmine-5.1.1/config/application.rb:18:in <top (required)>' from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:inrequire' from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in require' from /opt/www/redmine-5.1.1/config/environment.rb:4:in<top (required)>' from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in require' from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:inrequire' from /opt/www/redmine-5.1.1/plugins/redmine_dmsf/extra/xapian_indexer.rb:119:in `

'

[picman]

Redmine's install docs: https://www.redmine.org/projects/redmine/wiki/RedmineInstall#Step-8-File-system-permissions (Your user is www-data not redmine).

[canadabri]

I did think that document says you must have permissions set to redmine:redmine? I thought it was saying "assuming your application is run with a user named redmine...set to redmine:redmine

Other install docs have said to set file permissions to the same user running the application, which according to Apache2 default is www-data (seen in /etc/apache2/envvars both user and group are 'www-data' ), and other install docs have indicated to use www-data:www-data for file permissions.

Thanks for your help! I'll keep on chipping away at this. If I find something I will post back here.

[canadabri]

I did figure out how to get site indexing working via CRON task.

The cron task itself is set to run a script. The cron task is set for the web account (in my case, www-data)

sudo crontab -e -u www-data /1 * /usr/local/bin/indexer.sh

then the content of that indexer.sh file (obviously, the redmine root_dir will vary based on the installation):

!/bin/bash

cd /redmine/root_dir /usr/bin/ruby /redmine/root_dir/plugins/redmine_dmsf/extra/xapian_indexer.rb -v

make sure the SH file is chmod to 777. Done!